A Critical Analysis of Document Out-of-Distribution Detection
Authors: Jiuxiang Gu
Yifei Ming
Yi Zhou
Jason Kuen
Vlad I Morariu
Handong Zhao
Ruiyi Zhang
Nikolaos Barmpalios
Anqi Liu
Yixuan Li
Tong Sun
Ani Nenkova
Venue: EMNLP 2023 Findings (2023)
Keywords: Document Understanding
Pretraining
Out-of-Distribution
Document intelligence
Robustness

Abstract: Large-scale pre-training is widely used in recent document understanding tasks. During deployment, one may expect that models should trigger a conservative fallback policy when encountering out-of-distribution (OOD) samples, which highlights the importance of OOD detection. However, most existing OOD detection methods focus on single-modal inputs such as images or texts. While documents are multi-modal in nature, it is underexplored if and how multi-modal information in documents can be exploited for OOD detection. In this work, we first provide a systematic and in-depth analysis on OOD detection for document understanding models. We study the effects of model modality, pre-training, and fine-tuning across various types of OOD inputs. In particular, we find that spatial information is critical for document OOD detection. To better exploit spatial information, we propose a spatial-aware adapter, which serves as a parameter-efficient add-on module to adapt transformer-based language models to the document domain. Extensive experiments show that adding the spatial-aware adapter significantly improves the OOD detection performance compared to directly using the language model and achieves superior performance compared to competitive baselines.

A Critical Analysis of Document Out-of-Distribution Detection
Jiuxiang Gu
Yifei Ming
2+1
Yi Zhou'
Jason Kuen'
Vlad I. Morariu' Handong Zhao'
Ruiyi Zhang'
Nikolaos Barmpalios'
Anqi Liu Yixuan Li
Tong Sun'
Ani Nenkova'
Adobe Research Uniieersity of Wisconsin-Madison
"Johns Hopkins University
[jigu, kuen, morariu, hazhao, barmpali, ruizhan,, tsun, nenkova nenkova]@adobe com
"[alvinming, sharonli @@cs. http://wwsss.....oo...... edu byzhou188@jhin edu Baliuêcs, juu. edu
Abstract
Laggehhan Secw BuRCE
OOD Deacke
A00
Zyable
Large-scale pre-training is widely used in re- cent document understanding tasks. During de- ployment, one may expect that models should trigger a conservative fallback policy when en- countering out-of-distribution (OOD) samples, which highlights the importance of OOD detec- tion. However, most existing OOD detection methods focus on single-modal inputs such as images or texts. While documents are multi- modal in nature, is underexplored and how multi-modal information in documents can be exploited for OOD detection. In this work, we first provide a systematic and in in-depth analysis on OOD detection for document understanding models. We study the effects model modal- ity. pre-training, and fine-tuning across various types of OOD inputs. In particular, we find that spatial information critical for document OOD detection. To better exploit spatial in- formation, we propose a spatial-aware adapter, which serves parameter-effiient add-on module to adapt transformer-bast based language models to the document domain. Extensive ex- periments show that adding the spatial-aware adapter significantly improves the OOD detec- tion performance compared to directly using the language model and achieves superior per- formance compared to competitive basclines.
ALET
Y
D
(
D
Figure I: Illustration of OOD detection for document classification. The pre-training and fine-tuning pipelines shown the top left and bottom left, respectively. During inference time, an OOD score can be derived based on logits g(x) or feature embeddings x := f((r). A document input is identified OOD if its OOD score is below some threshold ?.
Right:
changing data distributions in machine-assisted medical document analysis systems (Velavan and Meyer, 2020). This motivates the need for reli- able document understanding models against out- of-distribution (OOD) inputs.
The goal of OOD detection is categorize in- distribution (ID) samples into one of the known categories and detect inputs that do not belong to any known classes at test time (Bendalc and Boult, 2016). plethora of OOD detection methods has been proposed for single-modal (image or text) in- puts (Ge et al., 2017; Nalisnick et al., 2019; Oza and Patel, 2019; Tack et al., 2020; Hsu et al., 2020; Arora al., 2021; Zhou et al., 2021; Xiao et al., 2020: Xu et al., 2021a; Li et al., 2021b; Shen al.,
1 Introduction
The recent success of large-scale pre-training has propelled the widespread deployment of deep learn- ing models in the document domain, where model predictions are used to help humans make decisions in various applications such as tax form processing and medical reports analysis. However, models are typically pre-trained on data collected from the web but deployed in environment with dis- tributional shifts (Cui et al., 2021). For instance, the outbreak of COVID-19 19 has led to continually
2021; Jin al., 2022; Zhou et al., 2022: Ming al., 2022b,c; Podolskiy et al., 2021; Ren et al., 2023). Recent works (Fort et al., 2021; Esmaeilpour et al., 2022; Ming et al., 2022a; Ming and Li, 2023; Bitter- wolf et al., 2023) also demonstrate promising OOD detection performance based on large-scale mod- cls pre-trained on text-image pairs, as pre-training enables models to learn powerful and transferable feature representations (Radford et al., 2021). How- ever, remains largely unexplored if existing find- ings in the OOD detection literature for images or texts can be naturally extended to the document
Equal contribution
Work done during the internship at Adohe Research

---

domain.
in the recent OOD detection literature on vision- language pre-trained models.
Multiple unique challenges exist for document OOD detection. Unlike natural images, texts, or image-text pairs, no captions can describe a docu- ment and images in documents rarely contain nat- ural objects. Moreover, the spatial relationship of text blocks further differentiates multimodal learn- ing in documents from multimodal learning in the vision-language domain (Lu et al., 2019; Li al., 2020). In addition, while recent pre-training meth- ods have demonstrated remarkable performance in downstream document understanding tasks (Xu et al., 2020, b021b; Li et al., 2021a; Gu et al., 2022; Hong et al., 2022; Huang et al., 2022: Li et al.. 2022: Wang et al., 2022a), existing vre-training datasets for documents
We further propose a spatial-aware adapter mod- ule for transformer-basee language models, fa- cilitating easy adaptation of pre-trained language models the document domain. Extensive ex- periments confirm the effectiveness of our mod- ule across diverse types of OOD data.
Preliminaries and Related Works
2.1 Document Models and Pre-Training
Large-scale pre-trained models gradually gain pop- ularity in the document domain due their success in producing generic representations from large- scale unlabeled corpora in vision and natural lan- guage processing (NLP) tasks (Devlin et al., 2018; Lu et al., 2019; Su et al., 2019; Schiappa et al.. 2022). As documents contain both visual and tex- tual information distributed spatially in semantic regions, document-speciffe models and pre-training objectives are often necessary, which are distinct from vision or language domains.
limited and lack di- versity. This is in sharp contrast common pre- training datasets for natural images. remains un- derexplored whether existing OOD detection meth- ods are reliable in the document domain and how pre-training impacts OOD reliability.
In this work, we first present comprehensive study to better understand OOD detection in the document domain through the following questions: (I) What is the role of document pre-training? How do pre-training datasets and tasks affect OOD detec- tion performance? (2) Are existing OOD detection methods developed for natural images and texts transferrable to documents? (3) How does modal- ity (textual, visual, and especially spatial informa- tion) affect OOD performance? In particular, find that spatial information is critical for improv- ing OOD reliability. Moreover, we propose new spatial-an spatial-aware aware adapter, a small learned module that can be inserted within pre-trained language model such as RoBERTa (Liu et al., 2019). Our module is computationally efficient and significantly im- proves both ID classification and OOD detection performance (Sec. 5.2). Our contributions sum- marized as follows:
We summarize common model structures for document pre-training in Fig. 2a. Specifically, Lay- outLM (Xu et al., 2020) takes a sequence of Op- tical Character Recognition (OCR) (Smith, 2007) words and word bounding boxes as inputs. It ex- tends BERT to Iearn contextualized word repre- sentations for document images through multitask learning, LayoutLMv2 (Xu et al., 2021b) improves on the prior work with new pre-training tasks to model the interaction among texts, layouts, and im- ages. DocFormer (Appalaraju et al., 2021) adopts a CNN model to extract image grid features, fusing the spatial information as an inductive bias for the self-attention module. LayoutLMv3 (Huang et al.. 2022) further enhances visual and spatial character- istics with masked image modeling and word pord- alignment tasks. Another line of work focuses on various granularities of documents, such as region- level text/image blocks. Examples of such models include SelfDoc (Li et al., 2021a), UDoc (Gu et al., 2021), and MGDoc (Wang et al., 2022b), which are pre-trained with cross-modal encoder to capture the relationship between visual and textual features. These models incorporate spatial information by fusing position embeddings at the output layer of their encoders, instead of the input layer. Addi- tionally, OCR-free models (Kim et al., 2022; Tang et al., 2023) tackle document understanding as a se-
We provide an extensive and in-depth study to in- vestigate the impacts of pre-training, fine-tuning, model-modality. and OOD scoring functions on broad spectrum of document OOD detection tasks. Our codebase will be open-sourced to fa- cilitate future research.
We present unique insights on document OOD detection. For example, we observe that distance- based OOD scores are consistently advantagcous over logit-based based scores, which is underexplored

---

quence generation problem, unifying multiple tasks through an -nage-lo-sequence generation network. While these pre-trained models demonstrate promising performance on downstream applica- tions, their robustness to different types of OOD data, the influence of pre-training and fine-tuning, and the value of different modalities (e.g. spatial, textual, and visual) for document OOD detection remain largely unexplored.
can be formulated as:
ID S(æ; h 9) 2 S(z; h, g) <
G, OOD
where S(.) is scoring function that measures OOD uncertainty. In practice, the threshold qy is often chosen so that a high fraction of ID data (e.g., 95%) )) is above the threshold.
OOD detection scores. We focus on two major categories of computationally efficient OOD detec- tion methods'; logit-based methods derive OOD scores from the logit layer of the model, while distance-based methods directly leverage feature embeddings, shown in Fig. 1. We describe few popular methods for each category as follows.
2.2
ont-of-Distributtin Detection
OOD detection has been extensively studied for open-world multi-class classification with natural image and text inputs, where the goal to derive an OOD score that separates OOD from ID sam- ples. A plethora of methods are proposed for deep neural networks, where the OOD scoring function is typically derived based on logits (without soft- max scaling) (Hendrycks al., 2022), softmax outputs (Liang et al., 2018: Hsu et al.. 2020; Huang and Li, 2021; Sun al., 2021), gradients (Huang et al., 2021), and feature embeddings (Tack al., 2020; Fort et al., 2021; Ming et al., 2023). De- spite their impressive performance on natural im- ages and texts, it is undercxplored if the results are transferrable to the document domain. A recent work (Larson et al., 2022) studied OOD detection for documents but only explored limited num- ber of models and OOD detection methods. The impacts of pre-training, fine-tuning, and spatial in- formation remain unknown. In this work, we aim to provide comprehensive and finer-grained analysis to shed light on the key factors for OOD robustness in the document domain.
Logit-based: Maximum Softmax Probability (MSP) score (Hendrycks and Gimpel, 2017) SMSP = max e /E ed,(z) natu- rally arises as a classic baseline as models of- ten output lower softmax probabilities for OOD data; Energy score (Liu et al., 2020): SEnergy = log Eie edi(w) utilizes the Helmholtz free en- ergy of the data and theoretically aligns with the logarithm of the ID density; the simple MaxLogit score (Hendrycks et al., 2022): SMexlogit max f(( has demonstrated promising per- formance on large-scale natural image datasets.
We select the above scores due to their simplicity and computational efficiency. In addition, recent studies demonstrate that such simple scores are particularly effective with large-scale pre-trained models in vision (Fort et al., 2021) and vision- language domains (Ming al., 2022a; Bitterwolf al., 2023). We complement previous studies and investigate their effectiveness for documents. Distance-baseds Distance-based methods di- rectly leverage feature embeddings z = h(x)) based on the idea that OOD inputs are rela- tively far away from ID clusters in the feature space, compared to ID inputs. Distance-basse methods can be characterized as parametric and non-parametric. Parametric methods such Mahalanobis score (Lee et al., 2018: Sehwag al., 2021) assume ID embeddings follow class- conditional Gaussian distributions and use the Mahalanobis distance as the distance metric. On the other hand, nor-mamae methods such as KNN+ (Sun et al., 2022) use cosine similarity the distance metric.
Notations. Following prior works on OOD de- tection with large-scale pre-trained models (Ming et al., 2022a; Ming and Li. 2023), the task of OOD detection is defined with respect to the downstream dataset, instead of the pre-training data which is often hard to characterize. In document classifica- tion, we use xin and yiin = {1..... K} to denote the input and label space, respectively. Let Din [x ")))) ) be the ID dataset, where x C X and nT E yin. Let Dout - {<xput "z:"t } J= de- note an OOD test set where yout E you't and yout n yin O. We express the neural network composition of feature ex- tractor h : X + R and classifier g: Rd K which maps the feature embedding of an input to K real-valued numbers known as logits. During inference time, given an input æ, OOD detection
model f : g h
We also investigate gradient-bused methods such as Grad- Norm (Huang et al., 2021) in Appendix C.

---

Vaie Sume (n)
Spatierlangggge
Spetial Lergare Gayerian
Spalierhiia
Frcidde
Multoeda Encoder
Seite postle
Malieodal Secada
Haad
Tormal
Spudd
Vauel
Toma
Weeda Weede Baess
Midh Uboas
(a) Illustration common structures for document pre- training and classification.
(b) A detailed comparison per-category accuracy on the RVI--DIP test set.
Figure 2: (Left) Illustration of models for document pre-training and classification, with our proposed spatial- models in green blocks. Modality information is also shown atop each architecture. (Right) Evaluating fine-tuning performance for document classification of pre- pre-trained models. Models are grouped into several categories (from left to right): language-only, vision-only, and multi-modal. For comparison, the performance of corresponding models in other groups is shown in gray. The average accuracy for each model is indicated in the parenthesis.
Evaluation metries. To evaluate OOD detection performance, adopt the following commonly used metries: the Area Under the Receiver Operat- ing Characteristic (AUROC), False Positive Rate at 95% Recall (FPR95), and the multi-class classi- fication accuracy (ID Acc).
Text-only: Alternativel,, we can view document classification text classification since docu- ments often contain text blocks. To this end, we use RoBERTa (Liu et al., 2019) and Long- former (Beltagy et al., 2020) as the backbones. RoBERTa can handle up to 512 input tokens while Longformer can handle up to 4,096 input tokens. We pre-train the language models with masked language modeling (MLM) on IIT-CDIP extracted text corpus.
3 Experimental Setup
Models. Fig. 2a summarizes common structures for document pre-training and classification mod- els?. While documents typically come in the form of images (Harley et al., 2015), OCR system can be used to extract words and their coordinates from the input image. Therefore, models can use single- modal or multi-modal information. We categorize these models according to the input modalities into the following groups: (1) models using only visual features, (2) models using solely textual features, (3) models incorporating both visual and textual features, and (4) models integrating additional spa- tial (especially layout) information. Further details can be found in Appendix A.
Text+Layout: Layout information plays cru- cial role in the document domain, as shown in Fig. 3. To investigate the effeet of layout informa- tion, we adopt LayoutLM as the backbone. We will show that spatial-awar models demonstrate promising OOD detection performance. How- ever, such specialized models can be computa- tionally expensive. Therefore, we propose a new spatial-aware adapter, a small learned module that can be inserted within pre-trained language model such as RoBERTa and transforms it into a spatial-aware model, which is computationally efficient and competitive for both ID classifica- tion and OOD detection (Sec. 5.2).
Vision-only: Document classification can be viewed as a standard image classification prob- lem. We consider ResNet-50 (He et al., 2016) and VIT (Fort et al., 2021) as exemplar document image classification models. We adopt two com- mon pre-training settings: (1) only pre-trained on ImageNet (Deng et al., 2009) and (2) fur- ther pre- vre-trained on IIT-CDIP (Lewis et al., 2006) with masked image modeling (MIM)3. After pre- training, we append classifier for fine-tuning.
For comprehensiveness, we consider LayoutLMv3 and UDoc, which are large and computationally intensive. Both mod- els are pre-trained on the full IIT-CDIP for fair- ness. These models utilize different input granu- larities and modalities, including textual, visual, and spatial information for document tasks.
AAart from document classification, the Appendix we also investigate OOD detection two entity-level tasks: document entity recognition and document object detection. Note that the document classification datasct we used in
this paper, RVL-CDIP (Harley et al., 2015). is a subset of IIT-CDIP. Hence, unless otherwise specified, the IIT-CDIP pre-training data used in this paper excludes RVL-CDIP.

---

Constructing ID and OOD datasets. We con- struct ID datasets from RVL-CDIP (Harley al.. 2015), where 12 out of 16 classes are selected as ID classes. Dataset details are in Appendix A. We consider two OOD scenarios: in-domain and out- domain, based on the content (e.g., words, back- ground) and layout characteristics.
Analyzing OOD Reliability for Documents
4.1 OOD Detection Without Fine-Tuning
In this section, we begin by examining the influ- ence of pre-training datascts zero-shot OOD detection. For each model, we adopt the same pre- training objective while adjusting the amount of pre-training data. Specifically, we increase the data diversity by appending 10, 20, 40, and 100% of ran- domly sampled data from IIT-CDIP dataset (around I1M) and pre-train each model. After pre-training, we measure the OOD detection performance with KNN+ score based on feature embeddings.
In-domain OOD: To determine the OOD cat- egories, we analyzed the performance of re- cent document classification models on the RVL- CDIP test set. Fig. 2b shows the per-category test accuracy of various models. Naturally, for the classes the models perform poorly on, may expect the models to detect such inputs as OOD instead of assigning specific ID class with low confidence. We observe that the categories feetter.ormm scientific report, and presentation) result in the worst performance across most of the models with different modalities. We use these as OOD categories and construct the OOD datasets accordingly. The ID dataset is constructed from the remaining 12 categories, which we refer as in-domain OOD datasets, as they are also sourced from RVL-CDIP.
We observe that: (1) for out-domain OOD data (Fig. 4a, right), increasing the amount of pre- training data can significantly improve the zero- shot OOD detection performance (w.o. fine-tuning) for models across different modalities. Our hypoth- esis is that pre-training with diverse data is bene- ficial for coarse -raeeerrrnddddd OOD detection, such as inputs from different domains (e.g., color schemes). (2) For in-domain OOD inputs, even increasing the amount of pre-training data by over 40% pro- vides negligible improvements (Fig. 4a, left). This suggests the necessity fine-tuning for improving in-domain OOD detection performance (Fig. 6).
In the open-world setting, test inputs can have significantly different color schemes and layouts compared to ID samples. To mimic such scenarios, we use two public datasets out-domain OOD test sets: NJU- Fudan Paper-Poster Dataset (Qiang et al., 2019) and CORD (Park et al., 2019). NJU-Fuddn Paper- Poster Dataset contains scientific posters in dig- ital PDF formatt. CORD is receipt under- standing dataset with significantly different in- puts compared to RVL-CDIP. As shown in Fig. 3, receipt images can be challenging and require models to handle not only textual but also visual and spatial information.
Out-domain OOD:
We further explore a more restricted setting for zero-shot OOD detection where potential OOD cat- egories are removed from the pre-training dataset LIT-CDIP. First, we use LayoutLM fine-tuned on RVL-CDIP to predict labels for all documents in ITT-CDIP. Fig. 4b summarizes the distribution of the predicted classes on IIT-CDIP. Next, we remove the "OOD" categories from IIT-CDIP and pre- train two models (RoBERTa and LayoutLM) with 10, 20, 40, and 100% of randomly sampled data from the filtered IIT-CDIP (dubbed III-CDIP)), respectively. The zero-shot OOD performance for in-domain and out-domain OOD is shown in Fig. 4c For RoBERTa, we observe similar trends as in Fig. 4a, where increasing the amount of pre- training data improves zero-shot OOD detection performance for out-domain data. However, the zero-shot performance of LayoutLM benefits from a larger pre-training dataset. In particular, given the same amount of pre- pre-training data, LayoutLM consistently outperforms RoBERTa for both in- domain and out-domain OOD detection, which suggests that spatial information can be essential
We further support our domain selection using OTDD (Alvarez-Melis and Fusi, 2020), flexible geometric method for comparing probability dis- tributions, which enables datasets regardless of their label sets. We observe clear gap between in-domain and out-domain data, which aligns with our data selection. Further de- tails can be found in Appendix A.
compare any two
NNot that we do not show (%% in Fig. 4c since we pre-train LayoutLM M from scratch.
"Extracted using http:://hhhh...comm//mmppttphhhp

---

The Sakee
lnviice
edvertionnan
budge
adeatific pullicution
quoiitmaire
handwritton
specifiatiion
CEAINE
SEVI ueiic
In-distrbutton Catepuries
keter
Ferms
presantation
sianddia repuct
witeolifi poder
Out-al- Outtal-dittribuuion Categuries (In- no-Doomnn)
dut-d--ie Categaries (Un-DDeman))
Figure 3: (Top) Examples of ID inputs sampled from RVL-CDIP (top). (Bottom) In-domain OOD from RVL-CDIP, and out-domain OOD from Scientific Poster and Receipts.
25%
+. Vaa
Vi Foars
Love Litraiii Nowo
ege hod Foae
655
25. Sh
Jirostgg Eln-cc daa
1284
Paceaagg CDC lil
(a) Pre-train on IIT-CDIP.
(b) Analysis of IIT-CDIP
(c) Pre-train on ITT-CDIP
Figure 4: The impact of pre-training data zero-shot OOD detection performance. IIT-CDIP denotes the filtered
pre-training data after removing the "OOD" categories.
for boosting the OOD reliability in the document domain. Motivated by the above observations, dive deeper and analyze spatial-aware models next.
We consider wide range of models pre- trained on pure-text/image data (e.g., ImageNet and Wikipedia) described Appendix A.3. During fine-tuning, we combine pre-trained models with a simple classifier and fine-tune on RVL-CDIP (ID). For models before and after fine-tuning, we extract the final feature embeddings and use a distance- based method KNN+ (Sun et al., 2022) for OOD detection. The results are shown in Fig. 6. We observe the following trends. First, fine-tuning largely improves OOD detection performance for both in-domaín and out-domain OOD data. The same trend holds broadly across models with dif- ferent modalities. Second, the improvement of fine-tuning is less significant for out-domain OOD data. For examplc, on Receipt (out-domain OOD), the AUROC for pre-trained VIT model is 97.13, whereas fine-tuning only improves by 0.79%. This suggests that pre-trained models do have the po- tential to separate data from different domains due to the diversity of data used for pre-training, while remains hard for pre-trained models to perform finer-grained separation for in-domain OOD inputs. Therefore, fine-tuning is beneficial for improving OOD detection performance for both types of OOD
While pre-trained models exhibit the capability to differentiate data from various domains as re- sult of being trained on diverse range of data. We observe that achieving more precise separation for in-domain OOD inputs remains difficult. Given this observation, we further analyze the impacts of fine-tuning for OOD detection with fixed pre- training datasets in the next section. By combining pre-trained models with a simple classifier and fine- tuning on RVL-CDIP (ID), we find that fine-tuning is advantageous in enhancing the OOD detection performance for both types of OOD samples.
4.2
The Impact of Fine-Tuning on Document OOD Detection
Recent document models are often pre-trained on a large-scale dataset and adapted to the target task via fine-tuning. To better understand the role of fine-tuning, we explore the following questions: 1) How does fine-tuning impact OOD reliability for in domain and out domain OOD inputs? 2) How does model modality impact the performance?

---

Te-demain oob Dag
Out-domaia 0oD (wi))
Maye
Lapecit Mi
RotERTruee
LayscttMous
Layret
Figure 5: Comparison between representative reature-based scores and logit-based scores for spatial-aware and non-spatial-awe -ware models. Spatial-aware models colored in blue.
In-damoia OoD LRVE-COEP, Averagel
Out denuia oop (Scicatifle Postet)
Out-demaals DOp (Receipt
duaay
Restlele
Stemtaue
ViTaur
RoBERTaaee Tanne
Langtomnerase
Reskitoo
SwiRtee
ViTwor
RoBERToooe
Longorrerron
Restinfer
Bertoos
ViT dwe
RoBERtlood
Longhormetoot
Figure 6: OOD detection performance for pre-trained models w. and w.o. fine-tuning. We use a custance-basee method KNN+ as the OOD scoring function. Fine-tuning significantly improves performance for both in and out-domain OOD data.
samples. To further validate our conclusion, consider two additional in-domain OOD settings for our analysis: (I) selecting the classes model performs well on, as in-domain OOD categories; (2) randomly selecting classes OOD categories (Appendix A.2). We find that fine-tuming improves OOD detection for both settings, further verifying observations.
models tailored document processing, which we refer to as spatial-aware models, such as Lay- outLMv3 and UDoc. Given fine-tuned models. we compare the performance of logit-based and distance- based OOD scores.
Preuuin
Next, we take a closer look at the impact of model modality on out-domain OOD detection. As shown in Fig. 6 (mid and right), both vision and text-based models demonstrate strong reliability against scientific posters (OOD). However, vision- based models display stronger performance than text-based models for Receipts (OOD). This can be explained by the fact that ViT was first pre-trained on ImageNet while scientific posters and reccipts contain diverse visual information such as colors and edges for vision models to utilize (see Fig. 3). On the other hand, although fine-tuning text-based models largely improves the detection performance compared to pre- trained counterparts, utilizing only textual information can be inherently limited for out-domain OOD detection.
hopergeet
Figure 7: Illustration of our spatial-aware adapter for language models. We present 2 adapter designs (marked green box): (I) insert the adapter into the word em- bedding layer during pre-training and fine-tuning; (2) insert the adapter into the output layer for fine- tening only. For the first design, freeze the word embedding layer and learn the adapter and transformer layers.
5.1 Analysis of Spatial-Aware Models
We summarize key comparisons in Fig. 5, where MSP and Energy as exemplar logit-based scores and KNN+ as the distance-based score. Full results are Appendix C. We can see that the simple KNN-based score (KNN+) consistently out- performs logit-based scores for both in-domain and
5 The Importance of Spatial-Awarenes.
previous sections, we mainly focus on main- stream text-based and vision-base based models for in- and out-domain OOD detection. Next, we consider

---

out-domain OOD data across different models with different modalities. This is in contrast with re- cent works that investigate large-scale pre-trained models the vision-language domain, where logit- based scores demonstrate strong OOD detection performance (Fort et al., 2021). As documents are distinct from natural image-text pairs, observa- tions in the vision-language domain do not seam- lessly translate the document domain. Moreover, spatial-aware models demonstrate stronger OOD detection performance for both in and out-domain OOD. For example, with the best scoring function (KNN+), LayoutLMv3 improves the average AU- ROC by 7.09% for out-domain OOD and 7.54% for in-domain OOD data compared to RoBERTa. This further highlights the value of spatial informa- tion for improving OOD robustness for documents.
In-domain OOD (Arg)
Out-domain OOD (wg
30
50
033
80
Pre-talbed
Pro trabedd
Pize-Aueed
Figure Comparison of OOD detection performance of Ppatial-RoBERTa and RoBERTa. All models are ini- tialized with public pre-trained checkpoints trained on purely textual data and further pre-trained IIT-CDIP. The only difference is that Spatial- RoBERTa has an ad- ditional spatial-ware adapter and takes word bounding boxes additional inputs.
pended to the word embedding layer, denoted Spatial-RoBERTa (pre), which requires both pre- training and fine-tuning. This architecture is il- lustrated in the top row of Fig. 2) The adapter is appended to the final layer the text encoder, denoted as Spatial-BoBERR requires fine-tuning as the model can utilize the pre-trained textual encoder, as shown in the bottom row of Fig. 7.
Despite the impressive improvements brought by spatial-aware models, acquiring a large-scale pre- training dataset that includes spatial information remains challenging. In contrast, there is a growing abundance of pre-trained language models that are based on textual data. This motivates us explore the possibility of leveraging these pre-trained lan- guage models by training an adapter on small dataset containing document-speccfic information. By adopting this approach, we can effectively uti- lize existing models while minimizing the time and cost required for training.
(post), which only
For S RoBERTa (pre), we freeze the word embedding layer during pre-training for several considerations: 1) word embeddings learned from large-scale corpus already cover most of those words from documents; 2) pre-training on docu- ments without strong language dependency may not help improve word embeddings. For exam- ple, in semi-structured documents (e.g., forms, re- ceipts), language dependencies are not as strong as in text-rich documents (e.g., letters, resumes), which may degenerate the learned word represen- tations. In practice, each word has normalized bounding box (xo. Vo. I1, 91), where (No. yo) (#1, 31) corresponds to the position of the upper left lower right in the bounding box. To encode positional information, we employ four position embedding layers, where each layer= encodes one coordinate (e.g., .0) and produces a corresponding position embedding. The special tokens ([CLS), [SEP], and [PAD]) bounding box (0, O, O, 0). As depicted in the top row of Fig. the spatial-aware word embeddings are formed by adding position embeddings to their corresponding word embeddings.
5.2 Towards Effective Spatial-Aware Adapter
During our investigation into the effects of model modality, pre-training, and fine-tuning on various types of OOD inputs, we find that spatial/layout information plays critical role in the document do- main. However, existing pre- pre-training models such LayoutLM series. SelfDoc, and UDoc do not fully leverage the benefits of well-pre-trained lan- guage models. This raises the question of whether a large-scale language model, such as RoBERTa, can be adapted to detect OOD documents effectively. In this section, we demonstrate that incorporating adapter module that accounts for spatial informa- tion with transformer-basee pre- treitrained models can achieve strong performance with minimal changes the code. To the best of our knowledge, this is the first study to apply the adapter idea to documents.
attached with empty
Spatial-aware adapter. Given a pre-trained lan- guage model such as RoBERTa, we propose an adapter that utilizes spatial information. We con- sider two potential designs: 1) the adapter is ap-
For Spatial-RoBRRTa (post), position embed- dings are added through late fusion in the final hid- den states during fine-tuning without affecting the

---

(D ACC. and Sardeaaai CULCCOR Aega OOD AUROC
1D Acc. and Out-dorals Del Pester 000 ACROC
ID Acc. and Osi-dozain (lacelgn OOD AUROO
Aenury ECB
natorthnne Pestento
Lnagireent Mater
Eapestaap Sareee
Special Rusrget
Ruseae and
Langoretnn harr
Loue sar
Specal muaatts
Aspretat stoe
Linelt Uneeirremaen
tuastra seer
Rertect
to7aas
spucial te Axpont
Figure 9: Correlation between ID accuracy and OOD detection performance. For most models, ID accuracy is positively correlated with OOD detection performance. Language models with spatial-aware adapters (highlighted in blue) achieve significantly higher accuracy and stronger OOD robustness (in AUROC) compared to language models without adapters. Here, (+) represents further pre-training on the IIT-CDIP dataset.
pre-trained encoder. Our experiments demonstrate that introducing spatial-aware adapters during pre- training yields better results than only adding posi- tion embeddings during fine-tuning. For additional details", please refer to Appendix C. In the follow- ing, we focus on analyzing Opatial-RooBRRe (pre) and comparing both ID and OOD performance with that of the pure-text pre-trained RoBERTa.
vision-only models. Overall, Spatial-RoBERY greatly improves upon RoBERTa and matches the performance of models with more complex and spe- cialized architectures such as LayoutLM. Specif- ically, Spatial-RoBERTI RoBERTajary: achieves 97.37 ID accuracy, which is even higher than LayoutLM (97.28) and UDoc (97.36).
To summarize, our spatial-aware adapter effec- tively adapts pre-trained transformer-based text models the document domain, improving both ID and OOD performance. In addition, by freezing the original word embeddings during pre-training, the models Sratial-RoBBRRaap... and Spatial- RoBERT are parameter-efficient and thus reduce the training cost.
Soatial-RoBERTa significantly outperforms RoBERTa. To verify the effectiveness of Spatial- RoBERTa, we compare the OOD detection perfor- mance of pre-trained and fine-tuned models. The results are shown in Fig. 8, where OOD perfor- mance is based KNN+ (K=10). Full results can be seen in Table 6. Spatial- significantly improves the OOD detection performance, espe- cially after fine-tuning. For example, compared to RoBERTA (base), Spatial-RoBERTT. (base) im- proves AUROC significantly by 4.24% averaged over four in-domain OOD datasets. This further confirms the importance of spatial information for OOD detection in the document domain.
6 Conclusions
In this work, we provide a comprchensive and in- depth study on the impacts of pre-training, fine- tuning, model-modality, and OOD scores on a broad variety of document OOD detection tasks. We present novel insights document OOD detec- tion, which are under-explored or in contrast with OOD detection works based on vision-language models. In particular, we highlight that spatial information critical for OOD detection in docu- ments. We further propose a spatial-awan adapter as an add-on module to transformer-basee models. Our module adapts pre-trained language models to the document domain. Extensive experiments on broad range of datasets verify the effectiveness of design. We hope our work will inspire fu- ture research toward improving OOD robustness for reliable document understanding.
Rpatial-RoBERTa competitive for both ID classification and OOD detection. Beyond OOD detection performance, we also examine the multi-class ID classification accuracy and plot the two metrics for all models with different modali- ties in Fig. 9. We can clearly observe a positive correlation between ID accuracy and OOD detec- tion performance (measured by AUROC) for both in-domain and out-domain OOD data. Moreover, spatial- spatial-aware aware models display superior ID accuracy and OOD robustness compared to text-only and
CoaatallRoBERRaaae (pre) incorporates position infor- mation during both pre-training and fine-tuning, while Spatial- RoBERTas (post) only inserts the adapter into the output layer for fine-tuning.

---

7 Limitations
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. 2021. Exploring the limits of out-of of-disiiiibttionn de- tection. In NeurIPS.
In this work, our main focus is on OOD detection for document understanding, with specific em- phasis the context of document classification. As OOD detection based document pre- pre-trained models remains largely underexplored, we believe establishing an in-depth and extensive study of OOD detection for document classification would be a valuable stepping stone towards more complex tasks. Apart from document classification, in the Appendix B, we also investigate OOD detection for two entity-level tasks: document entity recognition and document object detection. We leave more comprehensive treatment for future works.
Zong Yuan Ge, Sergey Demyanov, Zetao Chen, and Rahil Garnavi. 2017. Generative openmax for multi-class open set classification. arXiv preprint arXiv:1707.07411
Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao, Rajiv Jain, Nikolaos Barmpalios, Ani Nenkova, and Tong Sun. 2021. Unified pretraining framework for document understanding. In NeurIPS.
Zhangxuan Gu, Changhua Meng, Ke Wang, Jun Lan, Weiqiang Wang, Ming Gu, and Liqing Zhang. 2022. XylayoutIm: Towards layout-aware multimodal net- works for visually-rich document understanding. CVPR.
Adam W Harley, Alex Uikes, and Konstantinos G Der- panis. 2015. Evaluation of deep convolutional nets for document image classification and retrieval. ICDAR.
References
David Alvarez-Melis and Nicolo Fusi. 2020. Geometric dataset distances via optimal transport. NeurlPS.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. In IEEE Conf. Comput. Vis. Paltern Recog-, pages 770-778.
Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Manmatha. 2021. Docformer: End-to-end transformer for document understanding. In ICCV.
Dan Hendryeks, Steven Basart, Mantas Mazeika. Mo- hammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. 2022. Scaling oft-t--distributiin detection for real-world settings. ICMI..
Udit Arora, William Huang, and He He. 2021. Types of out-of-distribution texts and how to detect them. In EMNLP.
1z Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.055
Dan Hendrycks and Kevin Gimpel. 2017. A baseline for detecting misclassified and out-of-distribution ex- amples in neural networks. In ICLR.
Teakgyu Hong, Donghyun Kim, Mingi Ji, Wonseok Hwang. Daehyun Nam, and Sungrae Park. 2022. Bros: pre-trained language model focusing on text and layout for better key information extraction from documents. In AAAI.
Abhijit Bendale and Terrance E Boult. 2016. Towards open set deep networks. CVPR.
Julian Bitterwolf, Maximilian Mueller, and Matthias Hein. 2023, In out? fixing imagenet out-of- distribution detection evaluation. JCML.
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and 7solt Kira. 2020. Generalized odin: Detecting out-of- distribution image without learning from out-of- distribution data. In CVPR.
Lei Cui, Yiheng Xu, Tengchao Lv, and Furu Wei. 2021. Document ai: Benchmarks, models and applications. arXiv preprint arXiv:2
Rui Huang, Andrew Geng, and Yixuan Li. 2021. On the importance of gradients for detecting distributional shifts the wild. In NeurIPS.
Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai
Li and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In CVPR.
Rui Huang Yixuan Li. 2021. Mos: Towards scaling out-of -iit---disttiuttion detection for large semantic space. In CVPR.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-traie Pre-training of deep bidirectional transformers for language understand- ing. In NAACL.
Yupan Huang. Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. Layoutlmv3: Pre-training for doc- ument ai with unified text and image masking, In ACMMM.
Xuefeng Du. Zhaoning Wang, Mu Cai, and Yixuan Li. 2022. Vos: Learning what you don't know by virtual outlier synthesis. In ICLR.
Guillaume Jaume, Hazim Kemal Ekenel, and Jean- Philippe Thiran. 2019. Funsd: A dataset for form understanding noisy scanned documents. In IC- DAR Workshop.
Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu. 2022. Zero-shot open set detection by extending clip. In AAAI.

---

Di Jin, Shuyang Gao, Scokhwan Kim, Yang Liu, and Dilek Hakkani- Tur. 2022. Towards textual out-of- domain detection without in-domain labels. TASLP.
Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. 2022a. Delving into out-of- distribution detection with vision-language represen- tations. In NeurIPS.
Geewook Kim, Teakgyu Hong, Moonbin Yim, Jinyoung Park, Jinyeong Yim, Wonseok Hwang. Sangdoo Yun, Dongyoon Han, and Seunghyun Park. 2022. Donut: Document understanding transformer without oct.
Yifei Ming, Ying Fan, and Yixuan 2022b. Poem: Dut-of-distribution detection with posterior sampling. In ICML. PMLR.
Diederik P Kingma and Jimmy Ba. 2014. Adam: method for stochastic optimization. In ICLR.
Yifei Ming and Yixuan Li. 2023. How does fine- tuning impact out-of-distribution detection for vision- language models? IJC..
Stefan Larson, Gordon Lim, Yutong Ai, David Kuang, and Kevin Leach. 2022. Evaluating out-of- distribution performance on document image classi- fiers. NeurIPS.
Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. 2023. How exploit hyperspherical embeddings for out-of-distributiio detection? In ICLR.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018. A simple unified framework for detecting out- of-distribution samples and adversarial attacks. In NeurlPS.
Yifei Ming, Hang Yin, and Yixuan Li. 2022c. On the impact of spurious correlation for out-of-distributinn detection. In AAAI
Ajoy Mondal, Peter I. Lipps, and CV Jawahar. 2020. Tiil- ar-13k: 13k: a new dataset for graphical object detection in documents. In International Workshop on Document Analysis Systems.
D. Lewis, G. Agam, S. Argamon, O. Frieder, D. Gross- man, and Heard. 2006. Building a test collection for complex document information processing. In SIGIR.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshninarayanan. 2019. Do deep generative models know what they don't know? In ICLR.
Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou. 2020. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. AAAI.
Poojan Oza and Vishal M Patel. 2019. C2ac: Class conditioned auto-encoder for open-set recognition. In CVPR.
Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, and Furu Wei. 2022. Dit: Sulf-supervised pre- training for document image transformer. In ACM MM.
Seunghyun Park, Scung Shin, Bado Lec, Junyeop Lee, Jaeheung Surh. Minjoon Seo, and Hwalsuk Lee. 2019, Cord: A consolidated receipt dataset for post-oer parsing. In NeurlPS Workshop.
Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu. 2021a. Selfdoc: Self- supervied docu- ment representation learning. In CVPR.
Alexander Podolskiy, Dmitry Lipin, Andrey Bout, Eka- terina Artemova, and Irina Piontkovskaya. 2021. Re- visiting mahalanobis distance for transformer-bas based out-of-domain detection. In AAAL.
Xiaoya Li, Jiwei Li, Xiaofei Sun, Chun Fan, Tianwei Zhang, Fei Wu, Yuxian Meng. and Jun Zhang. 2021b. kfolden: k-fold ensemble for out-of-distribution de- tection. In EMNLP.
Yu-Ting Qiang, Yan-Wei Fu, Xiao Yu, Yan-Wen Guo, Zhi-Hua Zhou, and Leonid Sigal. 2019. Learning generate posters scientific papers by probabilistic graphical models. JCST.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. 2018. Enhancing the reliability of -istr--distribution image detection in neural networks. In ICLR.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In ICML.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. 2020. Energy-basee based -ut-of-distribution detection. In NeurlPS.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv:
Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mo- hammad Saleh, Balaji Lakshminarayanan, and Pe- ter J Liu. 2023, Out-of- -iittid-istiittionn detection and selective generation for conditional language models. In ICLR.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguis- tie representations for vision-and and-langa tasks. In NewrIPS.
Madeline C Schiappa, Yogesh S Rawat, Shruti Vyas, Vibhav Vineet, and Hamid Palangi. 2022. Multi- modal robustness analysis against language and sual perturbations. In NewrIPS.

---

Vikash Sehwag, Mung Chiang, and Prateek Mittal. 2021. Ssd: A unified framework for self-supervised outlier detection. In ICLR.
Keyang Xu, Tongzheng Ren, Shikun Zhang, Yihao Feng, and Caiming Xiong. 2021a. Unsupervised out-of-domain detection via pre-trained transformers. In ACL.
Yilin Shen, Yen-Chang Hsu, Avik Ray, and Hongxia Jin. 2021. Enhancing generalization intent classification and out-of-domain detection in SLU. In ACL-UCNNLP.
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang. Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, et al. 202 Layoutmmv2: Multi- molti-modal pre pre-training for visually-rich document understanding. In ACL.
Ray Smith. 2007. An overview of the tesseract engine. ICDAR.
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020. Layoutlm: Pre-training of text and layout for document image understanding. In SIGKDD.
Weijie Su, Xizhou Zhu, Yue Cao. Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019. VI-bert: Pre-training of generic visual-linguistic representations. ICLR.
Yiyou Sun, Chuan Guo, and Yixuan Li. 2021. act: Out-of-distribution detection with rectified acti- vations. In NeurlPS.
Xu 7hong, Jianbin Tang, and Antonio Jimeno Yepes. 2019. Publaynet: largest dataset ever for document layout analysis. In ICDAR.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. 2022. Out-of-distribttion detection with deep nearest neighbors. In ICML.
Wenxuan Zhou, Fangyu Liu, Muhao Chen. 2021. Contrastive out-of-distribution detection for pre- trained transformers. In EMNLP.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jin- woo Shin. 2020. Csi: Novelty detection via con- trastive learning on distributionally shifted instances. In NeurIPS.
Yunhua Zhou, Peiju Liu, and Xipeng Qiu. 2022. KNN- contrastive learning for out-of-domain intent classifi- cation. In ACL.
Zineng Tang, Ziyi Yang. Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng. Cha Zhang, and Mohit Bansal. 2023. Unifying vision, text, and layout for universal document processing. CVPR.
Thimumalaisamy P Velavan and Christian G Meyer. 2020. The covid-19 19 epidemic. Tropical medicine & inter- national health, 25(3):278.
Wenjin Wang, Zhengjie Huang, Bin Luo. Qianglong Chen, Qiming Peng. Yinxu Pan, Weichong Yin, Shikun Feng, Yu Sun, Dianhai Yu, et al. 2022a. mm- layout: Multi-grained multimodal transformer for document understanding. In ACMMM.
Zilong Wang, Jiuxiang Gu, Chris Tensmeyer, Nikolaos Barmpalios, Aui Nenkova. Tong Sun, Jingbo Shang, and Vlad Morariu. 2022b. Mgdoc: Pre-training with multi-granular hierarchy for document image understanding. In EMNLP.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. 2019. Huggingface's 's transformers: State-of- the-art natural language processing, arXiv preprint ar1iv:1910.03771.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Yen Lo. and Ross Girshick. 2019. Detec- tron2. https://ght /ttp///gtttttt../. cwm/ hmm/faacbbokresearhhl detectron2.
Zhisheng Xiao, Qing Yan, and Yali Amit. 2020. Like- lihood regret: distt--iittionttiinn detection score for variational auto-encoder. In NewrIPS.

---

A Dataset and Model Details
The following are the hyperparameters of the mod- els used in our paper:
A.1 Datasets
Text-only:
The full RVL-CDIP dataset consists of training/vaidaaiionnessngg images under 16 categories. We select 12 of them as the ID (In-domain) data. We employ the Google OCR engine' extract the text and layout information, which provides tokens, text blocks and the corresponding bounding boxes.
320K/40K/40K
BERT and RoBERTa: We adopt RoBERTaBase (12 layers) and BERT (12 layers) as backbones and set the maximum sequence length to 512. For RoBERTa, the classifier consists of two linear layers followed by tanh activation function.
A.2 Quantifying OOD Dataset Construction
We also employ LongformetBase (12 layers) as the backbone and set the maximum sequence length 4,096.
Longformer
The distance between datasets can be measured via Optimal Transport Dataset Distance (OTDD) We visualize the OTDD distance between ID and the OOD (both in-domain and out-domain) data in Fig. 10a, where we highlight the in-domain OOD data blue and the out-domain OOD data in green. Specifically, we randomly sample 1000 images from each dataset and calculate the average dis- tance between pairs of datasets. We can see sig- nificant gap between the OTDD of in-domain OOD data and out-domain OOD data. To make the anal- ysis more thorough, we consider two additional in-domain OOD settings: (1) select the classes the model performs well as OOD data; (2) randomly select classes as OOD data. The results are shown in Fig. 10b and Fig. 10c. We can see that the dis- tance between ID and in-domain OOD is similar to the original scheme (Fig. 10a). This suggests that most in-domain OOD categories are not far from ID data.
Vision-only:
ResNet50: We adopt ResNet50 pre-trained on ImageNet- 1k the backbone. We fine-tune the model resolution of 224x224.
VIT: We consider ViT (vit-base-patchl6- 224, pre-trained ImageNet-21k) the backbon and fine-tune resolution of 224x224.
SwinB: We also use the Swin Transformer hitiibaaae-pacch--wddddd-22222 trained on ImageNet-21k) as the backbone and fine-tune the model at a resolution 224x224.
pre-
While this paper represents an initial endeavor, we hope that our work will serve as a stepping stone towards constructing more comprehensive and diverse OOD benchmarks in the document do- main, akin to those available in the NLP and natural image domain.
Text+Layout:
LayoutLMvl: This model employs the Lay- outLM (layoutlm wewptttl--ba---ueeeeee uncasddd 12 layers, pre-trained on IIT-CDIP) as the backbone. We set the maximum sequence length to 512.
A.3 Models and Training Details
Spatial-RoBERTT B (Pre): This model combines our spatial-aware adapter to the pre- trained RoBERTazass model. The adapter is applied to the word embedding layer. We freeze the pre-trained word embeddings and optimize the spatial-aware adapter and trans- formers.
All models reported in Fig. 2b, except UDoc, are initialized with pre-trained weights from Hugging- face? and fine-tuned on the full RVL-CDIP training set. During fine-tuning, we train these models on RVL-CDIP with the cross-entropy loss. The mod- cls were optimized with Adam optimizer (Kingma and Ba, 2014) for 30 epochs with batch size of 50 and learning rate
10 8 A100 GPUs.
Opatial-RoBERTIa (Post): Instead of in- serting the spatial-awar adapter in the input layer, this model integrates the spatial-awarc adapter at the output layer of the transformer.
"https://cloue google. com/vi hmmmviisio/ddos/o/ooo h/ttps://github.. com/mi com/microoof//otdd "https: /ttttii///huginggfac. co/ models

---

$054 1144
(a) OOD (Worst perfornance).
(b) OOD (Best performance).
(c) OOD (Random selection).
Figure 10: Visualization of optimal transport dataset distance for ID and OOD (in-domain and out-domain) datasets. We highlight the in-domain OOD data in blue and the out-domain OOD data in green.
Farme Erent
Souety
Nomal Eoenn
Teaehrk Everd
Foras Soaet
(a) RoBERTa Bagee (10%)
(b) RoBERTaB (20%)
(c) RoBER Bagase (40%)
(d) RoBERTaB Tap (100%)
Pobemaaa
(0) ViT Base (10%)
(f) VIT (20%)
(g) ViTease (40%)
(h) VIT ViT (100%)
Figure 11: Feature visualization for pre-trained (with different numbers pre-training data) and fine-tuned models. We show both in-domain (RVL-CDIP) and out-domain (CORD) OOD datasets.
2

---

MSP
MISP
MSE
Energy
Energy
Energy
KNN (k=110)
KNN
(k=10)
KNN
(k=10)
(a) ResNet-50
RoBeRTa-base
(c) LayouLMM3--aase
Figure 12: MSP. Energy, KNN, and Maha score histogram distributions of ID (blue) and OOD (green) inputs derived from fine-tuned ResNet-50, RoBERTa, and Layout.MMv. The KNN scores calculated from both vision and language models naturally form smooth distributions. In contrast, MSP and Maha scores for both in- and out-of-distribution data concentrate on high values. Overall our experiments show that using feature space makes the scores more distinguishable between and -iistiibbiiiitn and, as a result, enables more effective OOD detection.
Rusty Recepention pagagge
Layentt angreer
Obfect Peene RCWec
B Beyond Document Classification
In the main paper, we mainly focus on document classification to provide thorough and in-depth analysis. In this section, we go beyond document classification and explore OOD detection for entity-level tasks in documents: document entity recognition and document object detection. is natural to detect and recognize basic units in docu- ments such as text, tables, and figures. Document entity recognition aims to predict the label for each semantic entity with given bounding boxes. Docu- ment object detection is object detection task for document images. Specifically, denote the input as æ, the bounding box coordinates associated with object instances the image as R.4, and use the model with parameters to model the bounding box regression pe x)) and the label classification po(y22, b). Given a test input ê, the OOD detection scoring function for entity detection and recogni- tion can be unified as S(i, b), where denotes the object instance predicted by the object detector. In particular, for document entity recognition, since the bounding boxes are provided, the OOD score can be simplified as S(t, b), where the given object instance.
Vhee
Teakel
Surkel
Heud
Promow
Murrit Trtioo
Figure 13: The network architectures in green blocks are our proposed models. We also show the modality information on top of each architecture.
VisionTText++ Layout::
We use LayoutLMy3 (layoutmmy--base, 12 layers, pre-trained on IIT-CDIP) as the backbonc.
LaytouLMv3.
We use slight variant of UDoc with the only difference in the sentence encoder, where we adopt smaller version of the pre- trained sentence encoder fall-MiniLMM-L6-v222 layers) instead of the larger sentence encoder (bert-base base- nli-mean meen-tokkess, 12 layers).
UDoc:
3

---

Document Object Detection. For document ob- ject detection, we use PubLayNet as the ID dataset and construct the OOD dataset from 1IIT-AR-33K, Unlike Pet, where the documents are sci- entific articles, IIT--R--133 13K a dataset for graph- ical object detection in business documents (e.g., annual reports), thus there exists an obvious do- main gap. We select natural images the OOD entity and filter images that contain the OOD en- tity. Two object detection models are considered in this paper: (7) Vanilla -eNNer-RCNN with ResNet- 50 visual backbone, and (2) Faster- RCNN with VOS (Du et al., 2022), a recent aawae learning framework to improve OOD detection per- formance for natural images. Following the origi- nal paper, we use 1,000 samples for each ID class to estimate the conditoondiii Gaussian statistics. The models are trained for 180k iterations with base learning rate of 0.01 and a batch size of using the Detectron2 framework (Wu et al., 2019). The performance of the models is measured using the mean average precision (MAP) @ intersection over union (IOU) [0.50:0.95] of bounding boxes.
the split, if we consider other as OOD, we have total of 8,330 ID and 1,019 OOD entities. Oth- erwise, we consider header as OOD, we have 8,981 1 ID and 368 OOD entities in total.
For document object detection, we consider PubLayNet (Zhong et al., 2019), which contains 336K/1 IK training/valliatiion images with 6 cate- gories (lext, tifle, fig. and table). The original IITT-AR-13K (Mondal et al., 2020) contains (table, fig., natural image, logo, and signature). this pa- per, considering the overlap between IIT--RR-133 and PubL.yNet, we select those images containing natural images as the OOD test set. After filter- ing, we obtain 2,880 OOD entities across 1,837 document images. We consider three ID datasets in this experi- ment. (!) PubLayNet: This is the original Pub- LayNet dataset. We treat all the entities in train- ing/validation images as ID entities. (2) Consider- ing the domain shift between ID data (PubLayNet) and OOD data (IIT-AR--33) 13K). We combine the PubLayNet training data with the images from IIIT- AR-13K IR-13K with overlapping annotations (table and figure) and train the object detection model.
Document Entity Recognition. For entity recog- nition, we construct ID and OOD datasets from FUNSD. Each semantic entity includes list of words, a label, and a bounding box. The stan- dard label set for this dataset contains four cate- gories: question, answer, header, and other. In this paper, we select entities labeled as other or header OOD data, and the entities belonging to the other three categories ID. Instead of treat- ing entity recognition nition problem, we follow UDoc and solve this problem at the semantic region level. We replace the sentence encoder in UDoc with a smaller sen- from Hug-
B.2 Models
Fig. 13 illustrates the entity recognition models used in this paper. We consider the entities on re- gions instead of tokens, as regions provide richer semantic information. As for the pre-trained model, we adopt UDoc (trained on IIT-CDIP) since it mod- els inputs at the regional level. Based on the UDoc framework, we develop the following models.
named-entity recog-
AilonnVision+LLayuut:
ResNet-50: This model is composed of the ResNet-50 from pre-trained UDoc. It adopts the Rol pooling followed by a classifter to extract the entity features.
tence encoder (all-MiniLM--L--22) gingface (Wolf et al., 2019). We also have the following model variants to verify the effective- ness of the combination of modalities: textual-only, visual-only, textuall visual+spatial, and vi- sual-Hextulllspattia.
This model also adapts UDoc's pre-trained ResNet-50 for further im- provement. It makes the Rol features spatially aware by adding position embeddings, which are mapped from the bounding boxes via a lincar mapping layer.
ResNee-550PPosition
We provide details on datasets and models as follows.
B.1 Datasets
The original FUNSD (Jaume et al., 2019) dataset contains 149 training and 50 testing images. For document entity recognition, we treat entities with the category other/header as OOD entities. After "httts::/huugiingface wtrpr//eee.eeeeeeeeeeeeerrrneees
Text/Text=Layout:
Sentence BERT: This model adopts the lan- guage branch of UDoc and appends the classi- fier to the output the sentence encoder.
4

---

Document Eatlt Recegnition 000 Detection
Decument Object Detection 00d Detection
8.22
6-100 (a) Comparison of OOD detection methods on different (b) OOD detection results from different object detection models on two OOD classes: other and header.
ONC
methods and models.
Figure 14: Ablation on document entity recognition and object detection. Numbers are reported FPR95.
Sentence BERT+Position: This model is close to the above model but adds position embeddings to the sentence embeddings.
detail in Table 1. We can see that the OOD detec- tion performance further improved by introduc- ing document images from IITT-R--133 1-1 with the same ID annotations as training data.
Vision-Text+Layoutt
To provide more intuitions, in Fig. 15, we visual- ize the document entity recognition OOD detection results. In Fig. 16, we visualize the prediction on sample OOD images, using object detection mod- els trained without VOS (top) and with VOS (bot- tom), respectively. We can see that vanilla Faster RCNN trained on PubLayNet produces false posi- tives when applied to the OOD document images from IIIT-AR-13K. Table shows that introduc- ing the unknown-aware learning method optimized for both ID and OOD can reduce the FPR95 while preserving the mAP on the ID data. This experi- ment indicates that incorporating uncertainty esti- mation into the entity detection training procedure can improve the reliability of the document object detection system.
ResNet-50tseetence BERT:
This model fol- lows the same framework as UDoc, but re- places the sentence encoder in their original design with a more miniature sentence coder (all-MiniLM-L6-v2))
SwinT+Sentence BERT: This model re- places the ResNet-50 visual backbone with pre-trained tiny Swin Transformer (swin-
soyy-patch4-indoww-2222 adopted from the Huggingface.
All the models are fine-tuned with the cross-entropy loss for 100 epochs, using learning rate of 10-5 and batch size of A A100 GPU.
B.3 Summary of Observations
C Detailed Experimental Results
We provide summary of observations here and hope inspire future works on a thorough investi- gation of OOD detection for entity-level tasks. To identify entity types, models should not only under- stand the words but also utilize spatial and visual information.
Table 2 corresponds to the results shown in Fig. 15 and Fig. 14a.
Table 1 corresponds to the results shown in Fig. 16 and Fig. 14b.
For document entity recognition, the compar- ison of distance-based and logit-based OOD de- tection methods with different models shown
Table 3 and Table correspond to the results shown in Fig. 4a.
in Fig. 14a. More details are shown in Table 2. We see that models can better predict the entity type and also achieve better OOD robustness with the help of spatial information. Considering the weak language dependency between entities, is not surprising that vision-based models achieve better performance than text-based models. In par- ticular, UDoc with ResNet-50 achieves the best performance on two OOD test sets, illustrating that visual information plays major role in increasing the discrimination of entities with similar seman- tics. For document object detection, we summarize our findings in Fig. 14b and describe them in more
Table 4 and Table correspond to the results shown in Fig. 4c.
Table 6 corresponds to the results shown in Fig. 8 and Fig. 9.
Table and Table correspond to the results shown in Fig. 6 and Fig.
Table 10 and Table 11 correspond to the analysis for Sec. 4 and Sec. 4.2.
Table 12 corresponds to the results shown in Fig. 9.
5

---

Figure 15: Visualization of detected OOD entities on the form images. The top part shows the entities in blue are entities annotated as other. The bottom part shows the detected OOD entities (green). We also show failure cases on the right part.
(4) Publ ay/et (18)
(b) 1EF--RR-15K (7W3))
Figure 16: Visualization of detected objects on OOD images (from HIT-AR--3K) by vanilla Faster-RCNT RCNN (top) and Faster-RCNN with VOS (bottom) is shown. Objects blue boxes detected and classified as one of the ID classes. The detected OOD objects (green) reduce false positives among detected objects. We also visualize detected objects the ID images. There clear difference between PubLayNet and IIIT-AR-1333 entities and annotations of natural images rarely exist in PubLayNet.
Table 1: Comparison with different training and detection methods.
IIT--R--AS 13K (Natural Image as OOD)
Publay (T))
Models
Dataset
OOD Score
FFRUS
Vanilla Faster-RCNN
Puhl ayNet
MSP
74.33
79.12
98.41
92.6
Energy
55.96
83.55
98.73
MSP
63.65
79.37
98.57
Faster RCNN RaNN with VOS
PuhLayNet
92.2
Energy
55.61
80.60
98.67
Faste-RCNN with VOS
PodlayyNe--HI--AA--IKKDDD
MSP
56.57
82.94
98.59
92.4
Energy
47.73
84.04
98.67
6

---

Table 2: Comparison with different models on FUNSD OOD setting. All models are initialized with UDoc pre- trained on IIT-CDIP fine-tuned on FUNSD data with entities. All values are percentages. S-BERT deontes Sentence BERT. A lower FPR95 higher AUROC value indicates better performance.
Test
Method
Other (OOD)
Header (OOD)
Test
Method
Other (OOD)
10
Header (OOD)
ID
FPRS5
AUROL
FPR55
F1
FPR95
AUROU
F1
TPR55
AUROU
FI
73.21
73.19
61.42
RoNN4150
KNNi
cm
76.35
KNN
75.14
KNN
KNN cm
MaxLogit
52.40
73.70
76.72
MaxLogii
36.94
67.00
63.41
Energy
$2.50
73.70
75.8
76.55
Energy
76.64
66.95
$4.51
64.98
KNN
01.72
48.44
V1.85
60.99
KNN
07.45
41.24
01.75
03.53
KNN
90.93
47.65
91.93
59.00
S-BrUT--Paliao
KNN
97.55
39.91
93.15
61.51
KNN
93.82
48.94
90.21
$7.90
KNN
97.15
39 56
90.39
61 76
KNNN
93.92
48.79
90.21
$5.00
KNN
97.06
41.67
91.85
00.99
77.15
KNN
93.92
47.85
52.12
93.45
$2.86
$2.41
$2,69
KNN
96.57
41.85
$7.08
59.67
59.08
$7.01
KNNoo
94.11
46.21
95. RR
49.86
KNN
97.25
40.53
90.22
$4.01
MSP
93.62
54.91
94.29
52.14
MSP
58.42
61.11
90.76
59.58
MaxLogii
93.72
54.75
94.57
MaxLogii
60.19
85.85
60.92
KNN
$3.001
647.08
RodNet REERET
SwinT--5-BFRR
KNN
75.49
70.70
MaxLogii
Maxlogi
67.12
76.A1
$2.77
Fnergy
45.51
Foenyy
74.41
7

---

Table 3: OOD detection performance for document classification with different number of pre pre-training data from IIT-CDIP. (Acc) denotes accuracy obtained by testing on test data. We report KNN-based scores for both pre-trained and fine-tuned models. Sci. Poster denotes the document images converted from NJU-Fudan Paper- Poster Dataset. Receipl denotes the receipt images collected from the CORD receipt understanding dataset. For in-domain OOD test data, we also report the averaged scores.
OOD Onizwel (In-domain)
OOD (Outobmman)
Regurt
Arerage AUROC
Rercipi
Acc
AUROC
FIROS
AUROC
TP995
AUROC
FF1995
AUROC
FPR95
AUROC
FIR99
AUROC
Pre-frain on 10%
FCDIP Bine Tume on RYP-CIIP
data
MEPP
22.73
02.21
05.55
05.05
50.30
92.00
20105
52.30
07.92
56.31
06.22
155.10
3290
MaxLogg
98.36
71.85
87:23
18.51
98.75
72.84
96.50
75.05
75.92
100.0
73.09
100.00
03.74
Energy
98.50
71.81
98.98
72.79
78.00
76.77
100.00
63.70
RoBERRTRcua (00%)
90.39
KNNN
GradNonn
98.04
0721
83.13
79.26
03.11
97.07
76.25
38.05
98.56
1302
72,83
B1.03
98.62
01.74
80.55
B5.92
98.07
07.33
07.41
77.37
100.0
85.23
$5.35
100.00
50.30
64.10
$1.45
KNNNi
63.53
38.07
65.89
$7.90
72.75
61.03
62.33
87.32
68.60
83.13
91.10
KNNKo
64.17
87.89
60.97
57.37
73.34
$4.23
67.21
88.60
67.99
87.12
72.09
57.47
91.00
83.59
KNNio
64.49
47.64
67.78
57.55
75:46
83.94
67.29
88.57
08.20
56.35
72.09
$0.83
91.50
43.21
Fre-frain on 10% DIP
(80 Ioe-tnne)
KN
$3.00
6531
9211
06/02
05:13
01.00
34.30
54.37
02.15
02.11
01.44
87.01
00.10
$4.04
KNNNco
$8.59
66.02
92.63
05.3 05.25
94.13
00.83
94.72
53.79
92.52
01.47
77.1
$5.38
64.00
83.86
KNNNco
89.73
60.40
93.53
03.12
91.37
58.94
95.17
52.33
93.20
39.71
83.72
82.97
09.20
82.29
KNNNin
90.23
62.94
93.85
61.28
94.41
57.45
95.13
51.28
93.90
58.25
83.72
90.91
70.10
81.05
Pre-fraln ou 20% I DIP
Bice Tune on RYL-CDPP
data
MSP
54.25
65.02
54.45
65.38
96101
60.30
041.1
63.05
$4.80
65.14
05.35
63.35
90.10
$4.00
MaxLogit
97.36
77.82
97.19
79.16
98:40
72.64
98.34
77.68
07.82
76.82
100.00
77.36
99.50
66.63
Energy
98.04
77.80
97.43
79.15
98.76
72.61
98.59
77.64
08. 20
76.40
100.00
77.32
99.00
66.61
RollERTaaaa (2P3))
90.71
KNN
GnalNomm
07.36
63.57
88.30
80.68
67.06
96.81
$7.06
76.04
98:44
73.66
73.20
83.92
07.88
73.00
87.80
81:35
07.69
69.
86.77
77.85
100.00
69.77
88.01
86.19
09.50
67.50
67.40
83.81
KNN
63.85
85.20
67.46
96.00
73.94
53.78
72.93
87.70
09: 54.
80.04
60.77
87.03
53.20
83.55
KNNGo
67.89
$8.02
87.54
96.71
24.
83.55
72.04
87.45
69 51
70.03
87.00
58.20
83, 12.
KNN
64.85
87.81
67.62
56.45
74.90
63.25
72.65
87.24
700000
86. 19
72.09
86.63
$8.30
82.89
Pre-trsin en 20
HTTDDIP DIP
(nn line-tune)
$7.15
63.27
06,39
92:26
02.39
95.01
33.012
01.31
02.64
43002
92.25
57,000
W7.6T
KNN 87.51
67.35
92.04
65.54
91
61.40
94.07
$2.33
01. .66
61.40
47.67
91.18
60.50
86
KNN
$8.50
65.21
92.60
63.43
02:18
99.57
95.25
$0.97
02. 13
59.92
56.08
80.64
63.70
85.20
Ko
53.53
64.20
93.13
61.61
02.22
$7.99
95.45
20.95
92.41
58.44
58.14
$8.36
66.30
$4.17
Pre-esin om 40%
Time-ti Iune on EXTCCDIP DIP
data
MSP
92.87
70.09
93.M3
65
95.05
63.19
95.30
63.54
94.29
66.13
95.35
63.03
95 95.40
61.97
MaxLogi
90.00
76:72
97.87
79.85
98:44
71.83
98. 96.36
75.41
9%. 17
76.40
96.88
78.00
98 94
75.65
Frongy
96.48
78.69
07.91
79.85
98.68
71.61
98. 38.56
75.40
98. 70
76 76
100.00
78.04
98.
75.60
90.76
KNN
GrodNorm
98.04
00.37
39..9
31.03
83.05
97.47
76.25
36:16
75.26
98:44
81.55
72.77
97.40
73:30
79:11
B1:12
19:13
90.84
77.41
55.16
100.0
07:11
87.47
97.20
72.70
19.79
77.12
KNN-n
61.57
56.72
69.05
56.74
75:46
63:43
73:46
87.00
66 64
86 36.35
68.60
80.00
73.50
89 25
KNNco
60.21
88.52
89.15
56.08
75.66
63.21
73.42
86.71
70.12
88
70.93
80.30
74.70
88.89
KNN
63.77
30.30
69.79
$5.84
76.02
$2.93
74.19
86.05
70.94
85.38
70.42
88.84
73.30
88.69
Pre-froim on 80% IET 11 DIT
ImU lin-ttnne)
KNN
65.71
67/0%
90151
65.88
90:46
67.53
94.76
$1.76
00:61
6310
15.38
95.81
57.30
85.60
KNNNo
85.27
68.21
91.64
67.48
89.74
61.32
94.51
51.01
90.35
62.00
2907
95.22
62.30
NF.61
KNN
90.19
66.60
92.21
65 23.50
90.30
59.35
91.99
29.00
90.91
60.27
41.36
94.33
06.90
86.25
KNN
$7.19
64.04
92.87
63.83
90.50
$3.94
55.09
48.44
91.34
56.76
45.35
93.86
68.30
85.14
Pre-froin 1095% LTFCPI
line fune
RVL-CDIP ID data
MSF
33.23
05.75
$4.24
63.35
35.65
63.11
34.11
65.25
34.64
66.35
78.84
02.50
35..10
71.25
MasLogit
97.54
75.86
07.95
90.23
98.48
74.01
98.25
73.59
98.13
77.67
100.00
78.73
98.50
79.36
Energy
99,20
78.84
97.95
90.22
98.52
74,00
98.77
73.35
99.35
77.65
100.00
78.72
94.770
79.29
(1005))
91.00
KNN
GiddNoon
62.57
07.58
95.26
80.81
68.90
07.91
36.96
76.37
72.39
98.25
54.73
75.25
70.37
98.25
88.25
80.09
68.52
96.30 08.08
78.
87.04
100.00
72.09
86.10
99.97
63.50
08.30
90.51
77.50
KNN
63.41
98.11
69.50
56.88
23.10
$4.56
70.30
85.11
09.20
86.92
74,42
$9.58
67.20
90.37
KNNNoo
63.85
$7.87
69.79
56.79
23.90
$4.30
71.14
87.37
09.00
86.71
76.74
$8.95
07.50
90.22
KNN
65.13
87.61
70.27
36.58
74.36
$4.00
71.75
87.65
70.50
86.46
79.07
58.44
68.30
90.19
Pre-train 100 XYT DTE Tne Bice- (uBc)
K
$4.43
70.20
90.20
08.34
60.18
94.72
32.15
19308
63.32
21:91
94,.00
16,00
91.37
KNNNo
$4.51
69.30
01.26
67.35
90.38
61.96
94.73
51.43
90.22
62.51
33.72
93.59
$1.50
90.55
KNNNo
$5.67
67.75
01.92
65.35
00.82
99.79
94.59
49.77
90.82
69.66
39.53
92.28
56.70
89.32
KNN
$6.55
66.08
92.97
63.46
91.46
58,00
95.41
48.39
91.60
58.98
44.19
91.29
61.60
88.15
8

---

Table 4: OOD detection performance for document classification with different number of pre-training data from IIT-CDIP- (remove pseudo OOD categories).
GOD Unstasel (In-dommin)
(Outodmmai))
I
Method
Scl. Regort
Presentation
Form
Letter
Averaer AUROO
Poder
Recelpt
Ace
AUROC
TIRU5
AUROC
FFRE
AUROC
TPRI5
AUROC
FFR29
AUROC"
Pre-irsin en 10%
ITT-PIIP
line-lune on RVL-CIP
MSP
90.00
69.00
82.92
68.56
02.58
64.16
91.07
66.75
90.91
66.220
96.51
$4.47
99.70
-9063
Maxlargi
97.76
76:40
07.71
30.58
98
71.26
98: 98-70
76.35
98. 20
76.68
100.0
73.51
09.80
13.52
Fnomy
98.
78:35
07.75
50.55
98.84
71.20
98.56
76. 36.93
98.41
76.60
100.0
73.46
09.80
73.31
90.62
KNN
GodlNomm
65.85
07.68
87.89
79.02
66.89
07.27
18.12
79.42
98:56
75.98
71.31
K2.83
98.56
74.55
70.44
86.85
98.00
70.77
86-38
77.52
100.00
87.21
82.62
K5.16
99.50
K3.50
87.91
75.85
KNN
66.11
87.80
66.85
R8.04
75.94
82.37
73:04
86.75
70.76
86 92
87.21
84.67
63.50
82.71
KNNGo
66.77
87.66
87.30
58.00
76.02
82.49
73,66
86.52
70.94
86.
88:37
83.73
63.50
87. 34
KNN
67.25
87.42
67.74
$7.84
76.55 18
82.15
73.99
86.26
71,29
85.92
39.55
K2.85
$3.90
8h. 95.95
Pre-fruin 18% 1EM DIP
(80 Bac c002)
K
50.35
05.18
85.74
70/24
12.51
59.55
93.34
30.02
$9.54
03:12
2017
95.31
57.50
83.13
KNNNo
86.87
64.48
87.14
60 68
03. 30
58.41
93.33
55.91
90.15
62.
37.31
94.75
58.00
81.44
KNNNoo
$7.75
62.75
99
67.3 .880
93.50
56.54
93.75
54.52
91.00
60.40
47.67
93.71
90.30
75.07
KNNNuo
$8.43
61.17
89.59
06.05
93.62
$4.91
93.99
53.40
91.41
58.35
48.84
93.09
91.50
77.00
Pre-Irain 20% IFF"
line-tuee RYL
MISP
07.55
03.32
95.15
64.31
55.90
67.36
10020
30.35
MaxLogi
69:21
98.8%
76.57
RoBEERDaua (31%))
GradNonn
98.00
77.07
K
31.60
50.10
86.65
$5.31
KNNNo
66.17
23.90
82.93
96.32
85.13
KNNNo
74.18
86.30
75.50
71.92
96.04
83.59
Pre-trsin 50%
KNNoo
Timettnne
RVT-CDIF
MaxLogit
07.16
71.94
76.26
Fnergy
65.
RoBERTon (405))
(lndliiomm
07.24
97.
76.07
77.61
75.38
KNNGo
67.57
27.27
82.
72
50
80.25
88.11
83.85
KNNco
76.01
79.01
85.02
82.56
Pre-troin on 49% DTP-
(Bo Bine Gune)
$3.79
65.14
85.33
08.92
95.30
00,30
95334
51.05
91.54
0100
3721
95.27
53.30
01900
KNN $9.50
65.07
89,80
67.61
93.89
59.10
95.38
50.17
91.21
60:49
46.31
94,41
61.50
91.00
KNN 90.59
63.39
91.64
63.68
03.77
57.55
95.66
45.63
92.92
58.76
53.40
93.06
66.40
89.72
KNNN
91.19
61.79
92.57
63.90
93.66
55.78
95.62
47.42
93.21
57.22
65.12
91.99
64.30
88.72
Tre-lroin on 100% DFFCPI
Inn-fune
RVL-UDIP ID data
MSF
91.12
68.21
51.29
56.18
95.93
51.33
95.21
63.65
$1.85
63.95
55.31
30.25
95.30
65.42
MaxLogit
97.24
78.15
97.19
90.27
98.36
72.16
98.33
75.82
57.79
76.00
100.00
73.28
59.30
75.55
Energy
97.32
78.15
97.51
50.26
98.64
72.12
98.70
75.78
98.04
76.37
100.00
73.27
59.00
75.52
RoBERTAR.n (100%) )
90.74
KR
GradNorm
00.11
97.16
3007
37.80
05507
97.39
36.51
77.36
98.40
77.39
71.83
96.05
74..9
79.05
86.25
97.75
72.14
8531
77.21
100.00
31.40
$7.74
86.32
99.00
71:30
$8.45
73.52
KNNGo
66.73
37.75
70.31
56.75
77.89
$2.51
75.28
86.13
72.55
85.79
31.40
$7.43
77.50
88.39
KNNNo
67.25
37.51
70.59
36.02
77.35
$2.32
75.41
83.84
72.75
85.35
33.72
80.83
77.50
88.25
KNNN
68.13
$7.34
71.47
$6.29
78.05
$2.08
76.14
85.60
73.45
85.35
83.72
96.39
73.50
85.21
Pre-train 100%% TTEC 1TP
(no (inu-tuma)
KNN
87.05
66644
84.49
72.34
95.01
$8.47
96.23
40.07
90.92
61.58
31.40
96.19
41
94.78
KNNGo
$3.91
71.25
95:35
$7.19
91.
60.47
36.98
95.55 90
18:40
94.
KNN
90.50
63.69
87.
69.45
95.53
$4.91
00.00
26.26
92.58
58 58
43.02
94.51
55.20
93.05
KNN
91.75
62.08
88.55
67.85
95.89
$3,05
92.30
44.81
91.35
56.95
50.00
93.60
61.16
92.04
9

---

Table 5: OOD detection performance for document classification with different number of pre-training data from IIT-CDIP- (remove pseudo OOD categories).
OOD Datasel (n-domain)
0OD Dataset (Out (Ouf-domala)
10
Method
St. Report
Presenation
Form
Leiler
Arergee AUROC
SCI Poseer
Kecelpe
Acc
AUROC
FTRSS
AUROU
ITR5
AUROC
FFRN
AUROC
TPRSS
AUROC
FF1995
AUROC"
Pre-train 10%
line tune
RVL-CDIP 11 data
MSP
42.43
76.31
35115
05:15
31.31
70.23
17.00
73391
15.03
72.47
1302
75.53
14..10
75.65
Maclaggi
41.91
91.27
55.04
59.35
54.19
65.20
44.97
90.91
49.01
89.
18:37
94.27
41.30
91.38
Energy
41.83
91.29
54.92
59.25
54,11
55.22
45.01
9097
03,97
89.21
38,37
94.29
41.30
91.42
95.89
KNNN0
GradNorm
29.15
31.81
97.25
91.30
55.01
15532
56.93
31.35
16.77
90.03
30.05
$2.49
10.33
91.05
92.79
41.31
66.89
39.11
12.13
17.42
38.37
91.79
05.05
41.40
10:30
91.82
95.65
KNN
$1.03
94.11
45.65
90.50
42.01
90.33
41.60
92.61
41.82
91.99
36.74
95.76
51.90
98.44
KNNNo
$4.39
95.25
19.31
90.46
19.36
59.94
44.52
92.21
64.90
91.00
13.72
95.33
33.20
95.35
Layaut
KNN
36.15
95.47
51.27
90.19
31.36
69.03
16.63
91.99
66.35
91.32
33.72
95.10
35.10
95.15
Presirii 10% 11 CPPP
(mn Ine-4ume)
RNN
00.95
72:30
94
65.49
90.94
72.58
94.40
67.32
92.74
69.37
48.84
91.56
56.00
75.08
KNNN0
91.50
70.54
34.95
63.91
91
70.74
94.51
65.95
93.25
67.78
$3.40
90.41
57.50
73.51
KNNNo
93.077
67.76
05.54
61.24
92.78
68.27
95.25
64.01
94.16
65.32
55.81
59.37
59.50
71.06
RNN
91.55
65.41
95.90
59.13
93.10
66.19
95.54
62.41
94.52
63.75
67.44
86.44
60.20
69.00
Pre-trsin
DTP
RYL-CDIP
data
MSP
29.20
76.78
02.37
57.15
7251
3030
75.90
MasLogit
40.95
56.19
90.59
Gradivorat
53.05
90.91
47.39
91.97
95.15
KNNNo
32.55
48.32
91.34
94.97
51.04
36.75
51.68
99.71
Loyuat.
Emu-tamm
$1.76
75.76
63.37
MacLogii
42.03
59:47
57.
64.64
69.81
38
52.35
93.00
33.00
92.55 02.89
Fnimay
85.66
KKNNo
$3.69
95.45
50.63
48.20
90..5
38.11
83.30
41.23
22.11
24.70
LaysrL Dus.
KN
90.23
KNNN
KNNN
80.57
91.39
Pre-troin on 40%
DTP- (be fin-ttne)
N7.07
20.44
11.76
53.72
86.75
$2.31
30.10
15.30
85.42
B0L71
1333
$4.95
3,30
9X244
KNN
83.95
79.03
74.93
$2.31
88.99
81.11
96.71
25.01
87.
79 36
30.23
82.56
7.30
97.03
KNN
01.47
77.23
80.30
79.90
01.78
79.75
97.40
72.60
00.25
77.37
87.21
78.19
9.00
97.02
KN
90.75
75.27
84.77
77.48
91.74
78.31
90.36
70.26
91.10
75.33
80.53
74.11
14.20
97.49
Ire-troin em 100% DIP
line-uuee on RVLACUIP 1D data
M5F
43.43
76.12
31.21
291.16
38.33
08.35
46.14
13.76
31.29
72.15
19.37
75.00
23.30
83.75
MacLoogit
35.19
91.29
50.23
$9.98
53.19
$4.54
39.99
90.71
24.64
88.35
24.42
96.39
21.40
95.57
Loynut )
Energy
35.23
91.32
50.22
59,00
53.19
$1.55
39.58
90.73
64.53
38.90
20.42
96.64
21.40
95.55
96.38
KNNN0
GradNoras
30.30
25.00
$1.35
92.54
48.61
43.47
91.29
58.18
48.56
45.29
30395
86.58
36.16
14.99
83.86
92.63
41.01
31.25
12.30
89.5%
19.77
19.77
97.39
96.71
19.20
17.50
96.35
96.37
KNNGo
27.22
94.33
45.07
91.58
45.41
98.79
34,02
95.71
37.53
92.75
19.77
97.22
18.40
96.26
KNNNo
29.66
98.49
36.28
91.32
47.09
90.45
37.50
93.33
60.23
92.35
17:48
97.04
18.70
96.30
KNNNoo
32.15
94.26
48.17
90.85
50.64
30.21
40.38
93.12
42.53
92.11
19.77
96.58
20,70
96.74
Prestrain T0P% WTSC
(mu (im--tumm)
RNN
78.74
31.67
74.45
50.56
80.53
83.71
95.01
27.33
82.18
80.89
38.37
94.62
17.70
96.12
KNN
K7.30
$0.13
77.85
79.31
83-48
82.75
95.45
75.91
64.80
39.55
46.00
93.42
14.50
95 13
KNNo
86.03
77.65
82.80
76.40
86.9
61.30
96.10
75.07
87
77.16
54.65
91.09
9.50
97.21
RNNo0
80.11
75.51
88.03
74.08
90.62
70.78
96.71
20.43
91.12
74.05
66.28
$8.50
18.00
96.82
10

---

Table 6: OOD detection performance for document classification. Spatial-RoBERRaag (Pre) or SR (Pre) denotes applying the apatial-awar adapter in the word embedding layer. Spatial-Ro RoBERTag.ee (Post) or SR (Post) denotes applying the spatial-aware adaptor output layer.
(Out-domaia)
Report
Aisrage
75.49
Maxlagit
79
Energy
70.31
90.19
KNN
RobERRAnn.
KNNos
KNNGa
38.52
88.42
62.01
33.81
61.16
87.80
39.36
87.54
32.55
56.07
70.30
91.37
KNN+ss
55.35
$8.10
60.57
88.20
$5.41
64.77
87.42
61:17
34.88
95.57
76.50
90.81
No Iine-lune
KNNGs
9511
63.52
38.15
66.34
91.37
06.92
98.82
33.37
93.35
02.54
25.55
95.39
36.00
72.99
KNNNe
63.18
58.3
91.57
56.08
32.10
93.59
26.
KNNNs
64.72
94.17
93.62
26,77
KNNN
92.67
61.57
59.3
63.37
96.01
03.85
98.17
48:35
93.61
59.23
29.07
94.34
92.80
61.62
Pre-traim on ITT-CPP
lin- Sune on RVL LLDPP (10))
MISP
40300
+4.52
$4.64
20.3%
36.26
09.72
$4.30
70.74
33.00
71.39
24.19
75.79
57.20
69.23
Maslagi agit
30.43
58.64
46 48
89.92
49.96
85.73
48.30
87.66
46.04
67.00
33.72
03:42
50.60
88:
Energy
30.45
$8.56
46
89.94
50.00
85.76
43.30
87.67
46.05
$8.01
53.72
93.45
50.60
85.71
07.11
KNN
21.91
94.41
42.19
92:05
36.65
89131
43.29
92.65
40.71
10.47
97:45
$2.30
02.05
KNN
32:31
04.28
42. 59
02:64
47.01
90.21
43.43
92.55
41.3
90.
11.63
90:31
53.30
92.80
KNNN 34.30
02.
43.63
02:
49.04
$8.03
45.41
92.
43.17
91.87
12.70
97.01
53.10
92.51
KNN 35.15
93.76
44.27
92:15
40.48
$8.63
46.14
91.97
43.76
91.63
15.12
96.51
49.70
02.
Pre-train on IITSC
KNN 78:82
78.00
79.09
73.89
77.69
$1:33
01:48
76.52
82.00
77.56
10.47
98.08
$7.30
B0U89
KNN 79.74
77.95
$2.64
72:17
79.81
$0.40
92.11
75.11
83.58
76.41
16.25
97.60
92.10
76.94
KNNes 80.42
76.67
65 13
69
82.12
78.0 91
90.00 02.98
23.01
85.18
74.61
22.09
06 66
95.70
70.53
KNN 81.43
73.70
$6.00
67.19
83.40
77.13
93.38
71.07
86 78
72.77
27.91
95, 86
96,60
64.55
MaxLogtt
Encrgy
71.96
KNNNa
KNNGa
KNN
92.77
Bur-Sune
MSP
71:27
65.38
69.1
67.00
agit
Energy
36.00
97.37
KNN
94.47
42.43
93.01
37.43
91.74
91.
31.79
25.56
95,
38.07
KNN
43.71
40.05
KNNNes
30.55
93.71
44.11
92.56
00.65
91.08
35.47
93.38
37.28
92.
25.74
95.22
21.70
11
SRC.
(mo [URM]
KNNNs
63.09
90.13
38.23
6933
71.75
83.11
39.11
75.32
79.14
76.07
75.38
$4.36
19.30
92.02
KNN
71.26
78.77
90.2
67.41
75.60
$1.38
39.04
71.15
81.07
74.88
81.80
$1.35
62.20
90.29
KNNes
63.82
80.17
78.70
50:42
67.84
81.72
71.72
K1.55
77.15
18.20
87.
75.62
74.59
94.43
60.44
85.16
76,25
91.50
65.30
86.75
09.27
$4.88
73.54
$8.20
84.96

---

Table 7: OOD detection performance for document classification with the different number of pre- pre-training data from IIT-CDIP.
OOD Dalasee Lin Jn-domain)
OOD Dutaset (Chs domalia
ID Acc
Method
ScL Report
Presentatinn
Form
Letler
Arerge
SO. Posler
Kecelpe
FFROS
AUROC
FIRSS
AUROU
TIRS5
AUROC
FFR3
AUROC
FFI05
AUROR
TPRSS
AUROC
FFRS5
AUROC
Pre-Train du 10% II DIP
Bine tune on RVLCCIIP
(1U)
MSP
$5.00
85.37
45.61
01.35
65.35
K1.33
55.35
85.35
55.95
86.05
5375
80.00
14.10
35.04
MasLogit
50.36
91.51
37:77
04.30
66:33
87:97
53.60
92.11
51.03
91.47
38.36
94.24
28.50
96.05
Energy
50.56
91.48
37.08
94.23
63.49
$7.89
55.19
92.00
51.58
91.42
38.36
94.29
29.40
95.95
(1P%)
94.89
KRNNo
GradNonn
55.56
00.40
0260
79.75
45.96
43.51
$4.79
03.02
66.92
31.60
74.00
00.34
59.44
14.27
83.07
85.87
56.72
35.00
01
79.92
47.95
20.35
$2.04
95.15
34.30
3.55
91.65
9X.21
KNN
40.80
02.70
40.38
94.43
$3.39
90.26
74.72
85.77
54.57
91.54
23.29
96.98
10.40
98.05
KNNSo
46.72
92.89
34.27
05.24
56.07
89.92
74.35
88.45
$2.90
91.62
27.40
96.56
12.50
97.80
KNN
45.48
92.89
20.33
05.67
57.62
80.56
75.04
88.25
$1.87
91.59
10.14
96.21
15.00
97.57
Pre-Irsin on ITT-CIPP
(no Bac-tume)
KNNN
58.92
45008
97.67
40,00
99.32
$4.41
29.35
20.26
08,86
46.09
95.15
92.51
6.30
KNNGo 09.88
42.47
07.75
48.57
99.3
51.75
09.15
90.56
98.83
46
96.52
92.24
8.50
97.91
KNN 58.80
41.20
07.83
48.04
99.52
53.91
99.35
38.62
98.83
45.32
95.89
91.80
10.50
97.66
KNN
58.76
41.20
97.79
47.70
99.48
52.52
99.35
35.01
09.84
44.81
98.63
91.31
14.50
97.41
Pre-irsin on 20%
1T Time-tone on RVI-CIIP (TD)
MSP
$4.56
89.01
$1.63
91:31
64:37
R5.23
60.51
85.67
57.77
88.55
60.27
80.3
44.23
93.75
MaxLogs
44.32
92.16
38.21
98.18
66.92
67.83
38.56
91.31
$1.50
91:32
15:21
91.63
36.70
94.36
Enenyy
44.56
92:17
37.89
94.34
66 56
67.5
60.19
91.22
53.
91: 13.78
46.30
92.80
41.50
94. 18
94
KNNN
GnalKum
52,20
00.51
92.38
54.92
02.04
15.81
93.75
51.67
94.29
33.79
45.41
90.775
98.11
77.54
87:012
$2.36
57.5
91.74
91.02
46.09
05.89
17.81
97.33
40.44
80.70
16.99
97.40
50.01
KNN
$1.80
92.65
03.55
98.15
55.65
90:46
78.
85. 79
57.3
91.02
19.18
97.06
19.40
97. 11
KNN
50.12
92.85
70.95
94.82
58.02
90.
78:77
85, 64
56.77
91:10
19.18
96.55
23.10
66
KNNN
45.04
92.91
34,75
95.28
60.38
59.88
78559
85.42
55.54
91.12
20.55
945.7
26.20
90.35
Fre- Irain on ITF (no
Bin- (mme))
KNNN
96.16
4113
97:51
47.12
19.45
51.06
99.11
35.79
95.67
45112
59.52
91.80
8.00
V7:41
KNN
96.12
40.71
97.51
46.79
99:48
53.50
99.31
38.31
98.80
44.5 38
94.52
91.45
8.70
97.25
KNNNoo
95.04
40.10
97.55
46.31
99
51.84
99.14
37.63
98.62
43.97
95.89
91.01
11.50
90. 99
KNNNue
96.00
39.74
07.55
45.96
99.45
51.54
99.19
37.26
98.00
43.58
97.26
90.55
14.50
98.70
Pre- froin on 49% IFT CCDP-
Bine tune
RVP- (LD)
MSF
55.03
35.05
52.21
91.31
65.39
53.52
38.38
89.20
57.53
88.75
0V.12
11.02
13.30
93.82
MasLogit
47.12
91.74
40.05
94.09
61.05
59.68
56.57
92.01
$1.20
91.63
09.86
59.81
52.90
95.45
Energy
47.12
91.75
39.94
94.10
62.35
89.62
59.00
91.85
52.00
91.58
09.86
89.65
32,70
95.44
(40%)
98.03
KRNNT
GradNonm
31.3
47.00
92113
35.76
41.90
45.31
72.79
39.64
60.69
40.45
10.30
$1.37
53.73
75.51
87.05
83.87
50.83
35.00
91.35
85.96
64.38
14:25
03523
81.12
34,00
6.30
92.95
98.30
KNNN0
52.76
92.24
45.85
93.57
48.12
91.95
74.84
85.75
55.40
91.63
32.88
95.21
7.50
98.36
KNNNco
$1.28
92.52
20.94
94.51
50.52
91.70
75.38
85.05
54,46
91.30
25.62
94.07
10:90
98.04
KNNN
50.32
92.62
36.16
95.12
53.35
91.36
75.93
85.21
53.94
91.84
34.73
94.25
13.50
97.76
Pre-frus traii DYP
10n Binc- FM00)
KNNN
97.36
40.60
07.03
46.28
99.24
53.76
39.15
39.02
98.24
4506
32.19
92.00
1.00
99:35
KNNN20
97.56
4000
95.95
45.26
99.24
53.18
99.15
39:12
98,
41.54
82.19
91.03
1.00
99.55
KNNLo
97.56
39.24
96.99
45.20
99.24
$2.50
99.15
38.40
96.3
43.83
36.30
91.07
1.00
99.50
KNN
97.60
38.78
07.03
44.79
99.24
51.76
99.15
35.15
98.25
43.37
90.41
90.07
1.20
99:45
Pre-train 100% TTT 11P
RVL-CDIP (1T))
MSP
$4.28
$5.80
49.14
91.50
64:60
84.45
58.85
88.76
56.72
88.46
61.64
89.44
41.00
94.27
MaxLage
44.96
92:13
38.01
94.52
63:97
67.90
56.49
91.81
50.86
91.61
66.49
90.65
54.50
95.26
Faomyy
45.72
92.11
38.01
94.55
65.84
67.86
57.01
91.70
51.
91 16
72:60
90.41
14
95.14
(1099))
94.79
KNN
GndlNom
45.16
48.72
84.21
93.14
44.36
39.13
98.62
87.50
63.49
51.68
90.85
78.07
73.58
56.35
84.79
83.81
$3.99
51.30
83.64
60.37
50.68
93.09
K2.06
35,60
91.24
9K.04
KNN
44.88
93.14
36.64
95.04
53.35
90. 5g
74.27
83.67
$1.28
91.86
50
92.67
13.00
97.81
KNNuo
43.67
93.10
31.18
95
56.74
90.20
75.28
89.40
$1.72
91.89
57.53
92.21
15.60
97.45
KNN
43.63
93.15
27.52
95.94
58.74
90,02
76.18
88.35
51.52
91.87
61.64
92.00
18.50
97.18
Pre-train IFF-CDIP (nu 6ime-tume
KNN
07.04
42:35
03.97
50.17
97.41
51.63
98.01
23.19
96.6
47:10
12:35
07:47
1.10
93 38-38
KNN
07.16
41.99
94.01
49.96
97.81
53,01
38.00
42.73
96.77
46.67
15,07
96.05
3.00
98.31
KNNco
96.96
41.62
94,34
19.56
98.00
$1.20
98.05
64
46.16
2182
95.08
2.70
98.1
KNN
97.00
41.48
94.90
49.31
98
90.85
98.19
42.03
07.04
45:37
36.99
95.29
3.30
98.27
Table 8: OOD detection performance for document classification. Longformer.noe denotes the original model adopted from the Huggingface model hub. Longformer (+) denotes the additional pre-training on IIT-CDIP.
(nndumain)
0OD (out-dmmain)
Report
Average
Poster
Acc
AUROU
AUROCC
AUROOC
3573
05.T2
MastLogit
97.07
75.23
97.05
74.00
Energy
72.52
97:35
75.21
77.56
90.71
KNN,
$8.04
65.57
86.60
8588
62:TY
62:50
29.07
KNN
87.64
66.3
86
68.91
63.64
30.23
27
KNN
87.19
68.1
85.81
70.15
85.60
65.18
No KNN.,
KNN, 64.04
97.63
90.97
51.75
93
$3.16
96.
55.00
70.93
100.00
97.67
95.
70.01
100.00
KNN,
97.63
90.13
50.57
98.18
73.26
100.00
KNNN
97.92
$1.48
97.67
58.62
95.84
50.00
98.34
51.62
98.19
53.88
74.42
$5.14
100.00
61.20
Pie (Fall on ITF-CDPP
Rine cone 68
RVL- (DD)
MSP
03.20
64138
55.22
01.35
95.03
25.37
$9.25
03.13
95.34
02.00
90.70
07.26
50.00
33.32
MasLlgii
96.06
75.41
96.54
76.03
07.89
70.15
96.71
74.56
07.02
74.44
100,00
78.65
99.70
72.88
91.13
RANNY KNNN
Eaergy
97.25
53.73
75.40
39255
96.54
66.21
87.37
76.03
95.25
72.03
33.76
70.14
03.08
97.16
35.72
74.55
97.32
65.15
$7.22
74,03
100.00
15.34
$4.78
78.59
99.70
36,40
$7:34
72.86
KNN,
55.61
20.18
65.07
87.45
71.67
$3.60
63.30
85.61
64.91
67.23
45.84
94.63
55.30
87.70
KNNN
67.17
$8.56
66.97
87.29
72.83
53.47
65.53
83.35
66.70
57.01
55.81
94.25
35.20
$7.30
KNNNuo
61.73
89.79
66.93
87.11
73.30
$3.24
66.15
85.15
67.03
36.52
55.81
94.00
$4.70
W7.21
Pre-train on ITE-CIIP
Lno fin--lnne)
KNN
95.45
61.40
98.07
33.66
97.23
55.55
98.66
48.70
07.40
54.53
$1.40
91.12
97.40
46.27
KNN 96.66
60.02
97.05
52.95
97.40
$4.97
98.55
41.21
97.3 36
54.66
84
00.63
97. 27.50
45.5
KNN,, 05.60
39.04
97.05
51.77
97.41
53.07
98.62
47.5 47.20
07. 40
$1.54
85.21
80.05
98.70
44.15
KNN
05.60
$9.04
97.99
50.74
97.21
52,99
98.58
46.51
07.33
52.52
$5.37
80.52
98.50
43.09
12

---

Table 9: OOD detection performance for document classification. All models pre-trained on ImageNet.
0OD Bntasel (Iadhmmim)
GOD Taitwt (Cht--dumai))
ID
Method
Set. Report
Presentation
Form
Liiler
StL. Poster
Recript
TPR95
AUROC
FiRoos
AUROC
FIR99
AUROC
FPR95
AUROC
FFROS
AUROC
FPR99
AUROC
FPR95
AUROC
Imaggeett
CDP (ID)
NISP
63.49
$7.37
53.87
30.54
11:35
80387
05.22
36.75
51,15
63.10
90L35
MaxLogi
64.39
$7.97
81.87
53.95
57.50
41.86
38.00
43.29
67.09
65.03
93.65
KNNN
1031
9290
73.74
10.39
93.95
RosNet-3n
KNNGa
74.00
86.41
66.29
$7.70
93.05
76.51
6.08
KNNNa
86.04
79.22
77.54
73.57
90.33
KNNNo
51.14
94.62
48.58
KNNGo
KNNNo
97.99
train ImageNet
Energy
KNNNo
KNNNo
XNNN
51.16
ImageNNt
KNNNa
KNNGo
99.76
Pre-train on ImageNNe--
fime-
RVL-CDIPP (ID)
SISP
30.31
89.14
$2.19
91.30
67.38
91:26
39.90
85.77
35.10
33.49
47.07
92.98
39.30
91.95
MaxLogi
30.76
91.37
$1.60
93.73
68.01
35.15
91.81
54.64
50.97
10.70
94.20
32.40
13.15
Energy
$1.16
01.31
44.52
93.75
69.43
86.81
36.09
01.77
53,30
90.91
38.57
94.11
$5.20
93.11
94.38
KNNN
02.7
00112
37.75
50.91
33301
90.36
$4.30
$6.19
04.02
20,40
12.79
97.56
13000
U7.92
KNN
03.01
90.25
56.01
91.51
55.00
90002
81.38
86:01
54.63
15.12
92.76
14.90
97.67
KNNG
61.97
00.62
53.25
92.62
58.26
89.57
84.25
83.64
64.43
39.61
16.25
97.38
19.30
07.24
KNNN
00.29
9085
69.70
93.53
60.38
$9.07
$4.01
85.43
63.00
39.72
16.25
97.05
25.60
96.82
Pre-4rain on ImageNNt
KNN
98.48
52.15
05.02
56.94
99.48
53.77
99.47
$8.90
98.11
50.44
93.15
90.27
20.40
97.13
KNN
93.48
31.41
05.06
36.61
99.44
52.92
99.55
37.61
98.13
49.54
94..52
39.44
22.60
96.80
KNN
08.32
50.43
04.86
56.21
99.40
51.86
99.59
38.82
06 04
48.58
97.36
88.03
36.40
96.25
KNN
98.40
40.76
05.06
55.90
99.44
31.15
99.59
$4.99
98.12
47.85
98.63
87.24
31.20
95.76
13

---

Table 10: OOD detection performance for document classification (select OOD categories achieve the best
performance across most of the models with different modalities).
OOD Datanet (Indomalnn)
000 Dutaset (Out-domaln)
ID
Method
Densil
Ristme
Tile Toller
STL pubicalion
Arangy
Set. Poster
Receept
Axe
FPR95
AUROC
EPTERS
AUROCC
FPRUS
AUROC
TPR95
AUROC
FFRRS
ATTROC
TPR95
AUROC
ATTROC"
Pre-train
data
tune
RVL-CIPP
MSP
$6.12
50.57
71:72
M5W2
35.17
53.36
53.51
93.51
61.21
$1.55
10.57
92..0
69:99
99.31
66.57
95.80
73.66
95.47
66.31
67.09
65.67
96.
68.17
94.19
77.17
94.50
74.69
08:07
96.19
74.90
36.13
05:00
15.25
15.32
51.75
75.75
1502
BXS7
KNNNa
$4.50
K2.35
68.06
84.21
68.20
13.31
86.72
24.00
82.12
76.57
46.84
90.38
73.90
38.31
$4.58
84.06
85.05
82,25
87.44
83.80
88.25
83.06
77.24
78.20
W6.08
Pre-train
KNN
36.19
75.67
95:17
5K62
97.71
39.75
98.55
50.54
94147
61144
10:47
98.35
50.30
6301
60.35
12.79
61L59
KNNNs
97.83
94.77
59.
12.79
95.30
36.01
KNNNme
39.27
72.48
94.25
57.12
97.99
56.52
99.11
47.37
95. 16
58.37
11.63
97.89
94.300
$2.96
(rain
RYL-CPP (ID)
MSP
59114
00C31
35.72
35.12
96.81
37.03
15.35
593.0
3232
65.64
95.67
65.23
97.67
71.54
95.83
71.55
38.34
81.36
15:17
35.00
14.38
76.74
85.04
$1.50
77.53
82.92
86.49
69.26
82.92
78.29
92.62
H5.71
KNNNo
82.69
75.49
$4.21
74.86
82.
85.17
Pre-trsin
data
KNNNoo
05.72
50.09
07.07
46.5%
90.58
45.75
45.35
100.00
63.03
100.00
95.55
07.75
50.00
100.00
60.35
95.00
97.67
100.00
$8.89
MaxLogi
Emengy
87.5
65.25
RoNee-50 50
KNN
26.97
KNN
MaxLogi
Enerpy
91.25
KNNN1o
35.31
KNN2o
86
36.03
KNNNL
15:75
28.77
KNNN aou
t0
KNN
KNNNo
41.31
KNN uoe
09.88
30.97
94.39
09.03
64.67
91.97
3.70
98.81
Pre-rrin om Image Net
fine-tuee en RVL-CDI (ID)
85.51
20:44
93.10
K7.57
7597
50.30
MasLogi
10.91
88.
76.71
Energy
$1.79
90.89
55.00
05
17.53
96.53
92.69
79.20
54.27
88.81
79.45
85.01
50.10
93.20
$9.97
KNNN
34113
01.13
57.86
9118
55.49
37.40
97.35
53.93
51.53
85.9
42.47
95.00
11:00
KNN
90.99
9061
84
KNNco
50.55
95,08
53.33
90.79
01.95
92.32
9300
72.06
00.70
8h.00
42.47
94.74
17.30
97.12
KNN
50.5
90.81
54.12
90.55
30.79
95.78
95.08
75.39
58.15
88.1
45.21
94.24
22.00
98.5K
Pri-itii on Image
KNNN
59300
46.45
9938
25.30
58.65
50.61
98.72
46,10
89.21
32.49
87.67
90.39
19.30
97.25
KNNNo
66.00
99.03
25.00
37.30
91.01
98.72
46,16
52.29
50.41
90.87
21.70
97.01
KNNNo
24.55 76
55.16
91.52
96.76
52.1
99
24.30
98.62
KNN
99.80
45,33
99.03
24.45
54.81
91.90
98.72
43.10
58.24
52.19
95.39
89.51
24.50
96.27
14

---

Table 11: OOD detection performance for document classification (randomly select four categories as OOD).
OOD Datane (Indomain)
(OOD Dnitset (Chtt-lmaii))
10
Method
Letter
Handwritten
Adverthement
Memo
Asrrage
Scl. Ponter
Receipt
Avc
FFRRS
AUROC
TPRSS
AUTRCC
FFRUU5
AUROC
TPRE5
AUROC:
FPRUS
AUROC
FFRRU
AUROC
TPRSS
AUROCC"
Pre-train pure-fest
data
Tinr-uue
RVL-CDIP (TB)
MSP
70.22
79.21
50.33
85:24
61.64
67.80
91.42
57.59
74.10
13.06
95.25
59.75
95.30
55.12
Mantlogii
66.04
67.51
39.65
92.53
85.07
77.00
91.67
71.84
70.95
50.23
100.00
77.59
56.30
71.96
Energy
18.19
92.59
91.67
71.89
70.85
100,00
96.20
71.46
RoBER
KNNN
KNN
63.18
50.10
00.58
60.07
20300
21.17
75.00
80.00
65.72
07:20
68.55
71.22
71.25
75.04
94.19
51,75
91.20
50.30
12.15
81.89
KNNNa
57.30
71.00
76.34
68.38
70.82
74.94
94.19
91.30
81,09
79388
56.33
73.94
36.08
67.80
70.64
74.70
91.90
W0AT
Pre-train
data
ICSNG
59:30
58.11
21.79
96.1
97.29
53.39
14.51
39.20
12.79
$4.30
65.75
KNN
39.27
98.73
21.19
96.21
57.05
94.57
48.24
12.79
KNNN
98.77
20.45
96.89
54,12
98.30
55.35
95.19
46.96
13.05
97.60
$0.64
98.51
19.97
98.18
53.93
05.57
13.05
91.10
55.17
Pro-train
RVI--CDIP
65.96
77.93
81.92
60.80
90.21
54.23
72.02
65.66
$2.56
95.00
$0,90
MasLogit
Farrgy
57.35
45.61
65.04
77.37
Longformecr.co
92.08
30.77
71.38
73.72
66.35
75.19
61.21
90.00
71-76
77.12
51
KNN
61,39
79.36
52.75
73.61
51.76
73.91
70,90
62.79
75.29
pure dara
KNNN.
97.72
KNN
47.23
100.00
27.45
95.32
46.49
97:25
45.38
48.84
100.00
62.79
KNNG)
47:31
44.56
100000
KNN
Prostrain Image Nea lon-toue un RVT-CCMPP (1D)
MSP
80:88
64.29
85.58
65:33
$4.79
91.70
77.20
ManLogit
64.25
87.46
53.59
90.72
90.00
88.71
58.00
95.13
78.00
8686
Farrgy
62.65
$7.65
90.33
46.00
91.26
63.56
85:05
$0.57
52.83
$300
87.05
87.80
KNNG
00.99
79.37
90164
72.41
$6.30
59.17
81:74
77:25
2.74
19.70
93.70
RodNee-5n 511
KNN
78.00
47.47
92.61
68.27
58.42
80.23
2.74
93.08
KNN,,
75.96
28 44
49
65.65
92.78
77.91
70. 30
1:37
08.97
49.70
92:09
74.02
37:31
95.07
89.75 78
75.63
60.33 36
$3.60
2.74
98.67
$3.80
91:10
Pre-train on ImegeNNe
KNNNo
12.21
17:25
81:31
2787
91.16
59738
13.47
75.8%
61.50
80181
39.35
12.30
KNN
41:00
76 78
81.70
79.23
02:37
09.08
42.39
75.94
K3.56
14:10
97.97
KNNNg
95.55
29.53
76.28
81.81
31.01
92.05
59.12
80.80
76.32
63.55
K3.56
38.51
16.30
93.61
KNN
94.62
19
27.15
81.89
31.64
91.84
99.12
39.85
76.88
62.99
K3.56
$7.30
19.50
97.23
Pie Irain on ImtageNet Hoc-June on RVL
CDP (13)
MSP
61.95
$7.03
65:21
85.15
7335
70.75
61.40
85.45
66005
K3.14
64.05
74.34
49.20
MastLogit
$5.40
90.22
75.36
87.00
72.64
$4.36
44.22
93.01
62.18
53.62
72.60
$4.16
29.10
95.70
Eaergy
57.43
90.11
77.01
85.60
73.44
$4.17
43.78
93.06
62.92
$3.48
73,97
$4.25
28.00
95.69
92.42
KNND
6022
50.12
26.30
90176
43.85
50.15
47.87
02.87
56.12
50.58
43.41
54.58
7.201
95.35
am
KNN
61.32
90.01
61.37
91.31
48.83
90.33
49.00
92:52
55.13
91.04
50.14
95.56
8.50
98. 33
KNNG
62.22
39.78
56.44
91.55
50.34
39.55
48.52
92.22 30
54.35
90.50
26.03
95.72
11.30
97.97
KNN
62.62
$9.50
$4.08
91:85
50.70
$8.50 91
47.63
92. 18
53.98
50.64
30.14
95.54
13:50
97.65 97.66
Pre-train on Tme
KNN
95.15
45.57
$6.02
79. 64
$2.45
90,90
59.52
20
26
63.55
24,65
96.24
0.40
99 76
KNN
90.15
48.11
$6.30 89
80135 35
$3.48
92. 19
09.50
44.33 70
79.79
65.36
27.0
95.62
0.30
99.71
KNN, 90.23
42.30
87.99
81.66
36.76
91.59
99.50
43.07
80.00
64.68
43.84
94.57
0.50
99.63
KNN
19.19
41.46
$9.02
82.63
00.60
91.08
99.00
42.14
82.10
64.32
52.05
93:14
1.20
99.53
Pre tralm ImnogeNet Hoc--une on
RVL (1D)
MSV
09.65
$6.31
09107
87:85
72:25
30.78
00.38
16.01
70.21
53.52
07.12
$3.97
39.30
91.47
MaxLogit
67.35
99.20
08:40
88.55
69.35
$4.35
61.08
89.95
65.60
53.02
57.55
39.41
18.40
43.05
Eaergy
02.23
89.21
70.34
85.43
70.25
$4.37
60.75
90.03
65.89
$9.01
58.00
59.47
49.70
93.05
91.03
03.10
$3.39
$4.90
12.30
33.34
$9.08
$9.19
91.34
55.65
50.17
35.35
55322
1230
98.71
RNN
67.61
88.05
49.01
92.85
51.53
$9.25
58.59
91.16
$6.68
90.55
41.10
94.47
25.40
96.35
67.20
88.91
42.54
03.15
$3.95
$8.43
58.75
90.85
53.64
90.34
42.47
93.60
29.50
95.78
66.19
$8.50
43.50
93.19
55.71
$7.73
59.31
90.64
56.20
90.12
45.21
92.86
34.50
95.27
Pre-traim Image Net
RNN
95.00
41.98
90.96
77:15
34.87
90.60
99.40
41.21
81.03
62.76
$4.79
94.27
10.80
98.47
KNN
94.94
40.54
91.67
77.20
36,82
91.71
99.14
39.85
81.72
62.32
64.35
93.57
12.70
98.25
RNN 99.07
18.75
92.61
76.99
40.00
91.17
99.53
38.14
82.80
61.36
75.5
92.47
15:00
97.87
KNN 99.11
37.43
93.25
76.56
43.38
90.68
99.56
36.03
83.82
60.40
82.19 19
91.52
18.00
97.49
Table 12: OOD detection performance for document classification. All models are pre-trained on IIT-CDIP. For LayoutLM M models, we adopt the checkpoints from the Huggingface model hub. For UDoc, we pre-train the model on our side. All models are fine-tuned on RVL-CDIP ID data.
Dataset (Indomainn)
(OD Dataset
Method
Sri. Reppri
Proseiation
Sr5.
FFROS
AUROC
FFR1
AUROC
FPRRS
AUROC
FFR29
AUROC
FFIR95
AUROC
FPR95
AUROC
FPROS
AUROC
77.15
MaxLogit
27.06
92.38
37.95
15.65
36.65
901.5
24.42
94.96
57.30
56,70
Energy
27,06
92.40
37.97
15.00
35.92
91.23
36:05
50:55
24-42
94.977
57.30
56,70
KNN
31.21
17.44
07.00
KNN
21.74
95.93
36.20
41.42
91.12
32.45
17:44
96.82
51.70
KNN
24.34
38.25
43.93
60
33.64
35.04
46
06.44
$3.80
KNN
25.54
95.30
30.13
45.17
90.35
36.
96.24
$4.70
93.45
MSP
56.16
70.51
63.64
67.37
67.16
65.30
54,000
69.58
61.34
68 23
52.53
12.70
43.60
77.10
MixLogit
30.70
$9.17
40.42
$8.18
42.95
$4.09
35.12
88.22
16,90
87.
19:73
94,50
11.70
57.002
Energy
50.70
50.15
40.42
$8.18
43.98
$4.10
33.12
88.23
36. 80
$7.42
19:77
94.51
13.70
97.03
97.81
KNN
21.74
55:03
33.05
93.38
33,885
91.36
19.51
56.246
27.20
DA.13
11.03
97.35
5.90
57,577
KNNgo
22.74
94.90
16.56
93.20
33.96
91.56
19.54
96.15
28.22
93.94
12:79
97.44
10.00
97.89
KNN
24.62
94.63
34.57
92.71
35.83
91.38
21.63
05.93
30.11
93.66
13.05
97.30
10.70
97.72
KNN
25.22
94.38
30.29
92.52
36.55
91.09
22.48
95.79
30.53
93.40
16.25
97.04
11,80
97.55
SISP
56.13
63.73
09.43
64139
71.00
63:28
71.36
63.25
68.41
50.70
75.07
39.90
78.399
MasLegit
43.95
$2.12
41.21
36.19
49.64
83.16
19.39
83.13
48.
83.90
2.33
96.57
4.00
98.54
Energy
45.96
$2.12
47.21
$6.40
49.64
83.16
49.59
83.13
48.10
83.70
2.33
38,60
4.00
98.36
97.36
KNN
30.02
$4.27
41.21
13726
41.30
90139
16,35
92.28
17.35
91:50
1.15
99.13
3.30
98.11
KNN
31.10
94.36
41.96
$8.44
42,10
90.90
38.03
93.35
38. 30
91.76
1.16
59.04
6.00
98.32
KNNG
33.95
94.07
43.35
$7.50
44.01
90.72
40.71
93.06
40.51
91.43
1.16
98.84
7,40
98.26
KNN
34.83
93.84
41.75
$7.51
45.01
90.61
41.96
92.90
41.30
91.22
1.16
96.72
8.30
96.16
15


Reviews:
[{"review_id": "c2oX1Kb6KB", "reviewer": "Reviewer_M9aT", "summary": "", "questions": "A. Why the SOTA approach DocFormer [1] for RVL-CDIP is missed for comparison? \nB. What is the benefits of suggested approach with adapter compared to other multimodal approaches, i.e  LayoutLM v3? \nC. It is not quite obvious that the suggested zero-shot setup pre-training on the IIT-CDIP for vision models is really zero-shot. How can we guarantee that the four OOD classes (letter, form, scientific report, and presentation) are not presented on the documents in implicit form? D. D. It is not clear why multimodal models are omitted from the comparison in zero-shot setup?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments and valuable suggestions! We address concerns, questions, and comments below in detail.\n\n\n### Q: Baseline model selection\nIn this work, we categorize recent document pre-training models into four categories (see **L271-L277** in Page 4 and Appendix **A.2**): **(1)** Vision-only models, which exclusively leverage visual features; **(2)** Text-only models, relying solely on textual features; **(3)** Text+Layout models, which combine both visual and textual features; **(4)** Vision+Text+Layout models, integrating supplementary spatial information. For each category, we have selected several representative models to examine the influence of input modality on various OOD detection tasks. Specifically, we have considered ResNet-50, Swin Transformer, and ViT for category **(1)**; RoBERTa and LongFormer for category **(2)**; LayoutLMv1, Spatial-RoBERTa (developed by us) for category **(3)**; and LayoutLMv3 and UDoc for category **(4)**.\n\nWe acknowledge that, due to the extensive range of experiments, not every model in each category has been included. We also understand the importance of impactful models like DocFormer. Notably, its model structure shares similarities with LayoutLMv1/v2, both employing single-stream models, in contrast to the two-stream models (e.g., SelfDoc/UDoc). Relevant discussions on DocFormer can be found in **Section 2.1 (L145-L148)** within the related works.\n\nNevertheless, we have tried our best to reproduce DocFormer's results. However, it came to our attention that *the official implementation by the authors was not made available*. The closest available codebase we found was a third-party implementation [A]. Despite our endeavors using their codebase and checkpoints, regrettably, we were unable to replicate the results reported by DocFormer's authors in their paper within the limited timeframe for this response. After a thorough examination of the codebase and the original paper's descriptions, we identified discrepancies in training configurations. Specifically, the codebase is based on a subset from IDL (https://github.com/furkanbiten/idl_data), while the original DocFormer used a 5M subset from IIT-CDIP. Most notably, there exist differences in the pre-training tasks. The repository only contains Masked Language Modeling (MLM), while the original paper considers three pre-training tasks: Multi-Modal MLM, Learn To Reconstruct, and Text Describes Image [B]. Despite our earnest attempts to reproduce the original implementation within the rebuttal phase, considering the time constraints of extensive pre-training on IIT-CDIP, we plan to incorporate DocFormer in our subsequent manuscript revision.\n\nMoreover, following your advice to incorporate more baselines, we have added **DiT** [C] (w. pre-trained checkpoint from HuggingFace: https://huggingface.co/microsoft/dit-base), a self-supervised pre-trained document image transformer model that employs a large-scale dataset of unlabeled document images. We report the OOD detection performance below, after finetuning the model on RVL-CDIP (ID):\n\n| OOD Score      | Letter   |   --  |  --    | H.W. |  -- | --  | Adv. |  -- | --  | Memo | --  | --  | Average | -- | --  | Sci. Poster   |   --  |   --   | Receipt | --  | --  |\n|-------------|---------|---------|---------|---------|---------|--------|---------|---------|--------|---------|---------|--------|--------|----------|--------|---------|---------|---------|---------|---------|---------|\n| | FPR95   | AUROC   | AUPR    | FPR95 | AUROC | AUPR | FPR95 | AUROC | AUPR | FPR95 | AUROC | AUPR | FPR95 | AUROC | AUPR | FPR95   | AUROC   | AUPR    | FPR95 | AUROC | AUPR |\n| Energy      | 43.71   | 90.92   | 99.06   | 38.69   | 93.28   | 99.38  | 67.92   | 85.85   | 98.38  | 29.46   | 94.83   | 99.53  | 44.94  | 91.22    | 99.09  | 47.95   | 92.91   | 99.98   | 7.70    | 97.68   | 99.92   |\n| Maha   | 30.06   | 95.40   | 99.59   | 35.15   | 95.15   | 99.59  | 53.67   | 90.97   | 99.13  | 74.88   | 90.33   | 99.13  | 48.44  | 92.96    | 99.36  | 50.68   | 92.89   | 99.98   | 6.80    | 97.74   | 99.92   |\n| KNN10     | 27.22   | 95.65   | 99.55   | 30.86   | 95.36   | 99.54  | 41.06   | 92.84   | 99.25  | 47.24   | 93.58   | 99.40  | 36.60  | 94.36    | 99.44  | 63.01   | 70.49   | 99.78   | 8.50    | 97.72   | 99.90   |\n| KNN20      | 25.66   | 95.65   | 99.56   | 31.30   | 95.43   | 99.58  | 41.34   | 92.68   | 99.24  | 41.15   | 94.08   | 99.45  | 34.86  | 94.46    | 99.46  | 16.44   | 97.79   | 99.99   | 4.00    | 98.95   | 99.96   |\n| KNN50      | 25.46   | 95.71   | 99.61   | 30.37   | 95.57   | 99.63  | 42.94   | 92.49   | 99.29  | 35.23   | 94.45   | 99.48  | 33.50  | 94.56    | 99.50  | 16.44   | 97.72   | 99.99   | 5.60    | 98.83   | 99.96   |\n| KNN100     | 25.90   | 95.62   | 99.60   | 29.33   | 95.66   | 99.63  | 44.73   | 92.21   | 99.26  | 35.55   | 94.49   | 99.48  | 33.88  | 94.50    | 99.49  | 19.18   | 97.51   | 99.99   | 7.80    | 98.59   | 99.95   |\n\nRemarks: *Average* denotes the average for in-domain OOD datasets; KNN10 (20, 50, 100) denotes using the KNN+ score with K = 10 (K = 20, 50, 100); Adv. stands for Advertisement and H.W. stands for Handwritten.\n\n[A] Non-official implementation of DocFormer: https://github.com/shabie/docformer\n\n[B] Appalaraju et al., DocFormer: End-to-End Transformer for Document Understanding, ICCV 2021\n\n[C] Li et al., DiT: Self-supervised Pre-training for Document Image Transformer, ACM Multimedia 2022 \n\n### Q: Clarification on zero-shot OOD detection\n\nIn this paper, our experimental approach aligns with prior research on OOD detection using large-scale pre-trained models [A,B]: the task of OOD detection is defined with respect to *the downstream dataset (ID)*, instead of the pre-training data which is often hard to characterize (**L197 - L200, Page 3**). Furthermore, in the era of large-scale pre-training, identifying realistic OOD categories that aren't subtly connected to the pre-training data can be subjective and challenging.\n\nTo isolate the impact of OOD data from pre-training, we also curated a dataset (IIT-CDIP−) from IIT-CDIP by excluding potential OOD categories (detailed results can be found in **Table 4, 5, 10, 11** in the Appendix). We agree that due to the unlabeled and large-scale nature of the pre-training data, achieving a strict zero-shot setting is nearly impossible. To assess the influence of the pre-training dataset, we pre-trained models using 10, 20, 40, and 100% of randomly sampled data from IIT-CDIP−. Our analysis is provided in **Section 4 (Lines 405 - 416)**. We observed that the removal of pseudo OOD categories has a subtle impact on zero-shot OOD detection.\n\n \n[A] Ming et al., Delving into Out-of-Distribution Detection with Vision-Language Representations, NeurIPS 2022\n\n[B] Fort et al., Exploring the Limits of Out-of-Distribution Detection, NeurIPS 2021\n\n\n### Q: Spatial-aware adapter vs. other multimodal models\n\nGood question! The advantage of spatial-aware adapters lies in their ability to rapidly adapt existing well pre-trained language models for improved OOD detection. Concurrently, the spatial-aware adapter also leads to a noticeable increase in ID accuracy. As shown in Appendix Table 6, we can clearly observe this advantage (ID Acc): RoBERTa-Base (90.19), Spatial-RoBERT-Base (**97.11**), compared to LayoutLMv1 (97.28), UDoc (97.36), and LayoutLMv3 (97.81). Despite the small performance gap compared to document-oriented models such as LayoutLMv3, it's important to note that models with document-tailored pre-training tasks such as LayoutLMv3 pose inherent challenges when trying to leverage the benefits from other well-pretrained language models.\n\n\nDespite the rise of pre-trained LLMs, document-specific models lag notably behind the general-purpose counterparts. This disparity inspired our design of spatial-aware adapters for LMs. The lightweight design of such adaptors may further unleash the potential of future LLMs pretrained on diverse and abundant data. \nAdditionally, in contrast to multimodal models such as LayoutLMv3, the adapter acts as a streamlined add-on module, boosting computational efficiency while maintaining strong performance in both ID classification and OOD detection. \n\nBelow, we present a comparison between the parameters of our spatial-aware adapter and other models. It's evident that our approach allows for optimizing fewer parameters based on pre-trained LMs, thereby achieving a balance between the training cost and effectiveness.\n\n| Model Name           | Total Params (M)    | Word Embed. Layer Params (M) | Spatial-Aware Adapter Params (M) | Adapter/Total Params (%) |\n| :---                 | :---                 | :---                 | :---                 | :---                 |\n| RoBERTa$_\\text{Base}$| 124.65               | 38.60                | -                 | -               |\n| Longformer$_\\text{Base}$| 148.66               | 38.60                | -                  | -                  |\n| LayoutLMv1$_\\text{Base}$| 112.63               | 23.44                | -                | -                  |\n| LayoutLMv3$_\\text{Base}$| 125.33               | 23.44                | -                | -                  |\n| Spatial-RoBERTa$_\\text{Base}$ | 125.17               | 38.60                | 0.52                 | 0.42                 |\n| Spatial-Longformer$_\\text{Base}$ | 149.19               | 38.60                | 0.52                 | 0.35                 |\n\n### Q: Include AUPR and Mahalanobis score\n\nThanks for the suggestion! We have revised the manuscript (Appendix D) and included the AUPR metric for completeness. Some key results are attached below as well. \n\nWe acknowledge that, as highlighted by the reviewer, the Mahalanobis score has demonstrated promising OOD detection performance in the NLP domain. However, the arrangement of text blocks in documents is not solely sequential but also spatial. Moreover, documents frequently incorporate a wealth of visual elements. It remains underexplored how logit-based and feature-based OOD detection methods address the unique blend of visual, textual, and layout information in documents. Our paper's primary goal is to systematically explore these complexities and provide deeper insights.\n\nIn this work, we mainly use the KNN+ score as a representative feature-based OOD detection score. Compared to the Mahalanobis score, KNN+ is non-parametric (without assuming features follow class-conditional Gaussian), computationally efficient with the FAISS library (https://github.com/facebookresearch/faiss),  and has shown competitive performance compared to the Mahalanobis score with pre-trained models [A]. \n\n\nIn the initial version of the manuscript, we have included the results with Mahalanobis score in Appendix due to numerically unstable results when calculating the inverse of the covariance matrix for high-dimensional features. We genuinely appreciate your feedback and have meticulously reviewed our implementation. Our updated manuscript now employs a pseudo-inverse (using torch.pinv) for calculating the Mahalanobis score, resulting in numerically stable outcomes. Some key results are also attached below: \n\n**[1] LayoutLMv1**\n\n| OOD Score      | Letter   |   --  |  --    | H.W. |  -- | --  | Adv. |  -- | --  | Memo | --  | --  | Average | -- | --  | Sci. Poster   |   --  |   --   | Receipt | --  | --  |\n|-------------|---------|---------|---------|---------|---------|--------|---------|---------|--------|---------|---------|--------|--------|----------|--------|---------|---------|---------|---------|---------|---------|\n| | FPR95   | AUROC   | AUPR    | FPR95 | AUROC | AUPR | FPR95 | AUROC | AUPR | FPR95 | AUROC | AUPR | FPR95 | AUROC | AUPR | FPR95   | AUROC   | AUPR    | FPR95 | AUROC | AUPR |\n| Energy|27.06 | 92.40 | 99.21 | 37.97 | 91.54 | 99.15 | 45.65 | 88.36 | 98.79 | 35.92 | 91.23 | 99.09 | 36.65 | 90.88 | 99.06 | 24.42 | 94.96 | 99.98 | 57.30 | 86.70 | 99.47 |\n| Maha|17.85 | 96.66 | 99.70 | 33.91 | 94.24 | 99.48 | 36.99 | 92.46 | 99.23 | 24.88 | 95.31 | 99.57 | 28.41 | 94.67 | 99.50 | 24.42 | 94.97 | 99.98 | 57.30 | 86.70 | 99.47 |\n| KNN10 | 20.82 | 96.09 | 99.61 | 35.32 | 93.82 | 99.39 | 40.06 | 91.34 | 99.03 | 28.65 | 94.80 | 99.52 | 31.21 | 94.01 | 99.39 | 20.93 | 90.94 | 99.94 | 25.80 | 93.06 | 99.64 |\n| KNN20 | 21.74 | 95.93 | 99.61 | 36.20 | 93.77 | 99.43 | 41.42 | 91.12 | 99.03 | 30.44 | 94.61 | 99.50 | 32.45 | 93.86 | 99.39 | 17.44 | 97.00 | 99.99 | 49.80 | 93.92 | 99.78 |\n| KNN50 | 24.34 | 95.56 | 99.60 | 38.25 | 93.41 | 99.40 | 43.93 | 90.69 | 99.02 | 33.64 | 94.19 | 99.46 | 35.04 | 93.46 | 99.37 | 17.44 | 96.82 | 99.99 | 51.70 | 93.73 | 99.77 |\n| KNN100 | 25.58 | 95.30 | 99.57 | 39.13 | 93.20 | 99.38 | 45.17 | 90.35 | 98.98 | 34.78 | 93.99 | 99.45 | 36.17 | 93.21 | 99.34 | 23.26 | 96.44 | 99.99 | 53.80 | 93.70 | 99.78 |\n\n**[2] LayoutLMv3**\n| OOD Score      | Letter   |   --  |  --    | H.W. |  -- | --  | Adv. |  -- | --  | Memo | --  | --  | Average | -- | --  | Sci. Poster   |   --  |   --   | Receipt | --  | --  |\n|-------------|---------|---------|---------|---------|---------|--------|---------|---------|--------|---------|---------|--------|--------|----------|--------|---------|---------|---------|---------|---------|---------|\n| | FPR95   | AUROC   | AUPR    | FPR95 | AUROC | AUPR | FPR95 | AUROC | AUPR | FPR95 | AUROC | AUPR | FPR95 | AUROC | AUPR | FPR95   | AUROC   | AUPR    | FPR95 | AUROC | AUPR |\n| Energy|30.70 | 89.18 | 98.79 | 40.42 | 88.18 | 98.76 | 42.98 | 84.10 | 98.14 | 33.12 | 88.23 | 98.57 | 36.80 | 87.42 | 98.56 | 19.77 | 94.50 | 99.98 | 11.70 | 97.02 | 99.90 |\n| Maha| 20.98 | 95.28 | 99.55 | 35.72 | 93.41 | 99.39 | 30.57 | 92.70 | 99.23 | 17.33 | 96.31 | 99.67 | 26.15 | 94.42 | 99.46 | 19.77 | 94.51 | 99.98 | 11.70 | 97.03 | 99.90 |\n| KNN10| 21.74 | 95.03 | 99.48 | 35.68 | 93.38 | 99.33 | 32.88 | 91.86 | 99.05 | 18.51 | 96.26 | 99.67 | 27.20 | 94.13 | 99.38 | 51.16 | 66.13 | 99.68 | 34.40 | 86.03 | 98.94 |\n| KNN20    | 22.74 | 94.90 | 99.49 | 36.56 | 93.20 | 99.37 | 33.96 | 91.66 | 99.04 | 19.64 | 96.15 | 99.66 | 28.22 | 93.98 | 99.39 | 11.63 | 97.58 | 99.99 | 8.90  | 97.97 | 99.93 |\n| KNN50      | 24.62 | 94.62 | 99.48 | 38.37 | 92.71 | 99.32 | 35.83 | 91.38 | 99.06 | 21.63 | 95.93 | 99.64 | 30.11 | 93.66 | 99.38 | 12.79 | 97.44 | 99.99 | 10.00 | 97.89 | 99.93 |\n| KNN100      | 25.22 | 94.38 | 99.45 | 39.29 | 92.32 | 99.28 | 36.55 | 91.09 | 99.03 | 22.48 | 95.79 | 99.63 | 30.88 | 93.40 | 99.35 | 13.95 | 97.20 | 99.99 | 10.70 | 97.72 | 99.92 |\n\n**[3] UDoc**\n\n|  OOD Score     | Letter   |   --  |  --    | H.W. |  -- | --  | Adv. |  -- | --  | Memo | --  | --  | Average | -- | --  | Sci. Poster   |   --  |   --   | Receipt | --  | --  |\n|-------------|---------|---------|---------|---------|---------|--------|---------|---------|--------|---------|---------|--------|--------|----------|--------|---------|---------|---------|---------|---------|---------|\n| | FPR95   | AUROC   | AUPR    | FPR95 | AUROC | AUPR | FPR95 | AUROC | AUPR | FPR95 | AUROC | AUPR | FPR95 | AUROC | AUPR | FPR95   | AUROC   | AUPR    | FPR95 | AUROC | AUPR |\n| Energy| 45.96 | 82.12 | 97.82 | 47.21 | 86.40 | 98.58 | 49.64 | 83.16 | 98.14 | 49.59 | 83.13 | 98.08 | 48.10 | 83.70 | 98.15 | 2.33  | 98.57 | 100.00| 4.00  | 98.34 | 99.94  |\n| Maha | 25.38 | 94.71 | 99.49 | 39.61 | 88.87 | 98.84 | 42.18 | 90.45 | 99.01 | 34.74 | 93.53 | 99.40 | 35.48 | 91.89 | 99.18 | 2.33  | 98.60 | 100.00| 4.00  | 98.36 | 99.94  |\n| KNN10| 30.02 | 94.47 | 99.45 | 41.22 | 88.66 | 98.77 | 41.90 | 90.99 | 99.02 | 36.65 | 93.48 | 99.41 | 37.45 | 91.90 | 99.16 | 41.86 | 82.13 | 99.84 | 48.60 | 76.17 | 97.34  |\n| KNN20| 31.10 | 94.36 | 99.46 | 41.98 | 88.44 | 98.78 | 42.10 | 90.90 | 99.03 | 38.03 | 93.35 | 99.40 | 38.30 | 91.76 | 99.17 | 1.16  | 99.13 | 100.00| 5.50  | 98.42 | 99.95  |\n| KNN50| 33.95 | 94.07 | 99.45 | 43.35 | 87.89 | 98.70 | 44.01 | 90.72 | 99.05 | 40.71 | 93.06 | 99.37 | 40.51 | 91.43 | 99.14 | 1.16  | 99.04 | 100.00| 6.90  | 98.32 | 99.94  |\n| KNN100| 34.83 | 93.84 | 99.43 | 43.75 | 87.51 | 98.65 | 45.01 | 90.61 | 99.04 | 41.96 | 92.90 | 99.36 | 41.39 | 91.22 | 99.12 | 1.16  | 98.84 | 100.00| 7.40  | 98.26 | 99.94  |\n\n**[5]  Spatial-RoBERTa$_\\text{Large}$**\n| OOD Score      | Letter   |   --  |  --    | H.W. |  -- | --  | Adv. |  -- | --  | Memo | --  | --  | Average | -- | --  | Sci. Poster   |   --  |   --   | Receipt | --  | --  |\n|-------------|---------|---------|---------|---------|---------|--------|---------|---------|--------|---------|---------|--------|--------|----------|--------|---------|---------|---------|---------|---------|---------|\n| | FPR95   | AUROC   | AUPR    | FPR95 | AUROC | AUPR | FPR95 | AUROC | AUPR | FPR95 | AUROC | AUPR | FPR95 | AUROC | AUPR | FPR95   | AUROC   | AUPR    | FPR95 | AUROC | AUPR |\n| Energy| 33.39 | 90.16 | 98.89 | 39.25 | 89.88 | 98.84 | 42.30 | 88.13 | 98.65 | 37.05 | 91.66 | 99.10 | 38.00 | 89.96 | 98.87 | 31.40 | 92.42 | 99.97 | 27.70 | 94.22 | 99.78 |\n| Maha | 26.74 | 94.45 | 99.46 | 42.19 | 92.21 | 99.24 | 36.51 | 92.21 | 99.17 | 27.44 | 95.14 | 99.55 | 33.22 | 93.50 | 99.36 | 24.42 | 96.47 | 99.99 | 18.30 | 96.34 | 99.88 |\n| KNN10| 28.18 | 94.47 | 99.43 | 42.43 | 93.01 | 99.30 | 37.43 | 91.74 | 99.06 | 31.13 | 94.72 | 99.53 | 34.79 | 93.49 | 99.33 | 25.58 | 96.24 | 99.99 | 18.60 | 96.28 | 99.87 |\n| KNN20| 28.78 | 94.32 | 99.42 | 42.43 | 92.90 | 99.29 | 38.07 | 91.58 | 99.05 | 32.02 | 94.55 | 99.51 | 35.33 | 93.34 | 99.32 | 25.58 | 96.02 | 99.99 | 18.60 | 96.33 | 99.88 |\n| KNN50| 30.22 | 93.95 | 99.42 | 43.71 | 92.69 | 99.32 | 40.06 | 91.26 | 99.07 | 34.54 | 94.10 | 99.47 | 37.13 | 93.00 | 99.32 | 26.74 | 95.52 | 99.99 | 21.40 | 96.14 | 99.87 |\n| KNN100| 30.86 | 93.71 | 99.39 | 44.11 | 92.56 | 99.30 | 40.66 | 91.05 | 99.05 | 35.47 | 93.88 | 99.45 | 37.78 | 92.80 | 99.30 | 26.74 | 95.22 | 99.99 | 21.70 | 96.11 | 99.87 |\n\nRemarks: *Average* denotes the average for in-domain OOD datasets; KNN10 (20, 50, 100) denotes using the KNN+ score with K = 10 (K = 20, 50, 100); Adv. stands for Advertisement and H.W. stands for Handwritten.\n\n[A] Sun et al., Out-of-Distribution Detection with Deep Nearest Neighbors, ICML 2022\n\n### Q: Clarification on multimodal models in the zero-shot setting\n\nDue to the space constraints, we have included the zero-shot results on multimodal models in the Appendix (LayoutLMv1 in **Table 5**, p22 and Spatial-RoBERTa in **Table 6**). As shown in **Figure 2(a)**, Multimodal models are primarily categorized into single-stream and two-stream architectures. In our case, we opted for the single-stream variant (LayoutLMv1/v3) due to their incorporation of the CLS feature. However, when considering two-stream multimodal models, it's worth noting that during pretraining, there exist separate heads for visual and textual features. Consequently, such structure poses challenges for zero-shot scenarios as it prevents the use of feature-based and logit-based OOD detection methods. Instead, the feasible approach is to employ these methods after completing fine-tuning and multimodal feature fusion for two-stream models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The suggested adapter approach looks promising and seems beneficial in some scenarios\n2) The extensive analysis of the OOD method for the documents scenario is provided.", "reasons_to_reject": "1) Baseline selection (missing DocFormer from comparison) and non-obvious comparison of suggested approach with others.\n2) The zero-shot setup is not clear for vision models and should be clarified.\n3) From the OOD point-of-view there is no evidence that the suggested approach is better, better to provide additional metrics to prove it.\n4) In the recent OOD works the distance-based methods with Mahalanobis distance show great performance both for text and vision tasks, but this comparison is not provided. See [2],[3],[4]", "questions_to_authors": "A. Why the SOTA approach DocFormer [1] for RVL-CDIP is missed for comparison? \nB. What is the benefits of suggested approach with adapter compared to other multimodal approaches, i.e  LayoutLM v3? \nC. It is not quite obvious that the suggested zero-shot setup pre-training on the IIT-CDIP for vision models is really zero-shot. How can we guarantee that the four OOD classes (letter, form, scientific report, and presentation) are not presented on the documents in implicit form? D. D. It is not clear why multimodal models are omitted from the comparison in zero-shot setup?", "paper_topic_and_main_contributions": "The paper investigates how spatial information influence on for document out-of-distribution(OOD) detection. To support the claim the authors suggest using the spatial-aware adapter. The extensive experiments on RVL-CDIP (with OOD samples added from NJU-Fudan and CORD datasets) datasets are provided in different setups. More precisely the paper compares vision-only, text-only, text+layout, and vision+text+layout models. The suggested adapter approach with the RoBERTa model overperforms the baseline RoBERTa model and matches the performance of LayoutLM.\n\nThis work is well-written and effectively conveys the main idea. It gives an overview and comparison of existing approaches for document OOD detection. The suggested adapter approach looks promising for incorporating spatial information inside the RoBERTa model. This paper could be beneficial for the researchers in narrow field of OOD for documents.\n\nHowever, there are several concerns I would like to discuss. \n1) First, it is not clear how the selection of baselines is done and why the SOTA approach DocFormer [1] for RVL-CDIP is missed for comparison. To prove that the suggested approach is comparable with LayoutLM v3 it is better to provide a side-by-side comparison. Also, I want to see more discussions on which setup of the proposed adapter approach is better than existing multi-modal approaches.\n2) It is not quite obvious that the suggested zero-shot setup pre-training on the IIT-CDIP for vision models is really zero-shot. How can we guarantee that the four OOD classes (letter, form, scientific report, and presentation) are not presented on the documents in implicit form? Again it is not clear why multimodal models are omitted from the comparison\n3) In the field of OOD it is common to use AUPR_OOD in case of imbalance classes and FPR with the decision threshold to prove the method's ability to keep good in-domain quality with a given threshold. It will better reflect the OOD ability of the methods.\n4) In the recent OOD works the distance-based methods with Mahalanobis distance show great performance both for text and vision tasks, but this comparison is not provided. See [2],[3],[4]\n\nStrengths:\n\n1) The suggested adapter approach looks promising and seems beneficial in some scenarios\n2) The extensive analysis of the OOD method for the documents scenario is provided.\n\nWeaknesses:\n\n1) Baseline selection (missing DocFormer from comparison) and non-obvious comparison of suggested approach with others.\n2) The zero-shot setup is not clear for vision models and should be clarified.\n3) From the OOD point-of-view there is no evidence that the suggested approach is better, better to provide additional metrics to prove it.\n4) In the recent OOD works the distance-based methods with Mahalanobis distance show great performance both for text and vision tasks, but this comparison is not provided. See [2],[3],[4]\n\n [1] Appalaraju, Srikar, et al. \"Docformer: End-to-end transformer for document understanding.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.\n[2] Kimin Lee et. al. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NEURIPS 2018\n[3] Podolskiy, A.V., et al. Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detection. in AAAI 2021\n[4]  Yiyou Sun et al. Out-of-distribution detection with deep nearest neighbors. In ICML 2022", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The Figure 2b is very difficult to follow"}, "extra_text": {"Missing_References": "Kimin Lee et. al. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NEURIPS 2018\nPodolskiy, A.V., et al. Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detection. in AAAI 2021\nYiyou Sun et al. Out-of-distribution detection with deep nearest neighbors. In ICML 2022"}}, {"review_id": "ne8ul14QTv", "reviewer": "Reviewer_Rxki", "summary": "", "questions": "1. Is it possible that the ID data and OOD data you used are too far so that using spatial information + Roberta can be on pair with the complicated multi-modal method?\n2. If the vision model is good enough, I believe there is no need to do the OCR and then extract the textual information. For example, in GPT4 technical report, GPT4 can easily understand the image input with both picture and text. Why using the image feature alone can not do well in document OOD detection? Your Resnet and VIT baseline seems weak to me. Have you tried other advanced vision pre-training models?\n3. In the era of the large model, the training set is massive online data and it is hard to tell which is ID and which is OOD, what is the future direction of OOD detection?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments and valuable suggestions! We address concerns, questions, and comments below in detail.\n\n\n### Q: Are OOD data far from ID data?\n\nWe acknowledge that currently there exists no standard benchmark for document OOD detection, given the scarcity of document data, especially for cross-domain document classification datasets. RVL-CDIP stands out as a prominent dataset, and notably, its image data consists of black-and-white or scanned documents. \n\nIn this work, we consider both in-domain OOD and out-domain OOD data for RVL-CDIP. While the out-domain OOD datasets are semantically distinct from ID categories (e.g., colorful vs. black-and-white), the in-domain OOD data resembles ID data in both layout and contents (see Figure 3, top row vs. bottom row). We provide further qualitative and quantitative evidence below.\n\n**(1)** Qualitatively, we visualize the pre-trained and fine-tuned features for ID, in-domain OOD, and out-domain OOD data (w. ViT and RoBERTa) in **Fig 10**. We can clearly see that while out-domain OOD are far from ID data, the four in-domain OOD categories are close to ID clusters in the feature space.\n\n**(2)** Quantitatively, we measure the distance between each OOD dataset and ID data via Optimal Transport (https://github.com/microsoft/otdd) and visualize the results in **Fig 9a** (page 13). We can see that the distance between out-domain OOD and ID data is significantly (around 10x) larger than in-domain OOD vs. ID data.\n\nThe construction of in-domain OOD categories follows practical considerations where OOD categories are the ones multiple models (ViT, RoBERTa, UDoc, LayoutLMv1, and LayoutLMv3, etc.) perform poorly on. Therefore, we may expect the models to detect such inputs as OOD instead of assigning a specific ID class with low confidence. (L332-L335 in page 5)\n\nMoreover, we consider two additional in-domain OOD dataset selection schemes: (1) select 4 classes out of 16 from RVL-CDIP as OOD where the model performs well; (2) randomly select 4 classes as OOD. The OT distances are visualized in Fig 9b and Fig 9c. We can see that the distance between ID and in-domain OOD is similar to the original scheme (Fig 9a). This suggests that most in-domain OOD categories are not far from ID data. \n\nWhile this paper represents an initial endeavor, we hope that our work will serve as a stepping stone towards constructing more comprehensive and diverse OOD benchmarks in the document domain, akin to those available in the NLP and natural image domain.\n\n### Q: Experiment with advanced vision pre-training models\n\nGreat question! In fact, we have explored recent well-established image-based pretraining methods such as MIM and MAE to enhance our image encoder. However, our exploration revealed that the impact of natural image pretraining methods on document image data is comparatively modest. We outline several reasons for this outcome:\n\n**(1)** Document images possess sparsity, which implies that not every pixel holds meaningful information. For instance, in scientific papers or resumes, the pixels containing texts constitute a small fraction, while most pixels represent the white background. Consequently, traditional methods like MIM or MAE with random masking struggle to adapt effectively to document images.\n\n**(2)** Document images are inherently multimodal, differing from the conventional image-caption pairs. The multimodal nature of document data signifies that text and image modalities coexist within the same entity. For instance, both textual contents and figures may exist in a single paragraph. This introduces challenges in designing pretraining tasks.\n\nNotably, we have included a recent work DiT [1], an image-based pretraining method that draws inspiration from image tokenization and MIM techniques. We acquired the DiT base model from HuggingFace and performed fine-tuning on RVL-CDIP (ID), achieving an accuracy of 96.52%. (In comparison, the Swin-based model achieved an accuracy of 95.74%, as shown in **Table 9** in the Appendix.) The OOD detection results are also provided below. Comparing the table below with Appendix **Table 12**, we can see that DiT achieves better results (vs. ViT/ResNet) with the KNN-based OOD score. This also highlights the significance of document-specific pre-training in OOD detection.\n\n| OOD Score      | Letter   |   --  |  --    | H.W. |  -- | --  | Adv. |  -- | --  | Memo | --  | --  | Average | -- | --  | Sci. Poster   |   --  |   --   | Receipt | --  | --  |\n|-------------|---------|---------|---------|---------|---------|--------|---------|---------|--------|---------|---------|--------|--------|----------|--------|---------|---------|---------|---------|---------|---------|\n| | FPR95   | AUROC   | AUPR    | FPR95 | AUROC | AUPR | FPR95 | AUROC | AUPR | FPR95 | AUROC | AUPR | FPR95 | AUROC | AUPR | FPR95   | AUROC   | AUPR    | FPR95 | AUROC | AUPR |\n| Energy      | 43.71   | 90.92   | 99.06   | 38.69   | 93.28   | 99.38  | 67.92   | 85.85   | 98.38  | 29.46   | 94.83   | 99.53  | 44.94  | 91.22    | 99.09  | 47.95   | 92.91   | 99.98   | 7.70    | 97.68   | 99.92   |\n| Maha   | 30.06   | 95.40   | 99.59   | 35.15   | 95.15   | 99.59  | 53.67   | 90.97   | 99.13  | 74.88   | 90.33   | 99.13  | 48.44  | 92.96    | 99.36  | 50.68   | 92.89   | 99.98   | 6.80    | 97.74   | 99.92   |\n| KNN10      | 27.22   | 95.65   | 99.55   | 30.86   | 95.36   | 99.54  | 41.06   | 92.84   | 99.25  | 47.24   | 93.58   | 99.40  | 36.60  | 94.36    | 99.44  | 63.01   | 70.49   | 99.78   | 8.50    | 97.72   | 99.90   |\n| KNN20      | 25.66   | 95.65   | 99.56   | 31.30   | 95.43   | 99.58  | 41.34   | 92.68   | 99.24  | 41.15   | 94.08   | 99.45  | 34.86  | 94.46    | 99.46  | 16.44   | 97.79   | 99.99   | 4.00    | 98.95   | 99.96   |\n| KNN50      | 25.46   | 95.71   | 99.61   | 30.37   | 95.57   | 99.63  | 42.94   | 92.49   | 99.29  | 35.23   | 94.45   | 99.48  | 33.50  | 94.56    | 99.50  | 16.44   | 97.72   | 99.99   | 5.60    | 98.83   | 99.96   |\n| KNN100     | 25.90   | 95.62   | 99.60   | 29.33   | 95.66   | 99.63  | 44.73   | 92.21   | 99.26  | 35.55   | 94.49   | 99.48  | 33.88  | 94.50    | 99.49  | 19.18   | 97.51   | 99.99   | 7.80    | 98.59   | 99.95   |\n\nRemarks: *Average* denotes the average for in-domain OOD datasets; KNN10 (20, 50, 100) denotes using the KNN+ score with K = 10 (K = 20, 50, 100); Adv. stands for Advertisement and H.W. stands for Handwritten.\n\n[1] Li et al., DiT: Self-supervised Pre-training for Document Image Transformer, ACM Multimedia 2022\n\n### Q: The future direction of OOD detection in the era of large models\n\nIndeed, in the era of large-scale pre-training, it is challenging to identify OOD categories not semantically related to the pre-training data. Therefore, in this work and previous works on OOD detection with pre-trained models, OOD is defined with respect to the downstream dataset (ID), instead of the pre-training data which is often hard to characterize (L197 - L200, page 3).  While our work aims to establish an in-depth and extensive study of document OOD detection, we believe large models open up new opportunities for OOD detection and highlight several promising avenues for future research:\n\n**(1)** Develop computationally efficient adaptation (fine-tuning) algorithms to safeguard pre-trained large models against OOD data for a wider range of downstream tasks. In this paper, we notice that retraining large models is both time-consuming and costly. Additionally, OOD scenarios are often task-specific. Therefore, similar to our spatial-aware adapter, exploring how to introduce additional modality information, such as spatial data, into well-pretrained models such as Llama2 [1], while minimizing resource consumption, poses an intriguing question.\n\n\n**(2)** Provide theoretical understanding on how pre-training impacts downstream OOD detection. Despite extensive empirical studies presented in our work, it remains underexplored how various aspects of pre-training (e.g., pre-training objectives, the diversity of pre-training data) impact downstream OOD detection theoretically. A novel theoretical framework would be helpful to further guide the design of practical algorithms.\n\n**(3)** Developing new OOD benchmarks become increasingly vital for pre-trained large models. Merely having large models doesn't ensure they've encountered the diverse range of data users might interact with, especially in specialized areas such as documents. Notably, datasets like Common Crawl may have inherent biases from the crawling process, limiting their coverage of diverse documents (PDFs/images/...). Additionally, documents are created by humans, resulting in long-tailed distributions in the pre-training dataset. Even advanced models like GPT-4 can't be presumed to be adept at handling every document type. Therefore, addressing OOD reliability for large models necessitates revisiting data collection, aiming to construct a more comprehensive and effective OOD benchmark for evaluating the OOD capabilities of pre-trained large models.\n\n\n[1] Touvron et al., Llama 2: Open Foundation and Fine-Tuned Chat Models, Arxiv 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well-written and easy to follow.\n2. The authors provide a comprehensive empirical study on document OOD detection, with extensive analysis and experimental results., which could provide useful information to people in this field.", "reasons_to_reject": "1. This paper is too empirical and some of the findings are intuitive, although it is acceptable for empirical NLP conferences.\n2. The evaluation of their proposed adapter is not convincing. The baseline method (such as Roberta) is not strong, since it is intuitive that adding multi-modal information can help. And as I mentioned before, adding spatial information can be on par with the multimodal method could be due to the ID and OOD data being too far away from each other.", "questions_to_authors": "1. Is it possible that the ID data and OOD data you used are too far so that using spatial information + Roberta can be on pair with the complicated multi-modal method?\n2. If the vision model is good enough, I believe there is no need to do the OCR and then extract the textual information. For example, in GPT4 technical report, GPT4 can easily understand the image input with both picture and text. Why using the image feature alone can not do well in document OOD detection? Your Resnet and VIT baseline seems weak to me. Have you tried other advanced vision pre-training models?\n3. In the era of the large model, the training set is massive online data and it is hard to tell which is ID and which is OOD, what is the future direction of OOD detection?", "paper_topic_and_main_contributions": "Paper summary\nDocument OOD detection is a practical direction. However, previous works on document OOD detection either only using single-modality rather than multi-modality information, which lost some of the information from the document, or using too large models, which leads to high computational costs.  This paper first provides an extensive study on the impacts of pre-training, fine-tuning, scoring methods and model-modality on the performance of document OOD detection. They highlight several findings, such as pre-training helps, fine-tuning helps, distance-based OOD scoring method outperforms logit-based scoring method and spacial information is important. Based on these observations, they propose a spatial-aware adapter on Roberta model, which can outperform the vanilla Roberta model and is on par with multi-modal document OOD detection methods.\n\nPresentation: Good\nThis paper is well-written and easy to follow.\n\nNovelty: Fair\nFor the method, the authors proposed an adapter to encode the spatial information, which is new but intuitive. The paper is more like an engineering paper.\nFor the findings, the authors provide several findings, but most of the findings are intuitive.\n\nSoundness: Fair\nThe technical part looks sound to me, but I have a concern about the evaluation part. The authors construct the ID and OOD data on their own to evaluate their method. I understand it may be the common practice in the OOD detection field but is it possible that the ID data and OOD data they used are too far so that using spatial information + Roberta can be on pair with complicated multi-modal method?\n\nSignificance: Fair\nThe proposed method can outperform Roberta, a pure-text model, and be on par with the multi-modal methods.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No", "Typos_Grammar_Style_and_Presentation_Improvements": "No"}, "extra_text": {}}, {"review_id": "JAWWntmAhd", "reviewer": "Reviewer_os2G", "summary": "", "questions": "A. Given the wealth of world knowledge encoded in large language models (LLMs), is it possible to integrate LLM knowledge and spatial-awareness for better document understanding?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful comments and support of our work! We address concerns, questions, and comments below in detail.\n\n\n### Q: The difference between In-domain and Out-domain OOD, and the motivation of ID/OOD dataset selection\n\n\nWe highlight the key differences between in-domain and out-domain OOD as follows:\n\n**(1)** Visually, out-domain OOD data are semantically distinct from ID classes, the in-domain OOD datasets resemble ID data in both layout and contents. As shown in **Fig 3**, out-domain OOD data such as scientific posters and receipts are colorful, while in-domain OOD data (letter, form, presentation, and scientific report) are black and white, more resembling ID data.\n\n**(2)** In terms of feature representations, we visualize the pre-trained and fine-tuned features for ID, in-domain OOD, and out-domain OOD data (w. ViT and RoBERTa) in **Fig 10**. We can also see that while out-domain OOD are far from ID data, the four in-domain OOD categories are close to ID clusters in the feature space.\n\n**(3)** Quantitatively, we measure the distance between each OOD dataset and ID data via Optimal Transport (https://github.com/microsoft/otdd) and visualize the results in **Fig 9a** (Page 13). We can see that the distance between out-domain OOD and ID data is significantly (around 10x) larger than in-domain OOD vs. ID data.\n\n\n**Motivation of ID/OOD dataset selection:** We curate both the ID dataset and in-domain OOD datasets from the same RVL-CDIP dataset. This construction follows practical considerations where OOD categories are the ones multiple models (ViT, RoBERTa, UDoc, LayoutLMv1, and LayoutLMv3, etc.) perform poorly on, as shown in **Fig 2 (b)**.  Therefore, we may expect the models to detect such inputs as OOD instead of assigning a specific ID class with low confidence. Additionally, we also incorporated two other OOD category selection criteria in the Appendix (Appendix A.1, Table 10, and Table 11).\n\nOn the other hand, we construct out-domain OOD datasets to mimic scenarios in the open world where the test inputs can have significantly different color schemes and layouts compared to ID samples. This results in a substantial distinction between ID and out-domain OOD datasets in the feature space (see **Fig. 9** in Appendix).\n\n### Q: Is it possible to integrate LLM knowledge and spatial awareness for better document understanding?\n\nThanks for the insightful comments. LLMs excel in comprehending textual content owing to their extensive exposure to diverse text data. However, they often lack spatial awareness, a critical element in understanding structured documents. As demonstrated in this paper, the incorporation of a spatial-aware adapter significantly enhances both in-domain and out-of-domain OOD performance in comparison to text-only models. Exploring effective algorithms and modules that can extract structured knowledge from LLMs to augment document understanding is a promising avenue for future research.\n\nHowever, concurrently, integrating spatial-aware information into generative models (e.g., LLama2/GPT/...) would likely require a different approach than what is presented in our work, such as the Spatial-aware RoBERTa with Masked Language Modeling. Existing LLMs are predominantly built upon Causal Language Modeling, which implies a sequence-based order among words. Yet, transitioning from a 1D sequence to a 2D spatial layout introduces a more complex order that extends beyond a simple left-to-right structure. Therefore, it's essential to consider not only the forward prediction but also incorporate spatial-aware Causal Language Modeling, particularly for documents with a 2D layout.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper provides comprehensive and in-depth analysis for document OOD detection, which is supported by extensive experimental results.\n2. An add-on module is designed for transformer-based models, which has practical implications.", "reasons_to_reject": "1. It would be better to elaborate on the difference between In-domain OOD and Out-domain OOD, and the motivation of ID/OOD dataset selection.\n2. The authors mainly focus on the analysis of different aspects in document OOD detection, some are intuitive. Introducing more ID tasks (e.g., document-based question answering) other than classification could further enhance the understanding.", "questions_to_authors": "A. Given the wealth of world knowledge encoded in large language models (LLMs), is it possible to integrate LLM knowledge and spatial-awareness for better document understanding?", "paper_topic_and_main_contributions": "This paper presents a comprehensive analysis of out-of-distribution (OOD) detection for document understanding models. The authors investigate the effects of model modality, pre-training, and fine-tuning on OOD detection performance. It finds that spatial information is critical for document OOD detection and proposes an add-on spatial-aware adapter to improve OOD detection performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] Hao Lang, Yinhe Zheng, Yixuan Li, Jian Sun, Fei Huang, and Yongbin Li. 2023. A survey on out-of-distribution detection in nlp."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an extensive and in-depth investigation into the influence of pre-training, fine-tuning, model-modality, and OOD scoring functions across a wide range of document OOD detection tasks. To enhance OOD detection performance, the authors introduce a spatial-aware adapter. The paper's well-structured presentation effectively communicates the central concepts, and the experiments conducted are comprehensive.

All reviewers agree that this paper offers a thorough and extensive empirical analysis of document OOD detection, yielding valuable insights substantiated by robust experimental results. In light of this, the reviewers find the paper sound and express enthusiasm for the practical implications of the proposed add-on spatial-aware adapter and the depth of the analysis.

As a result, the final decision for the paper is acceptance.
Decision: Accept-Findings