A Critical Analysis of Out-of-Distribution Detection for Document Understanding
Authors: Jiuxiang Gu
Yifei Ming
Yi Zhou
Jason Kuen
Vlad I Morariu
Handong Zhao
Ruiyi Zhang
Nikolaos Barmpalios
Anqi Liu
Yixuan Li
Tong Sun
Ani Nenkova
Venue: Submitted to ICLR 2023 (2023)
Keywords: Document Understanding
Pretraining
Out-of-Distribution
Document intelligence
Robustness

Abstract: Large-scale pretraining is widely used in recent document understanding models. During deployment, one may expect that large-scale pretrained models should trigger a conservative fallback policy when encountering out-of-distribution (OOD) samples, which suggests the importance of OOD detection. However, most existing OOD detection methods focus on single-modal inputs such as images or texts. While documents are multi-modal in nature, it is underexplored if and how multi-modal information in documents can be exploited for OOD detection. In this work, we first provide a systematic and in-depth analysis on OOD detection for document understanding models. We study the effects of model modality, pretraining, and finetuning across various types of OOD inputs. In particular, we find that spatial information is critical for document OOD detection. To better exploit spatial information, we propose a simple yet effective special-aware adapter, which serves as an add-on module to adapt transformer-based language models to document domain. Extensive experiments show that our method consistently improves ID accuracy and OOD detection performance compared to baselines. We hope our findings can help inspire future works on understanding OOD robustness for documents.

Under review conference paper at ICLR 2023
A CRITICAL ANALYSIS OF OUT-OF-DISTRIBUTION DETECTION FOR DOCUMENT UNDERSTANDING
Anonymous authors Paper under double-bling review
ABSTRACT
Large-scale pretraining widely used in recent document understanding models. During deployment, one may expect that large-scale pretrained models should trigger a conser- vative fallback policy when encountering out-of-distribution (OOD) samples, which sug- gests the importance of OOD detection. However, most existing OOD detection methods focus on single-modal inputs such as images or texts. While documents are multi-modal in nature, it is underexplored if and how multi-modal information in documents exploited for OOD detection. In this work, we first provide a systematic and in-depth analysis OOD detection for document understanding models. We study the effects of model modality, pretraining, and finetuning across various types of OOD inputs. In par- ticular, we find that spatial information is critical for document OOD detection. To better exploit spatial information, we propose simple yet effective spatial-awar adapter, which serves as an add-on module to adapt transformer-based language models to document do- main. Extensive experiments show that our method consistently improves ID) accuracy and OOD detection performance compared to baselines. We hope our findings can help inspire future works on understanding OOD robustness for documents.
be
1 INTRODUCTION
The recent success of large-scale pretrained models has led to the widespread deployment of deep models in various applications. In the document domain, model predictions increasingly used to help humans make decisions important applications ranging from tax form processing, machine learning assistant medical reports analysis, deep analyses from financial forms, etc. However, most cases, models are pretrained on collected data but are then deployed in an environment with a different distribution over the observed data (Cui et al. 202T). For example, with the outbreak of COVID-19 (Velavan & Meyer, 2020), machine- assisted medical document analysis systems have to face continually changing data distributions. This mo- tivates the need for reliable methods in the document domain to detect
(OOD) inputs.
The goal of OOD detection is to categorize in-distribution (ID) test samples into one of the known categories and detect instances that do not belong to any known classes (Huang Li, 202T Bendale Boult, 2016). Generally, model is optimized on a particular task (e.g., image classification (Deng et 2009)), and a companion OOD detector is built as a safeguard for the classifier. Recently, large-scale pretrained models have demonstrated promising results in multiple domains (Dosovitskiy al., 2021 Hendrycks et al. 2020) as pretraining enables models to learn powerful and transferable feature representations (Radford et al., 2021). In particular, the models obtained by finetuning large-scale pretrained models are significantly better at OOD detection even with a simple distance metric (Lee et al. 2018 Radford et al. 2021).
It is underexplored whether existing OOD detection methods that demonstrate success for images text can be naturally extended to documents. The main challenges posed in document OOD detection stem from the fact that document understanding is inherently multi-modal, thus, it is suboptimal to rely single

---

Under review conference paper at ICLR 2023
modality. The majority of recent OOD detection approaches focus on single-modal learning (Hsu al., 202Ta Jin et al. (022), and they are not compatible with document un- derstanding tasks which require multi-modal learning. The spatial relationship of text blocks in documents further differentiated the document multimodal learning from the multimodal learning in vision-language domain (Lu et al., 2019 Li et al., 2020). In addition, recent document pretraining methods have demon- strated remarkable performance on various downstream document understanding tasks (Xu et al.] 2020 2021b Huang et al. 2022, Li et al. 2021; Cui et al., [022]] Hong et al., 2022 Gu et al.) 2022] Wang et al., 2022 However, existing pretraining datasets for documents are limited and lack diversity, in sharp contrast
2020 Zhou et al., 202T Xu et al.
to common pre-trainining datasets for natural images. Therefore, it is not obvious which OOD detection methods are reliable in the document domain and how pretraining impacts OOD robustness.
This paper investigates the OOD robustness in the document domain through the following questions: (!) Are pretrained models robust OOD examples? Is further pretraining beneficial? How do the pretraining data and tasks affect the performance? (2) How does multimodality (textual, visual, and spatial) affect OOD robustness? (3) Are existing OOD detection methods developed for natural images and texts transferrable to documents? We present a large-scale evaluation of recent approaches. We focus on models pretrained on different data types and evaluate them on a diverse range of document understanding benchmarks across visual, textual, and spatial modalities. Our key contributions are summarized as follows:
We show that pretraining datasets and tasks significantly impact OOD detection performance. Through extensive pretraining and finetuning experiments, we find that higher finetuning performance on ID data does not usually translate to better performance on OOD data. This observation emphasizes the importance of considering metries beyond ID performance for measuring model reliability.
We propose a spatial-awar adapter, which can serve as an add-on module to transformer-basee models and learn the spatial-aware representation. Our method can easily transfer the pretrained language models to the document domain. Extensive experiments show that our method can consistently improve ID accuracy and OOD detection performance across broad spectrum of datasets.
We show that recent conclusions drawn from OOD detection methods are valid for images and texts but do not always transfer to documents. For a wide range of document models, we observe that OOD samples are easier to identify in the feature space than in the logit space.
The rest of the paper is organized follows. Sec. provides the preliminaries and related works. In Sec. 3 we provide a comprehensive analysis of OOD robustness for document models and conclude in Sec.
2 PRELIMINARIES AND RELATED WORKS
2.1 DOCUMENT MODELS AND PRETRAINING
Large-scale pretrained models have attracted lot of attention in the document domain. In vision or natural language processing (NLP) tasks, pretraining has shown great success in producing generic representations that learn from large-scale unlabeled corpora (Devlin et al.) 2018 Lu et al. 2019 Su et al. 2019 He et al.) 2020 Document pretraining also seeks to find universal representations suitable for any downstream task. However, the unique characteristics of document images distinguish document pretraining works from pre- vious ones in vision or language domains. For documents, the contents are spatially distributed, and visual and textual information co-occurs within the semantic regions. In contrast, inputs in the language domain are pure texts, and inputs in the vasion-language domain are image-text pairs.
Recent document pretraining models differ in architecture and objectives, as depicted in Fig. 2 Lay- outLM (Xu et al. 2020) extends BERT to learn contextualized word representations for document images through multi-task learning. It takes sequence of Optical Character Recognition (OCR) (Smith 2007)
2

---

Under review conference paper at ICLR 2023
Out-af-littrburicon Detection
besef Scree Farction
h(x) Clacscey
Pretwaing lekt
Pvetrabsing
Gy (g(2))
Dunlabel
h(x)
8(2)
Met brie
Dosance bused Scoee Fisentoon
DRn
Dut
h(x))
g(c) (Clostee)
Finnttnning
Gz(
Gy
Threshld 7
Figure/ 1: Schematic description of OOD detection for document classification. The left part shows the pretraining and finetuning pipelines. During inference time, for a given input document image, we calculate the OOD detection score G. according to different methods (logit-base or distance-base). The OOD detector will identify the input document as OOD if the OOD score is smaller than the threshold value r..
words and word bounding boxes as inputs during pretraining and finetuning. LayoutL (Xu et al. 2021b) improves on the prior work by including an image encoder pretraining and training them jointly. Like Lay- outLMy2, DocFormer (Appalaraju et al. 2021 also adopts CNN model to extract image grid features. It fuses the spatial information as an inductive bias for the self- elffaatentionn module. The latest version, Lay- outLMv3 (Huang et al. 2022), shares similar ideas as LayoutLMM2 and further enhances the visual and spatial characteristies by introducing two other tasks: masked image modeling and word-patch alignment.
Another line of works for document pretraining focuses on different granularities of document images and takes region-level text blocks as the basic input elements, such as SelfDoe (Li et al.] (02T) and UDoe (Gu ct al., 202Ib. The pretraining tasks of SclfDoc and UDoc are based on feature space. They adopt cross- modal encoder to model the relationship between visual and textual features. Instead of using the spatial information the input layer, SelfDoc and UDoc encode the 2D spatial information with linear mapping and fuse the position embeddings at the output layer of the image encoder and sentence encoder. Despite the promising performance of those pretrained models on downstream applications, it remains largely derexplored whether recent document pretraining models are robust to various types of OOD data, the role of pretraining and finetuning, and the key factors for document OOD detection.
2.2 OUT-OF-DISTRIBUTIIO DETECTION
Many OOD detection methods have been proposed for deep models, including generative model-based meth- ods (Ge et al., 2017 Oza & Patel 2019; Nalisnick et al.] 2019 Ren et al. 2019 Xiao et al. [2020] Morteza & Li 2022), , and discriminative-mon model based methods. For the latter category, an OOD scoring function can be derived based on the softmax output or logit space (Liu et al. 2020: Hsu et al. 2020 Huang & Li| 2021 Liang et al. 2018 Sun et al.) (202T). gradient information (Huang et 2021). or the feature space (Sastry & Oore 2020 Sehwag et al. 2021 Winkens et al. 2020 Sun et al. (0222) of a classifier, Despite their impres- sive performance, most of the scores are developed for natural images and text inputs. A recent work (Larson ct al. 2022) studies OOD detection performance for documents, but only explores limited number of mod- els and OOD detection methods. Furthermore, the relationship between pretraining, finetuning, and spatial information is underexplored. In this work, we provide Amer-grained and comprehensive analysis and hope to shed light on the key factors of OOD robustness for documents.
Notations We denote the input and label space in and yin {1. (1...., (.....,), K). respectively. Let Din {(xt", ) denote an ID dataset, where æ E X is the input feature vector, and n E y denotes the semantic label for K-way classification. Let Dout = {(0g"t, y } denote an OOD test set where you E yout, and yout n yam 0. OOD detection can be formulated as binary classification problem, which aims to distinguish between ID and OOD data. We express the neural network model f := g o h as a composition of feature extractor h X -> ddd and classifier g Bd - RKK,, which maps the feature
3

---

Under review conference paper at ICLR 2023
embedding of an input to K real-valued numbers known logits. During inference time, OOD detection can be performed by exercising thresholding mechanism Gz(z)) ) = [{(()) 2 ? where by convention samples with higher scores ((x) are classified as ID and vice versa. The threshold "y is typically chosen so that high fraction of ID data (e.g., 95%) is correctly classified.
We group OOD detection methods into two major categories: logit-based scores are derived from the logit layer of the model, while distance-based methods are directly based on the feature embedding layer, as shown Fig. We describe few popular OOD detection methods for each category as follows.
Logit-based: Maximum Softmax Probability (MSP) score (Hendrycks & Gimpel 2017) SMSP max4e[K] edi(w)/ - efx(2) naturally arises as classic baseline since logits can be converted to a categorical distribution p(y]=); Energy score (Liu et al. 2020): SEnergy = log Σ ef((*) utilizes the Helmholtz free energy of the data and theoretically aligns with the logarithm of the ID density; MaxLogit score (Hendrycks et al. 2022): SMasloggt [ (xx removes the softmax function in MSP and demonstrates promusing performance on large-scale natural image datasets recently.
Distance-based: eestance-based methods directly leverage feature embeddings h based on the idea that OOD inputs are relatively far away from ID centroids or prototypes. Depending on the distributional assumption of feature embeddings, methods can be characterized as 1) parametric methods such as Ma- halanobis score (Lee et al.) 2018;; Sehwag et al. [22T]) which assumes ID embeddings follow class- conditional Gaussian distributions and use Mahalanobis distance from the ID centroid the distance metric; 2) non-parametric methods such as KNN+ (Sun et al. 20022) which uses cosine similarity as the distance metric.
Evaluation Metrics To evaluate OOD detection performance, we adopt two commonly used met- rics (Hendrycks & Gimpel| (017): Area Under the Receiver Operating Characteristic (AUROC) and False Positive Rate at 95% Recall (FPR95). For ID test sets, we report Accuracy (Acc), Fl score, and Mean Average Precision (mAP).
3 ANALYZING OOD ROBUSTNESS FOR DOCUMENT MODELS
In this section, we consider the task of document classification, where models are expected to classify documents into categories such as scientific papers, resumes, efc. However, it is underexplored whether models are robust OOD samples at test time. Most document classification datasets exist in the form of images (Harley et al. 2015). Usually, the first step is to pass the image through an OCR system to obtain a set of text blocks along with their coordinates in the image. Given the input image, extracted words, and coordinates, models can utilize single-modal or multi-modal information to classify the document.
Models Fig. (a) shows common structures for document image pretraining and classification model According to the input modalities, we categorize them into the following groups:
(I) Vision-based: Since current document datasets exist as images, can treat document classification the standard image classification problem. In our experiments, we consider ResNet-50 (He et al., (016) and VIT (Fort et al. 2 as exemplar document image classification models. As for pretrained weights, we consider two settings: pretrained on ImageNet (Deng et al.) (009) and further pretrained on IIT-CDIP (Lewis) jet al. 2006). We adopt masked image modeling (MIM) for image pretraining with mask ratio of 0.6. Note that the document classification dataset we used in this paper, RVL-CDIP, is subset of II-CDIP. Hence, unless otherwise specified, the IIT-CDIP pretraining data used in this paper excludes RVL-CDIP.
See Appendix 1.1.2 for further details about the models and hyperparameters.
4

---

Under review conference paper at ICLR 2023
Voina detic eri
Bncaker
Nurtinddal Sacader terting pat
Huttrnadl Enroder
Vinet
Tonsal Excader
Spaid Aanver
Bansf
Winds
Words
Wonds Bodee
(a) Illustration of common structures for document pre- trainingcclassficatinn.
(b) A detailed comparison of per-category accuracy on the RVI.-CDIP test set.
Figure 2: (Left) Illustration of models for document pretraining and classification. Our proposed spatial- aware pretraining and finetuning models are the network architectures in green blocks. We also show the modality information on top of each architecture. (Right): Evaluating finetuning performance for document classification of pretrained models. We group models into three groups (from left to right): language-only, vision-only, and multimodal, For each group, we also present the performance of a model another group (shown in grey) for better reference. The average accuracy for each model is shown in the parenthesis.
(2) Text-basee text-based: Alternatvelyy, we can define the classification as a text classification problem since docu- ments typically contain words. In our experiments, we consider RoBERTa (Liu et al. 2019) the backbone and append a classifier for finetuning. Since some documents such as scientific papers consist of sentences with more than 512 tokens, we also consider Longformer (Beltagy al. 2020), which can handle max- imum of 4,096 input tokens. Similar to the vision-based models, we further pretrain the language models with masked language modeling (MLM) on IIT-CDIP extracted text corpus.
(3) Text+ Spatial:: Layout information plays a crucial role in the document domain. As shown in Fig. 3 a document is composed of words or images with some specific layouts. To investigate the effect of layout information, we adopt LayoutLM as a baseline. It is specifically designed for documents and trained on the full IIT-CDIP data. Inspired by the promising OOD detection performance of spatial-aware models (Sec (3.3] and the recent advances in adapter-based transformers (Pfeiffer et al.] 2020), we propose new spatial-aware adaptor, a small learned module that can be inserted within a pretrained language model. Besides the simplicity, our adapter is competitive for both ID classification and OOD detection (Sec 3.4
(4) Visual+ Trextull+Spatide Current state-of-the-art methods tailored to documents consider various input gramularity and modality and utilize textual, visual, and spatial information for document tasks. Despite the promising performance, such models are large in size and computationally heavy. We select two represen- tative models to evaluate upon: LayoutLMv3 and UDoc. For fair comparison, both models are pretrained on full IIT-CDIP.
Constructing ID and OOD Datasets We construct ID datasets from RVL-CDIP (Harley et al. 2015). Specifically, we specify 12 out of 16 classes as ID classes. For OOD datasets, we consider two scenarios:
(I) In-domain OOD: To determine the OOD categories, we extensively analyze the performance of recent document classification models. Fig. 2(b shows a detailed comparison of per-category accuracy on the RVL-CDIP test set. Naturally, for the classes the model performs poorly on, we may expect models to detect such inputs OOD instead of assigning a specific ID class with low confidence. We observe that the 4 categories letter. letter.form, scientific report, presentation result in the worst performance across most of models with different modality, which we use as OOD categories and construct the OOD datasets accordingly. The ID dataset constructed from the remaining 12 categories. We refer to these OOD datasets as in-domain, they are also constructed from RVL-CDIP.
5

---

Under review conference paper at ICLR 2023
Blke jolder
afvershammi
budget
handeritee
spectfication
POLLTE
ATA
Indistribuionn Calngurin
lenee
farse
presensation
sclendff repart
scientlic posks
recdit
distribution Categorres (le-Domalss
distribution Catogories (Cue-Domain)
Figure 3: (Top) Examples of ID inputs sampled from RVL-CDIP (top). (Bottom) In-domain OOD from RVL-CDIP, and out-domain OOD from Scientific Poster and Receipts.
the open-world setting, test inputs can have significantly different color schemes and layouts compared to II) samples. To mimic such scenarios, we use two public datasets as the out-domain OOD test sets. Specifically, NJU-Fudan Paper-Poster Dataset (Qiang et al. 2019) contains scientific posters in the digital PDF format, and we extract the document contents with CORD (Park et al. 2019 is receipt understanding dataset that contains significantly different inputs from that in RVL-CDIP. As shown in Fig. 3 document images in CORD are receipt images without creases or warping, which requires the model to be capable of handling text information but also visual and spatial information.
(2) Out-domain OOD:
In the following sections, provide detailed analysis and share insights various aspects of OOD de- tection performance for document understanding models under different OOD detection methods. Further details on the setup are provided in Appendix A
3.1 ARE PRETRAINED MODELS SUFFICIENT FOR OOD DETECTION?
As shown in Sec. 2.1 most domain processing models deployed in the real world are pretrained large- scale dataset. Naturall, one may expect pre-trained models to be robust to OOD data when equipped with competitive OOD detection methods. To better understand the role of pretraining, we first provide more nuanced discussions on the following questions: 1) Are models equally robust to in-domain and out-domain OOD inputs? 2) How does model modality impact OOD detection performance?
We consider a wide range of models pretrained on pure-text/image data (e.g.. ImageNet, Wikipdeia, etc). A detailed description of these models be found in Appendix AA.12 During finetuning, we combine the pretrained model with classifier and finetune on RVL-CDIP (ID). For models before and after finetuning, we extract the final feature outputs as the feature embeddings and the same KNN+ score (Sun et al. 2022) for OOD detection. The results are shown in Figure 4 We observe the following trends. First, finetuning largely improves OOD detection performance for both in-domain and out-domain OOD data. Pretrained models, despite the fact that they have "seen" a diverse collection of data during pretraining, do not yield sufficient OOD robustness. The same trend holds broadly across models with different modalities. Second, the improvement of finetuning is less significant for out-domain OOD data. For example, the AUROC on Receipt (out-domain OOD) for pretrained ViT model 13, whereas finetuning only improves by 0.79%. This suggests that pretrained models do have the potential to separate data from different domains due to the diversity of data used for pretraining, while it remains hard for pretrained models to perform finer grained separation for in in-domain OOD inputs. Therefore, finetuning is beneficial for improving both types of OOD detection performance as a consequence of improved feature representation.
6

---

Under review conference paper at ICLR 2023
100
In-domuta oop dIVL-CDIE, Anarugel
Out-domala oop Paslas
Postact
Out-pomata oop (heccipt)
Firticed
Eestald
RodNeeto
Switles
Wffuer
RoBERTaaee
Lanthrmmnttt
Restitte
Swittout
Wia
RaDERTT
Langformrrant
Restittte
Switteet
V7
ReBERTaan
Longformeraa
Figure OOD detection performance for pretrained models w. and w.o. finetuning based on distance-based score KNN+(K=10). Finetuning significantly improves performance for both in and out-domain OOD.
Da dww.dadd 1500 festom
Chak domasta UNID debeclicn
Lo disnaa Adeccioa
Dal desalt ont detedine
160
074
Wi Ohcpet Iodliiite
Lever Milna (hipp
(a) Pretrain on ITT-CDIP.
(b) Analysis of IIT-CDIP.
(c) Pretrain on ITT-CDIP".
Figure 5: The impact of pretraining data on zero-shot OOD detection performance. IIT-CDIP denotes the filtered pretraining data after removing the "OOD" categories.
To make the analysis more thorough, we have two additional in-domain OOD settings: (1) select the classes the model performs well on, as in-domain OOD categories; (2) randomly select classes as OOD categories. As shown in Appendix (Table T0 and Table II can that finetuning also improves both types of OOD detection, which further reaffirm our conclusion. We also visualize the optimal transport dataset dis- tance (OTDD) (Alvarez-Melis & Fusil 2020) between in-domain and out-domain OOD datasets in Appendix (Fig. 10(b) and Fig. T0(c). refer to the Appendix for more details.
3.2
THE IMPACT OF PRETRAINING DATA ON ZERO-SHOT OOD DETECTION
In the previous section, we analyze the impacts of finetuning for OOD detection where the pretraining dataset is fixed and unrelated documents. Next, we dive deeper and study the impacts of pretraining dataset on zero-shot OOD detection. For each model, we adopt the same pretraining objective while adjusting the amount of pretraining data. Specifically, we increase the data diversity by appending 10, 20. 40, and 100% of randomly sampled data from IIT-CDIP dataset (around 1 IM) and pretrain each model. After pretraining, we measure OOD detection performance with KNN+ score based on feature embeddings.
For out-domain OOD data (Fig. 5 right), increasing the amount of pretraining data can significantly improve the zero-shot OOD detection performance (w.o. finetuning) for models across different modalities. This fur- ther verifies our previous hypothesis that pretraining with diverse data is beneficial for coarse-grained OOD detection, such as inputs from different domains (e.g., color schemes). On the other hand, for in-domain OOD inputs, even increasing the amount of pretraining data by over 40% provides negligible improvements (Fig. 5 left). This also suggests the necessity finetuning for improving in-domain OOD detection.
We further explore zero-shot OOD detection by removing the potential OOD categories from ITT-CDIP. In practice, we first adopt the LayoutLMvl finetuned on RVL-CDIP as the classifier for predicting labels for all IIT-CDIP document images. Fig. 5(b) shows the distribution of the predicted classes on IIT CDP.... Next,
7

---

Under review conference paper at ICLR 2023
remove the "OOD" categories from the IIT-CDIP data and pretrain two models (RoBERTa and Lay with 10, 20, 40, and 100% of randomly sampled data from the filtered IIT-CDIP. Fig. 5(e) shows our zero- shot OOD performance. Note that we do not show 0% in Fig. [ since we pretrain LayoutL from scratch. For RoBERTa, we start from the public pretrained model and see similar trend in Fig. 5 the inffuence of pretraining for those well-pretrained language models is minor for in-domain OOD detection since there is considerable gap between OCR words and pure-text data. E.g., words in a document are spatially arranged, while words in text corpus are arranged sequentially. In contrast, pretraining data has a bigger impact those models trained from scratch the zero-shot performance of LayoutLMv increases when more pretraining data added. We provide more details the Appendix (Table and Table
106
In dedainnn OOD (Aeg)
Outt -er---mmnnnn OOD (hug)
Perryy
Figure 6: Comparison between representative feature-based based scores and logit-based scores for spatial-aware and non-spatial-awe -wwrr models. Spatial-awar -aware models are colored in blue.
3.3 INVESTIGATING SPATIAL-AWARE MODELS FOR OOD DETECTION
In previous sections, mainly focus mainstream text-based and vision-based models to analyze the effects of finetuning and pretraining on in- and out-domain OOD detection. Next, we largely expand the scope of our study by incorporating models tailored to document processing, which we refer aware models, such as I., LayoutLMMv. and UDoc. Moreover, given finetuned models, we are able compare the performance of logit-based scores and distance- baaae scores. Some key comparisons shown in Figure 6 Please refer to the Appendix for full results.
spatial-
Distance-based vs. logit-based score We can see that simple KNN-based score outperforms logit-based scores for both in-domain and out-domain OOD data. The trend holds consistently across mod- els with different modalities. More- over, spatial-aware models demonstrate both stronger OOD detection performance for in and out-domain OOD. For example, with the best scoring function (KNN+), compared to RoBERTa, LayoutLMv3 im- proves the average AUROC by 7.09% for out-domain OOD and 7.54%. The signif- icant improvement suggests the value of spatial and visual information in improving OOD robustness for document data. Note that despite this paper mainly comparing the logit-based and distance-based scores, need be aware that Gradient-based score has also been proposed for OOD detection. We also report the GradNorm (Huang al. 2021 OOD detection score in Appendix A.3 and achieves similar performance as logit-based scores.
Daunder
Leaguge
Essodr
Fomee Dokethig
Primaining
Finmtining
Figure 7: Illustration of our spatial-aware adapter design for language models. We have two adapter designs (marked in red box): (1) insert the adapter into the word embedding layer during pretraining and finetuning; (2) insert the adapter into the output layer for finetuning only. For the first design, we freeze the word embedding layer and learn the adapter and transformer layers.
8

---

Under review conference paper at ICLR 2023
3.4 TOWARDS SIMPLE AND EFFECTIVE SPATIAL-AWARE ADAPTORS
Spatial-aware models tailored for documents such LayoutLM rely spatial information and demonstrates superior OOD detection performance. This brings us a question: given the abundance of well-pretrained large-scale language models on text data such as RoBERTa, is there a simple and effective method that allows us to exploit the pretrained language model to document inputs for effective OOD detection? Next, we show that by enhancing transformer- based pretrained models with a spatial spatial-aware aware adapter module, we can achieve good performance with minimal code edits.
Spatial-aware adapter Given public pretrained RoBERTa model, depending on the position of the adaptor, we consider two architectures: 1) appending the adapter to the word embedding layer, denoted as Spatial- RoBERTa (pre). requires pretraining and finetuning, as illustrated in the top row in Fig. 7 2) appending the adapter to the final layer, denoted as Spatial-BoBERR encoder, it only requires finetuning (illustrated the bottom row). In the following, we only discuss Spatial- RoBERTa (pre). Full results for both Spatial-Ro RoBERTa variants are in the Appendix.
(post), As the model can utilize pretrained textual
We freeze the word embedding layer during pretraining for the following considerations: 1) word embed- dings learned from large-scale corpus already cover most of those words from documents; 2) pretraining on documents without strong language dependency may not help improve word embeddings. For example, in semi-structured documents (e.g-, forms, receipts), language dependencies are not be strong rich-text documents (e.g., letters, resumes), which may degenerate the learned word representations. In practice, each word has normalized bounding box (x0, y0, x1, y)), where (x0, y0) / (x1, yl) corresponds the position of the upper left / lower right in the bounding box. Each coordinate is fed into an embedding layer and outputs position embedding. All position embeddings are added to the initial word embedding to form a new spatial-an spatial-aware aware embedding.
Spatial-RoBERII significantly outperforms RoBERTa To verify the effectiveness of Pratial-RoBERIE
We compare OOD detection performance for pre-trained and fine-tuned models. The results are shown in Fig. Spatial-RoBERRa significantly improves the OOD detection performance, especially after finetuning. Specifically, compared to RoBERTa, Spatial-RoBERTa improves AUROC by 9.20% for in-domain OOD and 4.95% for out-domain OOD data. This further verifies the importance of spatial-awareness for OOD detection in the document domain.
Sootial-RoBERTa is competitive for both ID classification and OOD detection Beyond also examine
In-domatn (Avg)
Out-domatn (Avg)
300
OOD detection performance, ID classification accuracy and plot the two met-
56
AURDOCC (R)
AUROO (%)
rics for all the models with different modalities in Fig. 9 We find that there exists a positive correlation between ID accuracy and OOD de- tection performance (measured by AUROC) for both in-domain and out-domain OOD. Moreover, spatial-aware models display superior ID accu- racy and OOD robustness compared text-only and vision-only models. Finally, our Spatial- RoBERTa provides a simple and effective solu- tion that greatly improves upon RoBERTa and matches the performance of models with spe- cific architecture such as LayoutLM. Specifically, Spatial- Riiiial-RBERTaaaagg achieves 97.37 ID accuracy, which is even higher than LayoutL (97.28) and UDoc (97.36). Furthermore, since Spatial- Spatial-RoBERTTaa RoBERTapass and Spatial- Roatiall-RBERRaar... freeze the word embed-
80
20
Pretrslood Freatured
nlnetaned
00
Spartel- Bed
60
Spatto Wot
Figure 8: OOD detection performance between Spatial- RoBERTa and RoBERTa. All models are initialized with public pretrained checkpoints on pure-text data and fur- ther pretrained on IIT-CDIP with the same pretraining tasks. The only difference here is that Bootial has an additional spatial-ward ware adapter and takes the word bounding boxes as the additional inputs.

---

Under review conference paper at ICLR 2023
ding during pretraining, it can learn spatial-aware feature target document data while keeping the word embedding fixed, thus reduce the trainable model size and reduce the training cost.
scete itt coos
( IC the Boae
barcoraaion Longlrmentian mantian
Rostars date covresce
Leaffome bade bo
Seit base
Jpstel warg Lay Boar
Coner
sumplommnndan dote
aestart Bare dartatio
mypaue (1)
Spatis Katumti Anden
Apotia Mettrreeherr Cesvel
formorr aurtart rurr co
Lenglomee Noer de
Bestitto
tyxlem
Vmttue (P)
deleion
naroro hare
Lopest Mivr
Te wwtthiiiiii
Disse
Figure 9: Correlation between ID accuracy and OOD detection performance. For most models, IDD accuracy is positively correlated with OOD detection performance. Spatial-aware models display both higher ID accuracy and stronger OOD robustness (in AUROC).
4 CONCLUSION AND OUTLOOK
This paper presents large-scale study of various methods for quantifying OOD robustness across differ- ent data modalities and models for document domains. Our key novelties include a large-scale investi- gation of OOD robustness in the document domain and simple yet powerful spatial-aware adapter for transformer-basee language models. We start from document classification and explore the pretrained mod- els for document OOD robustness. A variety of substantial experiments in different settings demonstrates that pretraining datasets and tasks greatly impact OOD detection performance. Notably, OOD samples the document domain are more accessible to identify in the feature space than in the logit space. Investigations from various perspectives cxplain certain intriguing phenomena and inspire more research on evaluating OOD robustness towards more reliable document understanding models.
REFERENCES
David Aevarez-Melis and Nicolo Fusi. Geometric dataset distances via optimal transport. In NeurlPS, 2020.
Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and Manmatha, Docformer: End- to-end transformer for document understanding. In ICCV, 2021.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint 200::0000000550, 2020.
Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In CVPR, 2016.
Lei Cui, Yiheng Xu, Tengchao Lv, and Furu Wei. Document ai: Benchmarks, models and applications.
arXiv preprint arXiv:2111.08609 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In NAACL, 2018
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and
10

---

Under review conference paper at ICLR 2023
Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't 't know by virtual outlier synthesis. In ICLR, 2022.
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-diisriibtioo detection. In NeurIPS, 2021.
Zong Yuan Ge, Sergey Demyanov, Zetao Chen, and Rahil Garnavi. Generative openmax for multi-class open set classification. arXiv preprini (.0ivv/777.07444 2017.
Jiuxiang Gu, Jason Kuen, Vlad Morariu, Handong Zhao, Rajiv Jain, Nikolaos Barmpalios, Ani Nenkova, and Tong Sun. Unified pretraining framework for document understanding. In NeurlPS, 2021.
Zhangxuan Gu, Changhua Meng, Ke Wang, Jun Lan, Weiqiang Wang, Ming Gu, and Liqing Zhang. Xylay- outlm: Towards layout-aware multimodal networks for visually-rich document understanding. CVPR, 2022.
Adam W Harley, Alex Ufkes, and Konstantinos G Derpanis. Evaluation of deep convolutional nets for document image classification and retrieval. In ICDAR, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In cvpr, 2020.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and dustrf--dittibuution examples in neural networks. In ICLR, 2017.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pre- trained transformers improve diitt--ditt robustness. In ACL, 2020.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling diittrddistiiuttion detection for real- world settings. ICML, 2022.
Teakgyu Hong. Donghyun Kim, Mingi Ji, Wonseok Hwang, Dachyun Nam, and Sungrae Park. Bros: A pre- trained language model focusing text and layout for better key information extraction from documents. In AAAI, 2022.
Yen Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting image without learning from -iitt--ditti data. In CVPR, 2020.
Rui Huang and Yixuan Li. Mos: Towards scaling out-of-distribution detection for large semantic space. In CVPR, 2021.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. In NeurlPS, 2021.
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layouthmv: Pre-training for document ai with unified text and image masking. arXiv preprint arXiv:2204.08387, 2022.
Guillaume Jaume, Hazim Kemal Ekenel, and -hinlPhil Thiran. Funsd: dataset for form understanding in noisy scanned documents. In ICDAR Workshop, 2019.
11

---

Under review conference paper at ICLR 2023
Di Jin, Shuyang Gao, Scokhwan Kim, Yang Liu, and Dilck Hakkani-Tur, Towards textual out-of-domain detection without in-domain labels. TASLP, 2022.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. ICLR, 2014.
Stefan Larson, Gordon Lim, Yutong Ai, David Kuang, and Kevin Leach. Evaluating cut-of-distribution performance on document image classifiers. In NeurIPS, 2022.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of- distribution samples and adversarial attacks. In NeurIPS, 2018.
Lewis, G. Agam, Argamon, O. Frieder, D. Grossman, and Heard. Building test collection for complex document information processing. In SIGIR, 2006.
Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou. Unicoder-vi: A universal encoder for vision and language by cross-modal pre-training. In AAAI, 2020.
Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu. Selfdoc: Self-super document representation learning. In CVPR, 2021.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR, 2018.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. In NeurlPS, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907.1 1692, 2019.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic aaoon-and-lanuuge
representations for vision-and- language tasks. In NeurlPS, 2019.
Ajoy Mondal, Peter Lipps, and CV Jawahar. liit-ar-13k: a new dataset for graphical object detection documents. In International Workshop Document Analysis Systems, 2020.
Peyman Morteza and Yixuan Li. Provable guarantees for understanding out-of-distribution detection. In AAAI, 2022.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? In ICLR, 2019.
Poojan Oza and Vishal M Patel. C2ae: Class conditioned auto-encoder for open-set recognition. CVPR, 2019.
Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jacheung Surh, Minjoon Seo, and Hwalsuk Lee. Cord: A consolidated receipt dataset for post-ocr parsing. In NeurlPS Workshop, 2019.
Jonas Pfeiffer, Andreas Rücklé Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. arXiv preprint arXiv:2007.07779, 2020.
Yu-Ting Qiang, Yan-Wei Fu, Xiao Yu, Yan Guo, Zhi-Hua Zhou, and Leonid Sigal. Learning to generate posters of scientific papers by probabilistic graphical models. JCST, 2019.
12

---

Under review conference paper at ICLR 2023
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribttion detection. In NeurIPS, 2019.
Chandramouli Shama Sastry and Sageev Oore. Detecting out-of-distribution examples with Gram matrices. In International Conference Machine Learning (ICML), 2020.
Vikash Schwag, Mung Chiang, and Prateek Mittal. Ssd: unified framework for self-supervised outlier detection. In ICLR, 2021.
Ray Smith. An overview of the tesseract ocr engine. In ICDAR, 2007.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic visual- lisuul-inguistic representations. In ICLR, 2019.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. In NeurlPS, 2021.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neigh- bors. In ICML, 2022.
Thirumalaisamy P Velavan and Christian Meyer. The covid- 19 epidemic. Tropical medicine & interna- lional health, 25(3):278, 2020.
Wenjin Wang, Zhengjie Huang, Bin Luo, Qianglong Chen, Qiming Peng, Yinxu Pan, Weichong Yin, Shikun Feng, Yu Sun, Dianhai Yu, et al. mmlayout: Multi multi-grained multimodal transformer for document under- standing. ACMMM, 2022.
Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Ledsam, Patri- cia Mac Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al. Contrastive training for improved out-t--distribution detection. arXiv preprint arXiv:2007.00 0.000005666 2020.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.0377), 2019..
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Yen Lo, and Ross Girshick. Detectron2. https:
//qithub. cww. wtt.://ccc.ciaaeeeee http///abbbokkeeeercttee dewp/treeeeeeoeeeee 2019.
Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: An out-of-distribution detection score for variational auto-encoder. In NeurIPS, 2020.
Keyang Xu, Tongzheng Ren, Shikun Zhang, Yihao Feng, and Caiming Xiong. Unsupervised out-o--domain detection via pre-trained transformers. In ACL, 2021a.
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, et al. Layoutlmv2: Multi-modal pre-training for visually-rich document under- standing. In ACL, 2021b.
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pre-training of text and layout for document image understanding. In SIGKDD, 2020.
13

---

Under review conference paper at ICLR 2023
Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Publaynet: largest dataset ever for document layout analysis. In ICDAR, 2019.
Wenxuan Zhou, Fangyu Liu, and Muhao Chen. Contrastive out-of-distribution detection for pretrained transformers, EMNLP, 2021.
14

---

Under review conference paper at ICLR 2023
A APPENDIX
A.1 DOCUMENT CLASSIFICATION
A.1.1 DATASETS
RVL-CDIP consists of 320K/40K/40K training/validatioo/eesngg images under 16 categories. We select 12 categories and treat them as ID (In-Domain) data. We extract the text and layout information with Google OCR engine] which provides both tokens and text blocks along with their corresponding bounding boxes. Most recent models take the full IIT-CDIP as pretraining data and finetuning on RVL-CDIP. However, it is not reasonable for the OOD setting since RVL-CDIP is subset of IIT-CDIP. To make OOD results more reliable, in our experiments, we exclude the RVL-CDIP from the IIT-CDIP during pretraining.
We measure the distance between in-domain and out-domain datasets via OTDI We first visualize the OTDD distance between ID and the OOD data (in-domain and out-domain datasets in our main paper) in Fig. [O(a) During analysis, we sample the maximum number of 1000 images from each data and calculate the distance between datasets. It can be seen that there is clear gap between in-domain data and out-domain data. To make the analysis more thorough, we have two additional in-domain OOD settings: (1) select the classes the model performs well as OOD data; (2) randomly select classes as OOD data. Fig. To(a) and Fig. [T[c]] show the dataset distance. As for the other two selection strategies, we can see that the domain gap is not as clear as the subset we selected for the main experiments. Interestingly, for those rare-w word documents, such as file-folder and advertisements, the dataset distances are larger than documents with rich words. The background colors and layouts may yield a big distinction for those documents.
(a) OOD (Worst performance).
(b) OOD (Best performance).
(c) OOD (Random selection).
Figure 10: Visualization of optimal transport dataset distance for ID and OOD (in-domain and out-domain) datasets. We highlight the in-domain data in blue and the out-domain in green.
A.1.2 MODELS
All models reported in Fig. 2(b) except for UDoc, are initialized with model pretrained weights from Hug- gingface and finetune on the full RVL-CDIP training set. During finetuning, we train those models on RVL-CDIP with the cross-entropy loss. Models are optimized with Adam optimizer (Kingma & Ba] 2014) for 30 epochs with batch size of 50 and learning rate of 2 x 10-5 on 8 A GPUs. We list the hyperpa- rameters of models used in our paper as follows:
aatraa//ggttuubb.ccmmmmmrrrsstft
http:://huggdingfae.ccooo http://aaaaa.cccoc...eoee
15

---

Under review conference paper at ICLR 2023
Fatna ExOcaaai
Fritas 22
Forza Madiccal
Fritaio thd
Paraca Dacial
(a) RoBERTaBass (10%)
(b) RoBERTaBase (20%)
RoBERTaBas Basse (40%)
(d) RoBERT Base (100%)
hetur
(e) BaTe (10%)
(I) BaT Basn (20%)
(g) ViT Raze (40%)
(h) ViT (100%)
Figure 11: Feature visualization for pretrained (with different numbers of pretraining data) and finetuned models. We show both In-Domain (RVL-CDIP) and Out-Domain (CORD) OOD datasets.
KVN((-10)
KNN(K-10)
KNNNO-10
Moha
Maha
Energy
Mahu
(a) ResNet-50
(b) RoBERTa-base
(c) My3-baee
MSP, Energy, KNN, and Maha score histogram distributions of ID (blue) and OOD (green) inputs derived from finetuned ResNet-50, RoBERTa, and LayouLMv3. The KNN scores calculated from both vision and language models naturally form smooth distributions. In contrast, MSP and Maha scores for both in- and out-of-distribution data concentrate on high values. Overall our experiments show that using feature space makes the scores more distinguishable between and diitof--iisributions and, as a result, enables more effective OOD detection.
Figure 12:
16

---

Under review conference paper at ICLR 2023
Language-only: (1) BERT and RoBERTa. We adopt the RoBERTaBase (12 layer, 768 hidden size) and BERTBuse (12 layer, 768 hidden size) as the backbone and set the maximum sequence length to 512. For RoBERTa, the classifier is composed of two linear layers followed by a tanh activation function. (2) Long- former. We also adopts the LongformerRase (12 layer, 768 hidden size) as the backbone and sets the maxi- mum sequence length to 4,096.
Vision-only: (1) ResNet50: This model adopts the ResNet50 (pretrained on ImageNet- 1k) as the backbone. pretrained
We finetune it at resolution 224x224 (2) ViT: This model adopts Base sitt-bas-ppachh666 the visual backbone. We finetune it at resolution 224x224. (3) SwinB: This model adopt swin transformer We finetune it at resolution 224x224.
ImageNet-21kk
pretrained on ImageNet-2Ik) as backbone.
This model adopts the LayoutL (layoutim (layoutm-baseeunne)) base-uncased, 12 layer, 768 hidden size, pretrained IIT-CDIP as the backbone. We set the maximum sequence length to 512. (2) Spatial-RoBBRRRanaaee (Pre): This model combines our spatial-aware adapter to the pretrained RoBERTaBase model. The adapter is applied the word embedding layer. During pretraining, we freeze the pretrained word embedding and optimize the spatial-aware adapter and transformers. During finetun- ing, we optimize all the parameters. (3) Raatial-RoBERTapee (Post): Instead of inserting the spatial-aware adapter in the input layer, this model combines the spatial-aware adapter at the transformers' output layer.
Sayout+Languagee (1) LayoutLMvl:
(1) LaytouLMv3: This model adopt the LayoutLMv3 (layoutnv3--bsse, 12 layer. 768 hidden size, pretrained on IIT-CDIP) as the backbone. 21) UDoc: This model follows the design of UDoc. The only difference is the sentence encoder, we adopt smaller version of the pretrained sentence encoder (all-Minil M-L-6v2,, 6 layer, 384 hidden size) instead of the sentence encoder (bert-base-nli-menn tokens, 12 layer, 768 hidden size) used in their paper.
Vision+ Laaggaggg++Layooo
A.2 BEYOND DOCUMENT CLASSIFICATION
Going beyond document classification, we explore OOD detection for two entity-level tasks: document entity recognition and document object detec- tion. Basic units such as text, tables, and figures in the document are the ob- jects that need to be detected and rec- ognized. Document entity recognition aims to predict the label for each seman- tic entity with given bounding boxes. Document object detection is ob- ject detection task for document images. Specifically, denote the input as x, the bounding box coordinates associated with object instances in the image R1. and use the model with parameters 0 to model the bounding box regression Pa (b/zz and the label classification (o(y|2, b). Given a test input å, the OOD detection scoring function for entity detection and recognition the object detector. In particular, for document entity recognition, since the bounding boxes are provided, the OOD score be simplified S(i. b), where is the given object instance.
Enily Recepmition
Detection
Recagentinn Read
Breegeation Heal
Vinual
Blooes
Mosel Eacoder
Tistsef Encader
Bilans
Trine! Eacadec
Vinaa Encader
Proposel Gencration
Eneader
WaloooBBloos
Worderileeee
Figure 13: The network architectures in green blocks are our pro- posed models. We also show the modality information top of each architecture.
be unified S( b), where b denotes the object instance predicted by
17

---

Under review conference paper at ICLR 2023
A.2.1 DATASETS
Document Entity Recognition The original FUNSD (Jaume et al. 2019) dataset contains 149/50 train- ing/testing images. we treat entities with category other/header as the OOD entities. After doing the split, if we treat other as OOD, we have a total number of 8,330/1 1,300/1,199 ID/OOD entities in total. Otherwise, if we treat header as OOD, we have 8,981/368 ID/OOD entities in total.
Document Object Detection PubLayNet (Zhong et al. 2019) contains 336K/11K IK training/validation im- ages with 6 category labels (text, title, list, figure, and table). The original IITT-AR-13K (Mondal et al. (2020) contains (lable, Sigure, natural image, logo, and signature). In our paper, considering the overlap between IITT-AR-13K and PubLayNet, we select those images that contain Natural Image as the OOD test set. After filtering, we have 2,880 OOD entities across 1,837 document images.
We consider three ID datasets in this experiment. (1) PubLayNet: This is the original PubLayNet dataset. We treat all the entities in training/validation images as ID entities. (2) Considering the domain shift between ID data (PubLayNet) and OOD data (IIT-ARR 13K). We combine the PubLayNet training data with the images from IITT-AR-13 13K with overlapping annotations (table and figure) and train the object detection model.
A.2.2 MODELS
Document Entity Recognition Fig. 1 illustrates the entity recognition models used in this paper. We consider the entity on regions instead tokens since regions contain richer semantic information. As for the pretrained model, we adopt UDoc (pretrained IIT IIT-CDIP) since it models the inputs at the region level. Based on UDoc framework, we develop the following models.
This model is composed of the ResNet-50 from pretrained UDoc. It adopts the Rol pooling followed by a classifier to extract the entity features. (2) ResNet-500PPsitionn This model also adapts UDoc pretrained ResNet-55. It further improves the Rol features be spatial-aware by adding position embedding, where the position embeddings are mapped from bounding boxes via linear mapping layer.
(/) ResNet-50::
This model adopts the language branch of UDoc and appends the classifier the output the sentence encoder. (2) Sentence BERT+Positions This model close to the above model but adds position embedding to the sentence embeddings.
(I) Sentence BERT:
Vision-LanguageLLLyout (l) DesNet-50+sentent BERT: This model follows the same framework as UDoc, but replaces the sentence encoder in their original design with more miniature sentence encoder (all-MiniLMM-L--v)), (2) SwinT+Sentence BERT: This model replaces the ResNet-50 visual backbone with adopted from the Huggingface.
a pretrained Swin tiny model
All the models are finetuned with cross-entropy loss for 100 epochs with a learning rate 10-5 and batch size of 8 one A100 GPU.
Document Object Detection Two object detection models are considered in this paper: (1) Vanilla Faster- RCNN: This model is the Faster-RCNN with ResNet-50 visual backbone. (2) RaNN with VOS: This model enhances above model with V Following their paper, we use 1,000 samples for each ID class to estimate the class-conditional Gaussians. We train detection models with the Detectron2 framework (Wu) et al. 2019). Models are trained for 180k iterations with a base learning rate of 0.01 and a batch size of 8. Mean average precision (MAP) @ intersection over union (IOU) [0.50:0.95] of bounding boxes is used to measure the performance.
'nttps://github. http://hhhhb.ccmmm..eeeammammaa/
18

---

Under review conference paper at ICLR 2023
Decument Eutity Recoguition Detection
Document Object Detection Detection
MED (langaagrt
Vie
(a) Comparison of OOD detection methods on differ- (b) OOD detection results from different object detec- cnt models on two OOD classes: other and header. tion methods.
Figure 14: Ablation on document entity recognition and object detection. Numbers reported in FPR95.
A.2.3 EXPERIMENTAL SETUP
For document entity recognition, we construct ID and OOD datasets from FUNSD. Each semantic entity includes a list of words, a label, and a bounding box. The standard label set for this dataset contains categories:
question, answer, header, and other. In this paper, we select the entity labeled as other or header category as OOD, And the entities belonging to the other three categories are ID. Instead of treating entity recognition as named-entity recognition problem, we follow UDoc and solve this problem at the semantic region level. We replace the sentence encoder in UDoc with a smaller sentence encoder (all-MiniLM- L6-v27 from Huggingface (Wolf al. 2019). We also have the following model variants verify the effectiveness of each modality combination: textual-only, visual-only, textual+spatial, visual+spatial, and visuaalteextual+sppaaal..
For document object detection, we use Publ as the ID dataset. We construct the OOD dataset from II'--RR-33K Unlike PubLayNet, where the documents scientific articles, IIT--R--13K is dataset for graphical object detection in business documents (e.g., annual reports). Hence, there exists an obvious domain gap between these two datasets. We select natural images as the OOD entity and filter images that contain the OOD entity. We first adopt the vanilla Faster RCNN with ResNet-50 backbone for document object detection as the baseline model. We also enhance Faster RCNN with VOS (Du al. 2022). a recent unknown-award learning framework to improve OOD detection performance for natural images.
A.2.4 OBSERVATIONS
To identify the entity type, models should not only understand the words but also require spatial and visual reasoning ability. We summarize our findings document entity recognition in Fig. [1 (a) and describe them in more detail in Table We can see that models can better predict the entity type with the help of the spatial position and also achieves better OOD robustness. Considering the weak language dependency between entities, it is not supervising that vision-based models achieve better performance than text-based models. We can see that UDoc with ResNet-50 achieves the best performance on two OOD test sets, il- lustrating that visual information plays a major role in increasing the discrimination of entities with similar semantics. We summarize our findings on document object detection Fig. [4 (b) and describe them in more detail in Table We can see that the OOD detection performance is further improved by introducing document images from IIT--AR-13K with the same ID annotations as training data.
In Fig. [5] we visualize some document entity recognition OOD detection results. In Fig. 16 we visualize the prediction on sample OOD images, using object detection models trained without VOS (top) and with voS (bottom), respectively. There is clear difference between PubLayNNe and LIT--AR-13K image annotations and entities rarely exist in PubLayNet. We can see that vanilla Faster RCNN trained on PubLayNet produces false positives when applied to the OOD document image from - 13K. After
natural
htttos///mgggingfacccco/// co/ frr.tttenecettaaaaaeoommeee
19

---

Under review conference paper at ICLR 2023
introducing the unknown-aware learning method optimized for both ID and OOD, as shown in Table 2 the FPR95 reduces while preserving the mAP on the ID data. This experiment indicates that bringing uncertainty estimation into the entity detection training procedure can improve the reliability of the document object detection system.
Figure 15: Visualization of detected OOD entities on the form images. The top part shows the entities in blue are entities annotated as other. The bottom part shows the detected OOD entities (green). We also show failure cases on the right part.
(a) PublLayNer (ID)
(b) DUT-AR--1E 13K (OOB)
Figure 16: Visualization of detected objects on the OOD images (from -IT--R--133 13K) by a vanilla Faster- RCNN (top) and RCNN with VOS (bottom). Objects in blue boxes are detected and classified as one of the ID classes. The detected OOD objects (green) reduce false positives among detected objects. We also visualization of detected objects on the ID images. There is clear difference between PubLayNet and IITT-AR-13K natural image annotations and entities rarely exist in PubLayNet.
A.3
ADDITIONAL EXPERIMENTAL RESULTS
Table
corresponds to the results shown in Fig. 15 and Fig.
4(a)
Table
corresponds to the results shown in Fig. 16 and Fig.
14(b)
Table
and Table
correspond to the results shown in Fig.
(())
Table
and Table correspond to the results shown in Fig.
5(c)
Table
corresponds to the results shown in Fig. g and Fig.
Table
and Table
g correspond to the results shown in Fig.
and Fig.
Table
and Table correspond the analysis in Sec.
Table
corresponds to the results shown in Fig.
20

---

Under review conference paper at ICLR 2023
Table Comparison with different models on FUNSD OOD setting. All models are initialized with UDoc pretrained on IIT-CDIP and finetuned on FUNSD data with ID entities. All values are percentages. A lower FPR95 or higher AUROC value indicates better performance.
Method
Other (GOD)
AUROXU
(OOD)
Method
Other
(O0I)
Sahawtoe
X5.32
59.21
12:12
15.35
05.35
97.25
20.01
KNN
MahamuSeen
94.11
29.
81:29
24.06
ResNee500 50-PPlttosstoo Posiiion
KNN
89.11
73.21
33.51
90.23
34.1
50
KNN
KNN
09.97
63.56
KNN
KNN
77.08
KNN
77.98
KNN
77:15
KNN
Malasyun
Mahis Mahauaaker
Malancheen
DERT
KMN
KNN
Sentence MERE-
KNN
KNN
KNN
KNN
KNN
KNN
KNN
54.91
Mislagii aigh
54.75
Mahisons
Sein SwinT+Smmnee Senneeeeeee NERT
RonNee-- Sin--SnSSmeeeee
$7.77
KNN
KNN
$1.50
75.54
75.79
KMN
$1.00
91.55
75.49
47.45
$7.52
67.39
70.70
52.02
MaxLogit
90.58
63.04
MaxLogit
74.41
52.77
90.57
63.85
74.41
52.77
Table 2: Comparison with different training and detection methods.
IIT-AR-13K (Natural Image OOD)
PubLayNet (ID)
Models
ID
Method
FPR93
AUROC
AUPR
mAP
MSP
74.33
79.12
98.41
Vanilla FaNter-RCNN
PubLayNet
92.6
Energy
55.96
MSP
63.65
79.37
98.57
Faster-RCM with VOS
PubLayNet
92.2
Energy
55.61
80.60
98.67
Faster-RCNN with VOS
PubLayNewHIIF-R-IIIKID
MSP
56.57
82,94
98.59
92.4
Energy
47.73
84.04
98.67
21

---

Under review conference paper at ICLR 2023
Table 3: OOD detection performance for document classification with different number of pretraining data from IIT-CDIP. ID (Acc) denotes the ID accuracy obtained by testing ID test data. We report the KNN- based scores for both pretrained and finetuned models. Sci. Poster denotes the document images converted from NJU-Fudan Paper-Posee Dataset. Receipt denotes the receipt images collected from the CORD receipt understanding dataset. For in-domain OOD test data, we also report the averaged scores.
OOD Duinel (I-Daaaaaa
OOD Dataset (Out- Dunain)
ID
Mathod
Sci. Report
Presentation
Form
Letter
Arange
Scl. Poster
Receipt
Ace
FPR595
AUROC
FPR95
AUROC
FPR
AUBOC
FPR95
AUROC
FPR95
AUROL
1 EPR95
AUROC
FPRU5
AUROC"
Pretram 10% TYT- DIP F Fineluus on RVL-C.DIP TD datb
MSP
22.75
07:22
12:21
68.33
94.65
05.20
92.00
70:35
92.50
67.92
96.31
56.55
10:10
52.30"
MasLagit
98.36
77.85
97:25
73.51
98.76
72.84
98.56
75.38
98.30
76.82
100.00
78.69
100.00
63.74
Eneny
98.60
77.81
97.55
78.49
98.96
72.79
98.94
75.00
98.51
76.37
100.00
78.68
100.00
63.70
GralNarm
98.04
79.26
97.07
76.35
95.55
72.83
98.62
50.55
98.07
17.37
100.00
85:23
100.00
64.10
97:72
58.35
97.11
36.97
95.45
45.02
93.43
16.56
96.18
41.72
95.84
55.19
96.80
48.45
RoB (10%)
90.59
Maharaater
97.36
39.23
97.27
38.31
95.25
45.97
90.59
16.50
96.27
42.55
95.85
56.11
96.90
49.25
KNNN
17.21
15.18
05.1
35155
73302
31.03
07.74
$5.91
61.35
$7.30
09771
$5.30
00.50
15.34
KNN
63.53
88.07
65.39
$7.90
72.75
54.45
67.33
59.51
67.36
87.32
68.60
55.13
91.10
84.09
KNN
64.17
67.89
66:97
$7.77
73.34
84.25
67.21
$9.50
67.92
87.12
72.09
$7.47
91.60
83.59
KNN
$4.09
67.64
67.75
$7.55
73.05
83.
67.29
$9.37
68.26
36.38
72.09
56.33
91.50
83.21
KNN
55.25
87.29
68.34
$7.27
74.12
93.52
07:45
$5.10
68.76
36.54
72.09
56.09
91.60
82.77
KNN
56.13
$6.84
69.45
$6.98
75.14
82.94
08.55
$7.81
69.11
36.14
74.42
$5.29
91.70
82.27
Pretram on 10% ITFCDIP (no finetune)
KNNN
68.07
94
92:1
66.62
94.13
61. 61.90
94.40
$4.50
92.1 92.18
60.51
67.46
$7.04
62. 10
84.08 84-94
KNNN
63.50
66:00
92
65:25
94.11
60.83
94.72
53.70
92.52
61.47
77.91
$5.18
64.60
83.85
KNN
80.75
64.40
93.55
63.12
94.33
58.95
95.17
52.53
93.20
59.71
85.72
$2.07
69: 10:20
82.29
KNN
90.23
62.94
93.85
61.28
94.41
57.45
95.13
$1.28
93.40
58.34
55.72
$0.91
70.10
81.05
KNN
90.55
60.90
94:17
58.98
94.53
55.50
95.50
40.91
93.69
56.34
55.72
77.86
71.00
79:31
KNNNce
91.43
59.73
94.54
57.24
95.01
$4.40
96.74
49.57
94.18
55.24
57.21
75.05
73.40
77.74
Preeram on 20% 11 NVE-DIP..... Findlome om RVI
-CPP
ID dataa
MSP
94.28
68.00
1/6 46:46
65.38
95.01
63.95
94.51
65.06
94.89
65.14
05:35
65.55
99:10
$4.99
MasLania
97.36
77.82
97.19
79.16
98.40
72.64
96.34
77.68
97.82
76.82
100.00
77.36
99.60
66.63
Enery
98.04
77.80
97:45
79.15
95.76
72.61
98.58
77.64
98.20
76.30
100.00
77.32
99.60
66.61
GaalNum
07.36
90:68
96.83
76.04
95.44
73.29
97.50
51.37
97.53
37:35
100.00
$6.18
99:50
67.3
Mahasion
98.80
36.53
98.27
35.70
97.73
39.46
98.54
36,66
98.24
37.08
98.84
49.45
05. 85.80
52.25
RobtieTa (20%)
90.71
Mahacasten
58.92
36.39
98.39
35.38
97.77
39.26
98.54
54,94
98.40
36.49
98.84
49.32
95. 05..0
52.25
KNNNIC
33.37
83.30
0736
37.00
73500
33.92
7330
87.00
09:24
30.77
09777
35001
$7.00
8331
KNNNs
53.85
89.20
67.45
73.94
83.78
72.93
80,00
09.54
36.00
09.77
$7.63
88.
83.55
KNNNe
53.80
83.02
36.71
24.38
83,55
72:34
81.46
69.51
70.93
$7.9
85:
83.12
KNN
54.85
87.81
36.45
74.90
83.25
72.65
K0.24
70.00
36
72.09
36.05
83.
KNNove
56.45
83.51
68
75.86
83.83
86.07
71.12
73.26
KNN
$7.09
$7.16
69:35
35.78
76.38
92.20
73.50
96.71
71.75
35:46
74.42
$5.37
$8.50
82.03
Pretramn on 20% ITF-COIP (mo linetune)
KNNNs
87.15
68.22
90:8K
66.39
92.26
62.39
95.01
$3,00
91:32
45.02
92.29
57.00
87.67
KNNNs
67.31
92.04
01.54
94.99
$7.33
91 91.46
47.67
91.18
85.61
KNNNG
$8.30
92
97.18
92.13
56.98
89.64
KNNice
03.15
92.23
57.99
92.41
58
58.36
84.
KNNoue
39.07
03:21
92.66
95:45
92.60
67.55
68
KNNrus
03.85
$7.44
47.79
70.10
Preeramm on 40% ITCCPPP
inetune
CDVP
data
MSP
20:00
0%.0%
65.54
02.70
66
05.35
Maxlagi
07.87
79.85
71.63
98.
75.41
98.1
76.40
08.84
78.07
08.00
75.65
Energ
98.48
07.91
79.83
08.68
71.61
75.40
9% 29.39
76 38
78.04
75.60
GualNum
47
72:77
79.11
97.84
67.47
77. 12:12
Maluxoon
97.60
92.53
95.77
77.11
62. 02.70
70.78
90.76
Mahacaston
96.33
40.07
94.44
39.14
96.48
20
77.91
71.13
IN
00.57
65336
33.35
73.90
57.13
72.70
KNNNc
73:46
$7.00
73.50
KNNNc
75.66
73:42
70.93
RobiRR
KNN
$1.93
KNNN:
14.76
63101
25.56
$7.30
85.60
KNNas
64
04.51
90.36
62.00
62.30
87.61
KNNNa
59.35
94.93
90.91
41.86
66.80
86.25
$7.74
92.05
83.74
93.77
$4.01
95.45
92.50
50.00
72.90
82.29
100%
DIF
LVL
ID
MSP
68.58
94.54 54
65.83
63.11
94.12
68.28
96.64
66.53
95.84
62.52
95.10
71.25
MasLogit
97.95
50.23
74.01
98.25
98.13
77.67
100.00
95.90
79.36
78.64
97.95
77.65
79
GialNum
90.61
77.50
Malassons
98.88
37.34
94.03
97.61
34.50
38.95
Roh (1405)
91.00
00
37.06
34.24
39.15
K
63.50
75.75
10.37
63.56
63.90
5003T
KNNNc
69.59
84.56
70.70
69.20
36.92
74.42
67.20
90.37
KNNNc
$7.87
69.79
36.79
84.30
71.14
6967
36.71
76.74
67.90
90.22
KNN
$7.61
70.27
36.58
74.86
84.00
71.35
70.50
36.40
79.07
68.30
90.19
KNN
66,57
68
36.27
76.34
83.58
$7.58
71.85
36.12
$2.56
$2.77
69.20
90.09
KNNara
67.32
86.80
71 96
$5.91
77.41
83.00
72.52
$7.00
72.32
$5.90
85.72
$7.06
699 09:00
89.89
110%
TDTP (nn Rintume)
KNNN::
70:33
65.34
63.15
54.72
50.16
30105
63.52
27.01
V0.111
46.01
VT.
KNN
$4.51
60.30
11
67.35
90.38
61.96
94.72
$1.43
90.22
62.51
35.72
93.39
$1.
90.55
KNN
85.67
67.75
91.99 92
65.35
90.82
$9.79
94.59
49.77
90.82
68.66
39.53
92.28
56.70
89.32
KNN
$6.55
66.08
92.97
63.46
91.45
55.00
95.41
48.30
91.60
58.98
44.19
91.29
61. 10.60
88.15
KNN1ce
$6.91
64.20
93.61
61.34
91.86
$3.95
95.66
46.65
92.61
$7.04
47.67
90.30
65. 70
87.03
KNNNCE
$7.99
62.45
94.09
$9.48
92.26
$4.05
95.90
45:07
92.56
55.26
$3.49
$9.19
69.40
85.65
22

---

Under review conference paper at ICLR 2023
Table 4: OOD detection performance for document classification with different number of pretraining data from IIT-CDIP (remove pseudo OOD categories).
0OD Duinsel (In In-Domaini
OOD Danaset (Oui- (on--inmiin
TD Ace
Method
Report
Presrntatoo
Form
Letter
Avvngge
SEL Poster
ldrcuige
AURIN
AURRC:
AUROC
TERUS
AUROU
AUROU
AUTRIC"
Protram on I0% ITF- DIP
Finctuns on
RVT IVIP
data
MSP
30.07
67:30
39:92
68:36
92:55
66::15
91:07
06.75
90.51
67:20
96.31
54.37
95.70
59163
MasLogit
97.76
78.40
07.71
$0.58
98.64
71.26
96.70
76.38
98.20
76.66
100.00
73.51
99.80
73.32
Enewy
98.16
78.35
97.75
$0.55
98.84
71.20
98.90
76.32
98.41
76.60
100.00
73.46
09.80
73.31
GndNoam
97.68
73.92
97.27
79.42
95.56
71.31
98.50
79.34
98.00
77.52
100.00
$2.62
09.60
75.85
Mahasiome
90.36
28.99
97.75
37.09
97:37
41.81
92.53
16.51
96.25
41.10
97.67
55.81
96.20
48.85
90,62
Mahaesstoo
97.40
19.19
97.87
37.86
07.41
42.54
92.49
16.82
96.29
41.00
97.67
56.33
49.50
KNNNt
10.35
87.35
00.09
15.05
12.32
14.35
55.35
71177
30.42
35.16
$3590
87.1
KNNNer
56.33
75.94
82:00
73.54
86.70
70.76
36.32
87.21
$4.65
83.
87.21
KNNNz
56.77
83.00
01.94
36:13
38.17
83:
87:58
RoBERR
KNN
67.25
86.30
71
39.53
KNNauo
69.02
81:71
74.47
81.76
86.
KNN
70.14
80.55
20.05
$7.37
75,13
81.05
75.53
85.51
73:46
35.12
90.70
30.78
85, 00.
85.58
Pretrain on 10% ITF-CDPP (po Bloelne))
KNNN
$6.35
65:45
85.74
70.34
92.11
39.35
90.14
36.00
$9.34
63.12
20.01
95.82
37.000
85:13
KNNNcc
56.87
64.48
87.14
09.68
93.30
39.41
90.30
55.91
90.15
62.12
37.21
94.75
88. 88.00
81.44
KNNKc
57.75
02.73
38.99
67.80
93.50
56.54
90.75
54.50
91.00
00.40
47.67
93.71
90.30
78.97
KNNNcc
$3.43
67.17
39.55 59
06.05
91.62
54.91
93.99
53.40
91.41
58.50
48.85
95.00
91.3 39.50
77.00
KNNKrr
68.05
19.28
90:32
63.95
94.29
53.09
94.30
52.00
91.33 57
57.1
52.33
92.40
93. 00.50
74.23
KNN
53.10
04.68
98
41.50
166
Pretram on 20% ITFCPPP
Fnvtume on
RVT DDP
TED dataa
MSP
36.04
67:58
122.911
65:332
95.05
61.92
06:23
00.02
95.20
67:36
100.00
61.30
95. 05.70
56.35
MasLagit
97.96
76.90
07.55 5355
$0.68
95.45
72:31
98.74
77.72
98.19
76.91
100.00
75.91
99:50
69:21
Eneay
98.16
76.80
98:25
$0.65
95.88
72.26
99.07
77.67
98.58
76.87
100.00
75.89
99:50
69.11
GadNNam
97.84
78.23
97:31
79.57
95.00
71.44
90.46
50.03
97.90
77.07
100.00
$5.30
99.00
69:54
Mahasione
97.84
45.53
98.31
40.63
97.25
47.35
96:31
50.09
97.45
45.90
96.51
39.30
100.00
51.69
RoBkRe (2P%)
90,65
Mahaesster
97.88
65.43
98:27
97.25
38
96.31
49.90
97.45
46.16
96.51
40.44
100.00
51.42
KNNNIC
50.15
87.00
07.70
37.5
13510
13:30
N7.30
7011
30.05
7731
90.19
10.10
35.32
KNNNer
56.17
87.50
38
$7.83
73.90
83.95
73.06
87.
70.53
36.32
72.91
39.34
890
84.11
KNNKr
57:21
67.03
83.63
70.88
79.07
890
KNN
78
08
75.90
16
71 1.92
32:56
83.
KNN
09.66
67.37
81.85
73.78
87.08
72.13
85.70
36.05
$7.08
83.26
KNNN
70.04
86.14
09:65
$7.06
75.02
81.24
74.00
$0.87
72.60
35:33
38.37
$7.24
89.30
82.88
Pretrain on 20% HTF-CIPP (no Bioelne))
KNNN
$3.03
66.10
85:17
70.34
91.33
000.25
50.43
30.33
39.20
63.40
30:25
55.72
83.20
85.34
KNNNcc
$6.31
65.17
85.90
09.13
93.30
39.09
90.47
56.05
89.77
62.36
34.88
95.08
84.90
82.15
KNNKsc
$7.31
63.50
87.63
67.11
93.38
57.17
94.16
54.00
90.62
08.00
46.19
94.077
83.50
79.24
KNNNice
$7.83
62.06
38:27
68:31
93.62
55.65
94.32
53.56
91.01
59.3 19.14
48.85
93.08
83.
79.27
KNNKer
$8.43
60:26
63:16
94.29
53.87
94.52
50.29
91.62
57:40
50.00
92.79
890
74.95
KNN
$9.00
61.33
94.73
53.89
94.96
$1.50
92.12
56.30
65
92.10
00.50
72.93
Predram on 40% 11 DIP
Inelume on RVT
DD1P
T19 data
MSP
03.54
18.85
03.
67.33
95.41
63.01
04.30
65.25
05.25
04:41
96.31
65.32
03.90
54.12
MasLagia
97.16
78.56
96:87
$0.18
95.68
71.84
98.58
74.44
97.82
76.26
100.00
76.72
99.03
65:41
Eneayy
97.40
78.53
97.15
50.17
95.65
71.79
98.78
74.39
98.00
76.22
100.00
76.67
99:50
65.39
GadNNum
97.34
80.59
96.95
79.01
95.52
72.12
98.34
77.16
97.26
76.97
100.00
$6.54
99:70
67.45
Mahasione
97.08
41.34
98:83
32.09
95.64
43.30
96.23
49.91
97.84
41.06
100.00
59.44
99.20
28.39
90.72
Mahaeaoter
97.72
41.59
98.85
32.93
98.60
43.70
96.23
50.00
97.84
42.07
100.00
59.11
09.20
30.87
KNNN
56.89
87.91
03.3%
30.30
71.61
32.21
76.38
35.25
12.11
35.03
75.38
395355
$85.00
KNNNe
57.57
87.80
21.71
93:
76.30
22
72.64
35.37
30.25
$9.17
86.
83.85
KNNNr
57.97
87.58
78.01
81:50
76.00 66
84.00
73.08
35:27
50.25
55.65
87:
83:21
KNN
79.01
81:33
$4.59
35.00
$2.56
KNNoue
70.96
72:30
79.79
81.33 31
78.00
$4.10 19
75.27
$4
$7.58
82:05
KNN
72.46
80.58
75.15
35.91
80.65
$0.70
79.99
83.54
76.31
34.26
83.72
36.36
89:00
$1.14
Pretrain on 40%% CTF-DIP (n0 linelue))
KNNN
$3.79
66:14
38.35
08.92
91.30
100.00
10:.4
$1.00
31.34
61.01
37.21
95.37
33.90
01.30
KNNNs
$9.59
63.07
89:80
67.61
93.89
39.10
95.58
50.17
92.21
60.49
46.51
94.41
61.50
91.00
KNNKsc
90.59
63.39
91.0 31.64
65.68
93.77
57.35
95.06
45.63
92.92
58.76
53.49
93.06
66. 06.40
89:72
KNNKice
91.19
61.79
92.37
63.90
93.66
55.55 76
95.00 82
47.42
95.21
57.22
05.1 12
91.59
68. 03.30
88.72
KNNNcc
92.00
60:06
92.85
94
93.63
54.02
95.55 36
46.08
95.60
55.55
76.42
90.94
71. 13.00
87.663
KNNNrs
97.10
S1.81
95:41
60.40 48
94.09
73.79
96.
45.47
90.01
54.09
72.91
$9.09
23.500
86. $6.59
Pretramm on 1000% 18 11 1PPP
Finedame em RVT-C [1PP
11 data
MSP
34.12
68.25
04
66.18
05.01
63.85
05.31
65.06
94.30
65.05
05.54
59.35
56.50
65:43
MasLagia
97.24
75.15
07.11 19
$0.27
98.36
72.16
98.38
75.83
07.79
76.40
100.00
73.25
09.33
75.58
Enery
97.32
78.13
07.51
50.26
95.64
72.12
98.70
75.78
98.04
76.57
100.00
73.27
09.60
75.52
GadNuam
97.16
90.07
07.39
77.86
95.40
71.83
98.05
79.08
97.75
77.21
100.00
$6.32
09.00 00.00
73.92
Mahasione
98.60
38.30
98:43
31.77
95.52
39.65
97.53
44.21
98.27
38.49
98.84
53.25
09. 00.00
33.41
(100%)
90.74
Mahacaater
98.60
37.83
98.55
31.90
95.64
39.09
97.65
43.11
98.36
37.98
95.84
53.82
09. 00
34.34
KNNN
56.81
$7.30
09007
$5.31
11.35
$2.00
14.59
50.24
72.14
$5.11
$1.40
$7.74
10.90
$5.39
KNNNc
56.73
87.75
10.31
36.11
71.89
$2.51
75.28
86.13
72.55
35.79
$7.43
71.50
88. 39
KNNNr
57.25
87.54
10.59
622
71.85
83.3 93.92
75:41
84
72.78
35
53.72
36.35
75. 10.
86
KNN
58.10 19
83.08
73.45
$5.95
85.92
86
KNNove
70.18
72.05
36.08
79.41
81:00 $1.60
85.28
75.00
$5.01
88.
KNN
71.06
80.54
75.08
35.77
80.61
$1.09
77.52
$4.95
75.87
$4.00
35.37
$4.87
79.20
87.99
Pretram on 1000% TFT DIPP
(pw linefunn))
KNNN
$7.50
34:40
72.34
05.01
35.47
30.23
40.01
30192
61.55
31.40
96.19
41.003
01.75
KNNNz
58.91
65.39
85:70
71.25
95.33
$7.19
96.59
45.06
91.63
60.47
34.88
95.50
48.40
94.12
KNNNsC
90.55
63.09
87.14
08.15
95.53
54.93
97.08
46.26
92.58
39.58
45.02
94.51
55.20
93.05
KNNNic
91.75
62.00
38.55
67.35
95.89
33.05
97.20
14.81
95.35
36.95
50.00
93.00
61.11 61.10
92.04
KNNNcc
91.91
60:13
39:35
06.23
95.17
70.78
97.00 40
42.85
95.1
3500
55.49
92.75
64.30
91.0 1..08
KNN
92.19
$9.33
90.12
64.47
97.56
41,40
96.09
5324
58.11 14
91.74
67.00 67.00
89.83
23

---

Under review conference paper at ICLR 2023
Table 5: OOD detection performance for document classification with different number of pretraining data from IIT-CDIP (remove pseudo OOD categories).
OOD Datasel Un-Demaini
OOD Dataset (Chii- Domainn)
ID
Method
Sc.
Report
Preseulation
Form
Letier
Arvrgge
SCL Foster
Reveigt
Ace
FPR95
AUROC
FFR55
AL KUR
1FR35
AUROC
FFRS5
AUROC
FPR95
AUROC
FFR35
AUROU
FPR95
AUROC
Pretroin on 10% DFFCDIP
F Finetune
KVL-CDIP
1D data
MSP
+2.43
16.31
56.35
09:35
54.31
70.25
47300
73.93
49.55
72.47
43302
76.35
04.10
75.65
ogsLoggt
41.91
91.27
55.04
99:33
54.19
85.20
64.97
90.93
49.05
69.18
18.37
94.27
41.33
91:35
Energy
41.83
01.20
54.92
19:35
54.11
85.22
45.01
90.90
48.97
99:31
18.37
94.30
41. 1110
91:42
AowNNor
90:15
91:00
54.04
06.91
$1.85
86.05
43.49
91.83
46.39
59.11
18:37
91:79
41. 11.40
91:82
LagontMM1la (108)
Molaassnn
20.04
04.77
45.44
01.53
04.20
01. 30
41.64
92.83
40.33
02.61
12.09
96.26
25. 63
96.03
05.89
Mala
40.11
85.80
52.87
78.32
55.11
78.51
62.01
71.56
52.52
78.55
25.58
90.55
40.50
87-88
KNN
31.03
04.25
$6.55
30.38
05.77
31140
30:53
02.75
41.34
02.13
15.42
55535
30.30
05:65
32.03
94.11
30.89
47.01
90.32
41.50
92.63
41.82
91.99
26.74
95.76
31. 10
95:44
KNNS
34.39
03:75
49.3 34
90,46
49.36
29.94
44.52
92.23
44.40
91.00
33.72
95.23
33. 20.20
95.35
36.15
93.47
51.27
90.19
$1.36
39.00 59.65
06.03
91.99
46.35
91.32
33.72
95.10
25. 10
95.15
KNN
38,67
93.15
$3.08
$9.95
53.99
39.35
89.31
91.78
49.91
91.00
30.88
94..38
33.50 90.90
94.95
KNNNar
80.11
92.87
54.92
$9.75
55.55
38.00 98.95
50.51
91.59
50.35
50.30 10.80
30.00 1.388
94.64
00.20
94.08
Pretrain on 10% ITFCCIPP
100 listune)
90393
72.30
94.26
15:49
90.54
72.38
94,40
67.33
92.24
09:37
40.34
91.36
36.00
75.05
KNN
91.59
70.54
94.98
63.91
91.65
70.74
94.51
65.95
93.26
67.78
53.49
90.41
57.60
73.51
KNNSso
93.00
67.76
95.55 54
61.24
92.78
08.27
95.25
64.01
94.16
66:32
35.81
58.37
53.50
71.05
KN
01.55
65:41
95.00
59.00 13.33
95.10
66.
95.54
62.41
90.52
63:28
67.48
36.44
60.20
69.09
KNNNoo
91.64
02.95
96.
66
93.24
63 75
95.70 72
000000 69
95 02
60:39
75.58
$4.12
63.
KN
05.32
55.89
96.67
53.42
94.53
00.24
96.14
57.55
95.66
$7.53
$0.23
50.73
66.80
64.45
Protrain on 30% ITFCPP
Finelune on RVI- 11 data
MSP
45.30
75.75
61:51
70:13
62.37
69:40
55.52
73.54
57.15
72.51
31100
77:00
50.70
75.00
MasLogit
41.03
91:57
54.00
$8.45
56.42
85.33 35.70
47.00
90.19
49.61
$8.98
38.37
93.62
41.80
90.56
Energy
60:05
91.80
53:76
05:47
56. 19
85.33 72
46.33 199
00.23
49.42
59
15.35
99.65
41. 13:70
90.30
GowNNom
35:15
91.89
54. 16
64.00
53.05
86 76
43:95
90. 04
47.07
98.50 03:52
40.70
90:41
43. 43.40
90191
Lagunt lose (20%)
Molaassnn
30.18
94:80
46 16
00.95
45.41
01.33 2.4
10:73
03.00
40.37
02.3 52
26.35 74
06 08
33.70
95.96
05.84
Molucaato
45.30
83.07
17.09
54.03
70 46
51. -.58
80.10 10
51.75
50.45
20:05
02.15
$1.70
92:00
KN
31.03
14.17
47.50
30.20
41.20
301.30
20055
41.34
01.27
11:40
55.25
$4.30
05:15
KNN
32.53
94.03
47.95
90.22
48.32
90.34
40:91
92.76
42.42
91.84
33.72
95.45
35. 35.40
94.97
KNN
35.71
93.67
49.74
$9.82
51.04
29.99
44.12
92.39
45.15
91.47
36.05
95.01
36.20
91.92
36.75
93.35
50.33
$9.00
$1.65
39:71
44.97
92.17
45.92
91.2
36.05
94.73
36.50
94.71
38,87
91.05
15
$9.38
53.51
39.3 99.39
47.48
91,94
48.00
90.95
38.37
94.55
38.00
94.45
KNN
80.87
92.75
55.40
$9.21
55.07
39:05
63.94
91.73
49.59
50 09
39.55
94.23
00.20
94.15
Pretrain on 30% ITFCCIPP
180 linttue))
90.39
75.25
79.39
79.43
95.14
72.41
97.12
66.95
90106
33.52
50.00
91.26
24.70
96.34
KNNK 90.63
73.75
30.47
79.51
93.81
70.5%
97.16
63.54
91.52
72.10
55.81
$9.91
26.90
95.94
KNNKso
91.67
71.19
$2.56
76,90
91.45
67.82
97.36
62.99
91.31
09.72
07.45
$7.29
290
95.31
91,95
09:1 09.19
83:73
75.35
95.33
65.3
97.36
50.84
92.09
67.76
70.42
$4.75
30.30
94.75
KNNooe
93:25
66.89
55.00
14.00
95.53
63.48
91.2 228
60
92.65
65 -6
$1.05
94.08
KNNum
56.78
57.77
07:44
93.52
61.70
$2.55
76.97
93.11
Protrain on 40% ITP
Fimetune on RVT (VPPPP T1 data
MSP
$1.76
75:76
62:39
99353
63.37
65.75
54.23
74.111
$7.54
72.04
55.81
71:20
42.50
R0U55
MacLogi
42.03
91.20
54.24
80.47
$7.30
$4.44
43.60
90.03
49.51
58.30
52.35
03.06
33.00
92.89
Energy
41.87
91.31
$4.20
$0.49
57.26
$4.47
45:50
80.05
49.71
58:53
52.35
93.13
32.32
92.92
GradNonn
38.19
91.65
$3.64
$6.85
55.03
85.00 56.66
43.18
91.45
47.51
$8.90
52.35
92.39
34.60
92.95
Laqent (40%)
Mahassen
28.90
94.95
44.72
91.14
41.97
91.47
36.36
85.61
39
92 02.30
24.42
96.22
22. 30
96.25
96.01
Mahaeaatar
39.83
87.21
11.91
79.81
53.15
83.13
46.83
85.33 28
47.95
$3.03
34.88
$3.36
26.00
92.78
KN
31.47
04.43
47.15
90.33
48.20
00145
S5.11
03.50
41.25
02.30
27.01
05.77
24.70
96.10
KNN
33.59
94.20
47.61
90.55
49.60
90.27
30.23 25
05.11 14
42.26
92.06
32.56
05.60
25.55 25.50
05.95
KN 34.87
03.95
49.33 00.50
90.10
52.11
89.87
42.29
92.75
44.69
91 91.66
38.37
95.16
26.40
05.95
KN 36.55
93.65
50.38
$9.82
53.55
89.57
43.71
82.51
46.05
91.33
43.02
94.89
27.70
95.77
KNNNan 39.07
93.37
$2.11
$9.54
55.47
39 29:25
43.55 82
92.26
48.12
91.10
46.31
94.62
30.60
95.57
KNN
41.47
91.07
$3.92
$9.25
57.38
$8.85
47.77
91.00
50.14
50.30
50.00
94.40
33.55
95.3 34
Pretrain on 48% UTFCDIP (ne lincumec)
IN
87.07
80.44
71.76
83.72
86.75
82.31
96.10
76.36
$5.42
50.71
75.58
$4.56
5.90
98.24
KN 88.93
79.03
74.95
$2.31
88.99
81.11
96.71
75.01
$7.40
79
$0.23
$2.56
3.20
97.93
KNNSs
91.47
73.23
80.39
79.50
91.78
79.75
97.40
72.60
90.26
37:37
87.21
79.19
9.00
97.92
90,75
75.27
$4.77
77.40
91.74
78.31
97.16
70.26
91.10
35.33
$9.55
74.11
14.20
97.09
KNNove
71
72.34
89.10
14.45
89.19
76 26:20
96. 53
67.90
90.59
12.59
91.30 13
69.01
23. 33.00
99 74
KNNun
89.87
67.51
94..22
6-4
89.86
71.59 1.
97.00
67.72
92.91
67.90
93.02
60.00
Protrain on 100% 1T1-2.00 101P
Vimslune em
RVT.-C D1P
11 dada
MSP
43.47
75.12
57:21
99:16
58.35
65 06:36
26.14
74.76
$1.20
72:15
15:11
75.87
25.
85.78
MasLagi
35.10 10
91.20
50.22
88.98
$3.19
$4.54
30.06
90.71
44.64
$8.58
24.42
06.30
23. 11.40
05.57
Energy
35.23
91.32
50.22
$9.00
$3.19
$4.55
39.96
90.73
44.65
$8.90
24.42
96.44
23. 33.40
05.58
GradNom
30.30
92.54
48.61
18.18
48.96
36.58
36.16
92.63
41.01
$9.98
19:77
96.71
19:20
96.35
Layel My (100%)
Mahassea
24.02
05.29
43.55
92.03
42.30
91.00
32.95
84.00
35.86
93:23
11.63
97.85
20.20
96.60
96.38
Mahacaatar
35.47
88.79
47.15
$1.21
50.32
82.1
43.71
$6.
44.15
$4.5%
13.95
97.15
58.50
82. 12.43
KKN
26.00
91.95
43.07
91.09
45109
90095
30.99
03.95
17.29
92.35
19.77
57.39
17.30
95.27
KNN
21.22
94.83
44.07
558
45.41
90 79
34.52
93.71
37.85
92.75
19.72
90:22
18.00
96. 25
KNNNo
29.65
12
50.45
93.93
40.25
90.95
17:46
57.00
18.00
96. 80
KNNNu
$3.15
94.26
50.64
90
93,11
43.8%
19.07
96.38
20.20
96.
KNN
34.35
94.00
49.17 50
90.55
$2.43
29.95
42.05
92.92
41.58
91.85
20.93
96.71
21.90
95.65
KNN
25.75
93.65
50.30
90.15
53.79
39.52
42,99
02.79
45.70
91.53
25.26
96.54
23.30
96.55
Pretrain on 1000 IITCCIIP" (n0 GusSune)
78.74
81.61
74.45
10.36
8033
33.71
10.001
11.33
$2.18
$0.30
38.31
54.02
17.70
96.12
80.39
80.13
77.5 25
79.31
83.45
82.75
95.45
75.93
$4.80
79.53
46.19
93.42
14.60
96.13
KNNS50
86.03
73.63
$2.50
76,60
86.91
81.30
96.10
73.07
$756
77.16
36.65
91.09
9.60
97:21
10
89.11
25.51
88.03
74.08
90.62
79.78
96.71
70.43
91112
74.95
06.28
$3.50
18.00
96.82
KNN
91.47
72.38
95.01
70.49
92:22
77.38
96.55
66.00 66
93.11
71:25
75.58
$4.60
34.50
95.79
KNN
93.07
61.61
96.95
85.71
46
25.15
97.96
61.59
96.21
6700
$0.25
78.99
51.20
95.83
24

---

Under review conference paper at ICLR 2023
Table OOD detection performance for document classification. Dpatial-RoBERTaan (Pre) denotes ap- plying the spatial-aware adapter in the word embedding layer. DpatialRRoBERTaaae (Post) denots applying the spatial-aware adaptor at the output layer.
OOD Datzert (In-oemain)
0OD (Out- (ummai)
ID
Repart
Prammtationn AUROC
Tatler
Aowagpe
SCL Pinter
Reccupi
Ace
FPRUS
FPR95
AUROC
ATROC
FFR95
AUROC
AUROC
AUTRCC"
Finetune on RVL-CDIP (ID)
MSSP
9111V
75.20
30.54
13735
51.83
71.35
91.00
12.35
-1.17
72.77
93312
30154
17-00
15.37
Matllggi
79.04
78:
77.45
77.93
79.99
Farrgy
07.23
99:07
77.25
77:51
80105
75.12
13.05
akene
72.44
81.76
90.19
KNNNo
KNNNa
35.
71.70
RollE
KNNNG
KNNN00
00.67
64.77
64.16
12.56
90.11
91.37
KNNNo
66.00
KNNam
26.74
63.57
$4.30
11:39
69.25
MasLogii
89.92
67.09
$0.60
86.70
Energ
46.48
99.94
48.50
58.01
93.45
50.60
89.71
25.77
20
42.19
V2.A5
39:31
23.00
90.55
10:47
11.10
02.93
42.59
92.64
43.43
11.63
$3.30
KNNN
34.39
43.53
92.36
12.79
92:51
44.27
91.63
15.12
49.70
45.04
47.04
91.31
17.44
KNNNam
37.43
46,00
91.50
37.75
43.30
45.65
90.94
20.93
Pretraln ITFC
Rineunee)
KNINo
78.82
78.92
79,95
73.
77.69
31.32
91.45
76.51
$2.00
77.56
10.47
98108
87.30
8039
KNNNa
79:20 04
77.95
62.64
72.17
79.81
50.40
91.13
75.11
96.41
16.28
97.60
92.10
76.
KNNNo
90.42
76.87
85.11
88.00 k3
82.12
78.93
97. 94.9
73.01
85.16
74.61
22.09
96.66
95.20
70.53
KNNN
81.43
75.20
86.90
67. 19
83.40
77.12
91. 58
71.07
36.78
72.97
27.91
95.36
96.60
64. 56
KNN
84.51
73,69
80.32
63.93
85.51
74. 44.38
94.07
68.53
33.34
70.13
$3.49
94.43
98.
$7.57
KNNaai
86.23
70.31
90.64
99.79
87.51
70.39
94.12
65.00
$9.62
66.38
58.14
92.50
99:10
48.95
on RVL-CDIP (ID)
65.00
61.81
27.59
14.10
32.85
Manlogii
57.82
90.04
34.88
73.37
Eurry
71.56
80:30
56.58
51.18
34.88
73.39
Spetiab-RollRR
Malaessa
98.71
34.37
95.98
34.73
97.83
32
33.60
100.00
100.00
15:31
16.85
97.10
KNNNo
54538
56.83
36.91
19.47
26.74
90110
KNN
56.52
43.09
26.74
79.57
KNNN
50.05
26.74
75.
51.22
29.07
53.05
75.81
on
MasLogii
Energy
94.10
97.37
S1.15
93.30
96
KNNN
42.43
92.90
38.07
91.5 01.58
32.02
25.5%
96.33
Spetat-ReBRRRTTe
KNNN
43.71
92.69
40.06
34.54
94.10
26.74
21.40
30.85
44.11
92.56
91.05
35.47
26.74
21.70
96.11
90.34
36.36
93.66
26.74
94.56
22.40
96.05
31.87
44,36
92.51
42.10
90.67
37.30
38.91
92.47
29.07
22.70
96.07
Pretralo IITECDDD
KNNN
15:23
86.33
15.11
15.31
16.57
15.38
$4.11
45:30
12102
KNDN
90.24
75.66
89.04
14.68
62.20
90.29
KNNN
92.51
63.82
80.17
78.72
90.42
67.84
71.72
$2.56
75.20
87.49
KNNNu
77.62
74.59
94.42
60.94
85.16
76.25
91.50
65.30
09.27
$3.88
88.20
84.95
KNNNa
81.43
96.06
50.24
87.67
75.01
93.51
59.67
66.04
87.21
68.30
94.90
81.47
KNNNa
87.15
97.71
$4.08
92.06
69.76
59.00
95.05
63.00
$9.55
64.54
97.5 77.80
78.05
25

---

Under review conference paper at ICLR 2023
Table OOD detection performance for document classification with the different number of pretraining data from ITT-CDIP.
OOD Dutaset (In- nomaoominn
OOD Dataset (Our Domain)
I0
Method
Sel. Repurt
Presentation FPR05 AUROC
Form
Letter
Set. Poster
Receipt
Acc
FPIRSS
AUROC
FFR05
AUROC
FFPR95
AUROC
TPR95
AUROC
FPR95
AUROC
FPR05
AUROC
Pretram 10% ITE-CIPP Fineluue RVI AVL-CDIPP (ID)
MSF
55,30
13.37
45.51
91.38
83.85
35.33
13.35
35.96
38.03
32.05
39.20
34.10
95.11
MaxLogit
50.36
91.51
37.77
94.30
87.97
53.89
51.05
91.47
38.36
94.24
25.60
96.05
Energy
50,56
91.48
37.08
96.35
63.09
87.89
91,00
51.38
91.42
38.36
94.29
29.00
95.
GndNomm
55.56
79:75
45.96
$4.39
66.91
74.07
53.44
61.07
56.22
79.52
47.95
$2.04
34.90
Mahagiun
06.00
64.51
67.35
62.51
94.20
66.65
91.88
70.25
95.11
65
96.63
54.58
98.
64-77
94.89
Maharaster
96.02
64.51
07.35
62.51
94.20
66.67
01.58
70.25
95.11
65.88
98.63
$4.59
98.
64-27
KNN
50.40
02.14
4333
05.32
51.00
911.4
14.47
13.57
35300
91.48
21.35
97.1
50.00
95.21
KNN
49.50
92.70
40.38
94.43
53.39
90.26
74.72
58.77
54.57
91.54
25.29
96.98
10.40
98.05
KNNN
46.72
92.89
34:27
95.24
56,17
89.
74.55
59.45
$2.90
91.62
27.40
96.56
12.90
97.50
KNN
45.48
92.89
29.35
95.67
57.62
89:55
75.04
58.25
51.87
91.59
30.14
96.21
15.00
97.57
WT
KNN
44,48
92.86
24.35
96.01
59.18
89.22
75.37
$9.30
50.84
91.57
34.25
95.93
16.90
97.35
KNN
14,32
92.76
22.95
96.17
60.81
88.82
75,24
$8.09
50.84
91.46
35.62
95.67
19.30
97.19
Feerram ITPP (80 Roclune)
KNNNY
06.97
R3708
97861
49730
99552
31.41
39.35
10.30
98.36
46.50
95.15
92.51
8.907
15.15
KNNNo
98.88
42.47
97.75
48.57
99.57
53.75
99:15
90.56
98.88
46.19
96.52
92.34
3.60
97
KNNNz
98.90
41:00
97.8%
48.04
90.52
52.91
99:55
58.63
98.88
45.92
95.89
91.80
10.60
97 65
KNN
98.76
41:20
97.79
47.700
90.45
52.92
99.55
58.01
98.84
44.81
98.63
91:31
14.50
97.41
KNNoze
98.48
40.72
97.65
47.338
99:44
51.74
99.31
57.54
98.72
44.30
98.63
90.83
20.60
97.01
KNNzce
98.28
40.11
97.55
47.02
99:45
51.05
99.31
56.42
98.66
43.64
98.63
90.38
25. 15.40
96.51
Pretramm on 20% ITFCDIP Finatune em RVV KVL-CCDIP (ID)
MSP
S3.70
31.R
Q1:11
BE-57
B5.25
101:1
55.57
$7.77
38.36
A0121
30112
14:23
15.75
MaxLagi
44.33
92.
36:21
94.18
64.92
87.63
58.56
91.31
51.50
91.92
45.21
92.63
99:70
94. 36
Eneny
44.36
92.17
37:89
94.24
66.56
87.51
60:39
91.23
52.30
91.38
46.58
92.62
41.50
94.15
GoalNum
50.51
54.92
92.04
51.67
94.20
45.41
09.
50:36
93-24
46.09
05.89
40.44
89:70
59:01
Mabasione
96.88
43.94
97.25
45.89
95.81
51.12
95.98
53.68
96.72
49.91
97.26
58.22
99.10
39.24
94.62
Makrewston
96.88
94
97.95
45.89
95.81
51.12
05. 00
$3.88
96.22
49.01
97.26
58.23
90.30 10
99 24
(215)
KNNNi
$2.30
02.58
45384
05.75
$1.75
90175
77.54
87.00
$7.42
01.02
17:1
97:35
16.90
97.20
KNNNc
$1.60
92.66
43.55
94.15
55.63
90.45
78.04
86.70 79
57.20
91.02
19.18
97.06
19.40
97.11
KNNNc
50.12
92.88
39.98
94.83
58.02
90.15
78.77
96.54
56.72
91.10
19.18
96.63
23.10
96.68
KNN
48.04
92.91
34.75
95.28
60.38
89.85
78.98
96.42
55.34
91.12
20:55
96.27
25.20
96.35
KNNNce
47.50
92.94
28:65
95.84
61.57
89.59
78.69
96.48
$4.18
91.16
26.03
95.95
25.00
96.11
KNN
46.58
92.99
26
95.79
62.88
89.30
77.60
$6.50
$3.48
91.13
32.88
95.67
25.70
96.15
Fretramn on IT--CIPP (no Risetune)
KNNN
55.76
41:13
97.3
47.12
95.33
33.05
55.11
33.75
98.2
45022
95.32
91.00
3,00
W7.3
KNNNe
40:71
97.51
46.19
52.52
99:31
$8.51
98.00
44.58
94.52
91.48
8:00
97.25
KNNNe
08.04
46.31
51.84
57.83
98.62
43.97
95.89
91.01
11.50
96 99
KNN
.5.08
53.36
49.58
97-26
90.95
96,
KNNozo
99.40
07.26
90.06
96.
KNN
45.37
09.31
98.63
Preeram on 40% CTT-CDPP Finelune om RVL-CDIP (ID)
KISP
35.10
32:21
51.38
01.05
35.36
35100
85.30
37.35
30.73
07122
45.80
Maxtagi
01:74
56.57
97.01
1.200
69.86
Eneryy
06.00
01.05
91.58
69.86
GralNum
47.00
$7.00
85.06
Maluxane
07.00
33.88
52.72
87.67
59.36
Malacaatun
52.72
87.67
KNNNI
75.61
34.25
KNNN
74.64
32.88
7.90
KNNNc
KNNice
73.03
91.2
39.73
97.76
58.17
47.05
97.51
47.56
51.97
$4.79
$2.19
KNNec
42.04
1.90
97.1
99.28
MaxLagit
GnalNum
64.21
Mahtauus
$8.58
59.5
62.59
94.79
96.53
59.22
03.50
62.59
KNN
$1.68
90.85
$2.39
91.5
$0.68
93.09
10.40
KNNNs
36.6
90.59
14.27
58.67
52.26
91.86
12.00
97.81
KNNNc
90.29
75.23
$3.49
51.72
91.39
57.53
92.23
15.60
97.45
93.15
27.52
58.24
90.02
76.18
$3.38
51.52
91.87
92.01
13.90
97.18
KNNece
93.05
25 25
89.72
76.06
$9.33
51.50
91.30
61.64
91.24
20,90
96.95
KNNuca
44.56
92,90
25.27
61.69
89.39
76:42
51.98
91.66
61.64
91.35
22.00
96.81
ITE-CDI
KNNN
$1.35
ST.TT
07.41
52.65
15
23.19
16.61
47.10
12:35
97:47
1.10
U5
KNN
49.56
52.01
46.67
15.07
98.31
KNN
96.34
98.00
51.20
58.05
42.24
96.34
46.16
21.92
2.70
98.18
KNN
97.00
96.90
49.31
98.12
50.65
58.13
42.00
92.04
45337
95.29
2.30
98.27
KNNsce
97.12
41.31
95 30
49.01
96.04
50.06
58
41.87
92.25
45.56
52.05
96.41
1.30
98-42
KNNILE
90.16
41.17
95.62
48.60
08.28
40.42
08.5
41.75
97.40
45.44
60.27
93.35
1.10
98.53
26

---

Under review conference paper at ICLR 2023
Table 8: OOD detection performance for document classification. Longformer denotes the original model adopted from the Huggingface model hub. Longformer (+) denotes the additional pretraining on IIT-CDIP.
OOD Dutasct In-Jommain
OOD Dataset (Our Drmnann)
1D Ace
Method
St. Report
Presentation
Form
Letter
Arenage
Sol Poster
Kecelge
FPR95
AUROC
FFR95
AUROC
FFRSS
AUROC
FFR95
AUROC
TERSS
AUROU
TPR95
AUROU
FFR95
AUROC"
Finclone ou KVL-CDP (IM)
MSP
15.00
64.32
05.03
02.11
05.35
001.5
23.35
66.85
55.12
03.45
35.31
77.30
35.10
34.72
MasLogix
97.12
72.84
97.07
75.22
98.24
70.39
95.52
77.57
97.00
74.00
90.70
36.62
99.60
68.10
Energy
97.43
72.82
97.35
75.21
98.36
70.37
96.59
77.55
97.48
73.99
91.86
36.63
09:80
68.05
Mahtesenn
58.21
04.95
$7.16
69.19
84.31
57.53
$9.54
62.47
$7.47
19.27
97.02
75.50
87.44
Mohnewwto
61.21
85.11
67.01
$1.93
71.99
78
63.19
$1.94
05.85
$1.79
20.42
94.13
26,80
8h.
90/71
KKNNO
33.6
85.21
00:05
10.30
07.80
33390
30.05
57.33
50.01
15.31
KNNNo
65.57
39.07
KNNNo
KNNNa
KNN
64.94
Io
20NN20
97.57
98.17
97.75
53.03
97.84
Energy
ITT-CDIP
KNNN
35.35
97.40
56.35
31.40
07.40
45.27
KNNNo
97.95
54.26
$4.88
07.50
45.55
97.95
98.62
53.24
44.15
97.99
97.34
52.32
98.50
43.09
52.01
97.44
$1.40
98.50
41.95
98.15
48.98
51.31
97.62
50.19
98.60
41.02
27

---

Under review conference paper at ICLR 2023
Table 9: OOD detection performance for document classification. All models are pretrained on ImageNet.
0OD Dalased In-Ibomainn
OOD Dataset (Out (nn-iimain)
Method
Preseniation
Poster
Receipt
FPR95
AUROC
FPRSE AUROC
FFR95
AUROC
TPR55
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
Pretrain ImageNNtt
Pinetune
RVL-CDIP (ID)
MISU
10.13)
17.31
77.35
65.22
56.75
03.10
15135
MasLogit
47.97
87.52
87.20
54,00
93.29
38.30
67.24
65.03
87.54
93.65
Matas
57.78
55.36
67.66
61.43
$1.25
$1.05
99.25
99.95
39.28
93.09
43.26
96.37
50.89
98.99
95.90
42.88
50.00
55.50
76.69
91.12
KNNNa
75.36
36.32
87.40
71.31
87. 14
92.50
77.34
76
84.96
6.96
98.98
KNNNa
74.90
36.41
66.20
87
73.00
87.
93.05
76.51
71.24
84.48
6.98
98.96
5.50
98.85
KNNNa
76.66
36.04
66.41
35.48
78.20
86.
05.50
74.76
79.22
83.02
5.81
98.68
5.90
98.70
KSINaai
77.54
$5.11
65.41
85 99
82.16
85.43
96.33
73.37
50.33
83.35
6.98
08.34
6.30
98.51
KNINasi
77.34
$4.97
64.89
89.41
64.44
14,40
96.88
71.70
$0.89
82.62
9.30
97.89
9.60
98.24
XX
77.98
$4.01
64.45
89.61
67.11
53.04
97.93
69.24
$1.85
$1.48
15.12
97.29
12.20
97.81
Pretrain ImsoeNett
KSNNa
96.96
51.14
B1.02
31.75
95.775
33.54
99.39
37.50
97.45
48.35
83.55
85.20
20.90
97.00
KNM
96.95
50.37
94.34
51.54
03.92
52.98
99.59
36.00
97.45
47.87
83.55
84.44
22.70
96.71
KNNNs
96.92
39.29
91.29
31.30
99.00
51.84
99.59
35.15
97.45
46.90
$3.55
$4.05
26.70
96.21
KNNNon
97.12
48.60
91.54
51.25
99.
51.11
99.55
34.36
97.59
46.35
82.
83.31
29 (0
95.67
X3
97.48
47.85
94.55
51.12
99.08
50.41
99.68
33.44
97.70
45.70
83.56
82.24
33.30
94.92
XNM
97.76
47.00
95.15
50.95
99.04
09.57
99.68
32.27
97.92
44.95
$3.55
80.92
40.10
93.85
Pretrain ImageNet
Pinetune
KVL-CDIP [TD)
MSSP
47.64
$3125
19:10
35.11
53.21
K3.11
30.28
$3.30
31.51
873.6
49.32
91.31
35.30
V3.63
MaxLLggit
42.39
93.11
42:47
93:45
59.02
88,
45.90
93.18
47.34
92.15
50.65
92.50
32.30
95.65
Energy
43.15
93.05
42.95
93.40
59.00
58.70
46.71
93.07
47.96
92.06
$2.05
92.38
33.60
95.49
Malasto
99.92
29.31
99.83
25.72
99.96
32.66
99.96
22.59
99.95
27.40
100.00
35.29
100,00
29.82
99.96
28.58
100.00
26.25
99.96
33.52
100.00
23.09
99.00
28.06
100.00
35.87
100.00
30.45
95.74
KNNNr
19.11
92.32
36.73
92.1
$2.500
91.57
72.29
$8.45
$2.01
91.65
16.35
96.75
6,10
98.30
KNNNa
48.86
92.55
43.27
93.51
64.53
92.32
72:28
$3.15
52.25
91.75
17.1
96.52
7.40
9%. 10
KNNNo
46.46
93.26
39
94.57
47.41
92.09
73.34
$7.87
51.61
91.95
26.05
96.11
5.60
97 90
KNNNon
43.7
93.42
35.03
95.29
50.08
91. 73
35.77
67.42
51.16
91 96
38.77
95.06
11.90
97.5 85
KNNNam
41.59
93.56
27.84
95.85
$3.10
91.45
76.66
67.13
49.57
92000
27:40
95.75
13.20
97. 07:51
KNNNa
40.79
93.68
24.71
96. 14
53.27
91.17
77.52
$6.07
40.07
91.99
26.03
95.72
16,00
97.18
Pretzaii on Imnggeea
XNNNu
98.56
52.75
95.06
55.14
99.36
58.85
99.80
41.56
95.20
$2.15
65.75
93.26
2.10
99.35
XNINa
98.44
$1.86
95.18
54.72
99.32
57.88
99.80
40.56
98.18
51.25
65.49
92.52
2.00
99.22
XNINs
98.52
$0.69
95.38
54.13
99.16
56.61
99.76
39.01
98.20
50.11
75.08
91.14
3.00
98.99
XNINoo
98.72
49.56
95.66
33.80
99.16
55.84
99.76
35.16
98.32
49.44
79.05
4.30
98.77
KNNNan
98.08
49.22
93.90
99.
55,
59.76
37.31
98.3
48.76
84.95
$8.40
5.00
98.47
XNNNa
08.84
40.36
96.22
52.74
99.40
$4.41
99.76
36.26
98.56
47.94
$4.95
$6.74
5.20
95.05
Freteain on Imnggeeet et
Panclune 68
KVL
MSP
36.21
$9.14
$2.19
91.80
67.43
64.25
39.00
$3.77
39.10
85.99
47.07
92395
59.30
91.99
MasLogit
30.76
91.37
41.60
93.75
63.04
86,
55.15
91.51
$1.64
90.7
40.70
94.20
52.00
93.15
lizergy
31.16
91.31
44.52
93.75
09:43
86.81
36.09
91.77
35.
90.11
38.37
93.11
53.20
93.11
Malaston
90.63
70.10
01.84
65.75
89.55
70.83
97.81
57.57
92.46
6600
100,0
58.34
82.20
77.09
90.63
70.10
01.80
65.75
89.55
70.83
97.51
57.57
92.45
66
100,00
58.34
82.20
77.09
94.38
KSNNon
02.77
20.12
37.73
90391
$3.00
B0.35
$4.50
36.19
01.02
39.40
12.79
97.56
13500
07.32
XNNNas
63.01
90.24
56.01
91.51
55.03
90.02
$4.38
36.01
64.61
39.44
15.12
97.76
14.90
97.67
KNNNo
61.97
90.62
53.23
92.62
59.26
89.3
$4.25
$5.54
61.45
39
16.28
97:38
19.80
97.24
KNNNon
90.85
39.30
93.55
$3.01
$5.43
63.00
39.72
16.26
92.05
23.00
KNNNan
45.04
67.88
$5.33
62.64
3933
23.09
27.50
9%
KNNNas
65.008
61.85
96.03
Imaagr
KNNNas
95.02
$3.77
XNINa
95.06
52
XNINu
07.26
XMMNu0
47.85
95.76
XNNNao
36.30
95.15
28

---

Under review conference paper at ICLR 2023
Table 10: OOD detection performance for document classification (select OOD categories achieve the best performance across most of the models with different modalities).
Datasel Domaini
Domain)
Method
publication AUROC
Recelpt
TPR05
FFR95
AUROC
AUROC
pare-test
MSP
35
71:73
03.88
01.50
65.51
64.23
10.57
03.03
69.99
Mislagit
99.21
65.17
74-69
Eaerg
99.60
73.57
05.14
07.21
65.35
68.07
94. 10
74-90
99.24
35,35
09.39
32.50
33.05
42.56
99.70
31.42
99.24
35.49
99.29
33.02
99:70
31.49
ROBER
56.13
KNNG
84.50
83.70
82.35
54.21
75.57
75.75
73. 13.80
72.00
35.37
85.31
KNN
84.98
54.00
$7.08
82.25
75.14
54.55
87.
KN
86.25
$0.85
57.40
83.05
77.24
58.14
91.50
78.20
85.65
KNN
86.72
$0.01
83.43
57.61
83.07
77.00
59.30
78. 08.90
94
Pretrain dasta
KNNNaa
86.09
75.65
05.12
58.60
97.71
59.75
58. 055
50.55
94.47
61.14
10:47
98.46
80-80
63.01
KNN
86.29
74.02
95.00
58. 14
97.71
58.85
09. 00.08
49.49
94.51
60.36
12.79
98.35
90.80
60.59
KNNGs
87.32
73.55
94.64
57.5
97.83
57.56
09. 15
48.11
94.73
59.19
12.79
98.11
93.
$6.61
89.27
72.48
94.28
57.12
97.99
56.52
99. 11
47.37
95.16
58.57
11.63
97.59
94.33
$2.95
KNNNe99
90.26
71.07
$3.96
$6.45
98.11
55.37
05.75
46.50
95.27
57.55
13.85
97.64
95.55 75.70
45.45
Fretriit pare- lest duta Fineeuse on RV
CDL 113))
XISP
96.90
61135
56,20
59.14
96.31
35.72
97.82
15.12
95.81
37.63
15.35
50.14
95.03
52.82
MaxLogit
98.97
08.97
97,00
05.64
95.67
63.42
98.63
62.57
97.72
65.23
97.67
58.42
99:70
71.54
Energy
99.88
08.96
97.92
05.03
95.83
63.42
98.71
62.83
97.95
65.21
97.87
58.46
99.
71.55
Mahasoon
97.85
41.33
95.88
45.36
99.93
41.95
96
95.35
05.54
94.11
09:02
977 800
53.64
92.97
15.08
16.90
44.99
95.59 90
22.08
68.11 17
600.3 07.37
95.23
43.96
550 97
60.33 34
95% 50
57.51
Longfrrme..co
88. 94
KNN
KNN
68.04
88.76
$8.61
35.72
70.10
K1.76
83:22
77.53
76:17
8511
84.92
00.75
00:35
74.95
76.00
75.74
82.90
16:28
96.55 64
35-30
81.33 11
86.49
KNN
69.28
$8.20
70.98
83.02
78.20
84.45
00.06
74.92
77.38
82.62
10.77
96.5 59
83. $3.40
85.71
69.28
85.15
71.34
$3.60
78.49
84.21
00:43
74.96
77.30
82.45
22.00
96.58
83.
85.17
KNN
69.44
35.06
72.14
$2.43
78.65
$4.00
$9.58
74.74
77.45
82.51
23.36
96.16
84.70
$4.60
Pretrain on pare- text datla
K
97.02
47.27
95.72
30,09
97.07
46335
99.32
28.01
97.35
43.76
45:35
92.92
100003
63.03
KNN
97.45
46.91
25.00
49.00 80
97.71
46.02
99.52
38.21
97.57
05.24
46.51
93.77
100.00
61.92
KNN
97.58
45.68
56
69.03 43
97.75
45.19
59.52
37.92
97.00
04.51
50,00
90.00
100.00
60.35
KNN 59
97.65
46.78
49.17 17
91.89
46.63
59.
39.57
97.867
04.04
51.1 16
99.88
100.00
58.89
KNN
97.85
43.79
95.72
48:71
46.05
37.38
97.26
43.46
51.16
09.42
100.00
57.02
Pretraio on ImuggeNet Inctune 08
RVL-CDIP (DD)
XISP
00.33
37.26
09.33
$7.00
27.95
95:13
$4.005
15.79
02.95
$5..00
74.30
27.30
95.47
MaxLogit
59.98
$9.27
72.61
$3.00 03.00
30.04
95.41
93.39
15.38
64.00
57.002
50.82
79.500
30.00
95:29
Energy
63.21
$9.14
15.64
83. 5355
45.71
94.15
52.77
05.00
05%
78.00 000
$1.007
02:
Mahassome
76.11
93:47
(1.76
87.52
59.
60.000
77.
11
12
0.20
11.67
8:54
99.00
26
91
85:05
KNN
72:46
55.35
RoNN4-50
KNN
76.15
56.31
76.44
56
KNNG
80.37
59.06
77.73
47.95
3.90
$4.70
50.64
51.29
91.75
97.
61.19
75.54
50.68
4.70
95.91
KNN
78.01
78.32
57.53
Fistrala ImungeNet
KNN
$4.93
KNN
41.18
66.30
64.90
Imoge
RVI
IVI. (ID)
70.25
R5:31
$3.75
70:40
15.82
R6.30
78:23
54.10
91.62
MasLogi
54.75
87.04
92.30
17.25
14.11
$7.49
82.19
63.20
94.82
$4.05
92.49
16.35
13.59
$7.46
$4.93
94.82
99.72
28.96
18.92
99.73
44.10
99.90
38.05
99.05
29.74
100.00
20.22
99.51
43.99
29.14
100.03
39.09
91.25
K
36008
92.34
95.31
91.02
55.55
15.93
3.30
95.35
KNN
56.61
90.95
64
27.55
95.24
91.0 0..44
68.54
55,00
85.85
26.00
4.00
98.76
KNN
55.25
90.68
92.37
15.25
97.28
91.33 25
31.62
53,00
$7.99
28:
10
98.55 59
KNN
56.20
90.31
$4.75
92.17
9.14
96.00
91.
75.01
52.90
85
50.
98.35 35
KNN
$5.64
90.00
$4.24
97.11 14
8:90
14
00 00000
17.40
$2.20
89.52
03.58
95.49
98. 12
Prstrain
KNNN
99384
43.55
91.76
30:64
47.92
93.20
98:01
37.55
85.61
43.74
53.00
93.35
1.60
99.32
KNNgz
99.84
43.78
99.76
19.61
44.76
03.61
98.91
37.01
85.82
43.50
65.75
93.42
2.10
99.20
KNNG
99.84
44.47
99.80
18.36
41.31
94.14
99.03
36.45
85.00
49.36
72.60
92.69
2.60
99.00
K3NN
99.85
45.26
99.80
17.92
39.97
94.39
99.03
36.71
84.07
45,57
79.45
91.97
3.70
95.81
KNN
99.85
45.90
99.72
17.52
36.64
94.64
96.76
37.33
83.75
45.55
50.52
91.29
4.40
98.35
Pretraim ImmggeNei
Vineluue
RVL
DIVE-CPIIP (1D)
MSP
61.25
35.84
66:57
83.04
40.44
03.10
55.54
$1.33
63.52
85:45
73.97
30:06
60:30
90:41
MasLogii
53.02
90.37
55.77
55.55 56
19.91
96.25
92.38
79.59
55.27
85.79
76.71
85.16
50.60
93.12
Energy
$1.79
90.49
53.07
99.03
17.53
96.53
92.00
79.20
$4.27
85.51
79.45
55.01
50.10
93.20
Mahasuuaa
97.26
45.71
93.89
56.52
96.76
37.35
97.47
63.36
95.34
50.51
100.00
55.27
87.20
65.24
Mahamaaeen
97.26
45.71
93.89
56.52
95.76
37.35
97.47
63.36
95.34
50.51
100.00
55.27
87.20
65.24
$9.97
KNN
5K.15
91.18
$2.56
91.18
55.89
87.45
92.55
05.36
64.50
$3.95
$3.07
95.07
11.00
97:04
KNN
56.21
91.18
53.17
90:59
30.61
89.35
90.04
60.52
62.75
$5.76
43.54
94.99
13.10
97.62
KNN
56.53
91.05
$1.33
90: 09
41.95
92.83
90/00
72.06
60.70
55.55
43.47
944 04
17.90
97. 12
KN
54.65
9881
$4.12
00.00 56
30.70
98.76
09.044
75.00 30
55.15
05.
45:31
04.33 244
13.00
96. 58
KNN
56.00
90.51
$5.42
00:40
15.31
97.11
02.61
77.76
54.84
05.05
43.05
99.51
27 40:40
95.93
KNN
57.35
90.11
55.77
00.30 36
11.56
97.45
02.49
78.73
$4.20
89.16
56. 16
00.87
34.00
05.51
Pretraie on Net
KNN
09330
45544
99.03
20.50
35.05
90561
30.72
46,40
89:21
$2.40
81.07
01.35
19.00
07.25
KNNNs
99.80
46.02
99.65
35.09
57.30
91.01
98.72
46.46
85.87
$2.30
90.41
90.57
21.70
97.01
KNNNs
99.80
45.48
99.61
24.76
35.10
91.52
98.76
46.09
85.53
$2.11
94.52
59.59
24.30
96.62
KNNN
99.80
45.35
99.65
24.43
34.81
91 $1.90
98.72
47.10
88.24
52.19
89
89.31
28. 28.90
96.27
KNNaas
99.68
45.19
99.65
24.18
53.65
92.27
98. 04
47.72
S7.91
$2.54
90.26
$3.57
32.00 00
95.5 88
29

---

Under review conference paper at ICLR 2023
Table 11: OOD detection performance for document classification (randomly select four categories as OOD).
ODD Batseri (In-Demain)
GOD Datasii (Ohi Dumain)
ID
Method
Letter
Handwwitten
Advertisetical
Memoo
Avvnge
Scl. Foster
Recelpt
Acc
FFR05
AUROC
FFR995
AUROC
TPR95
AUROC
FPR95
AUROC
TPR95
AUROC
TPR95
AUROC
FPR95
AUROC
Pretrain en pure-test data - Finetune em RVI-CDIPP (ID)
MSP
70.23
70:21
50.50 14
67:34
84.64
67. 67.80
91:42
57.99
14.10
73106
95.58
59.75
94.
55.12
agil
66.04
87:51
19.65
02:53
86.47
17.05
91.00 0/.77
71.84
70.96
$2.35
100.00
77.50
96.80
71.95
Faergy
66.20
67:57
18.19
02.59
87:35
77.05
91.67
71.89
70.85
$2:27
100.00
77.02
96.80
71.95
87.40
42:73
59.45
35.19
96.57
39.05
99.03
56.06
93.11
38:25
98.84
34.76
07. 07:10
57.71
89.07
60.42
69.35
$3.53
95.60
36.59
99.03
35.87
93.54
37.10
98.84
33.06
07.10
57.49
88.96
KAN
00.02
80115
00.36
10.30
1502
30121
R3.54
107.30
11:26
15:33
55.15
$1.30
500.0
R2.45
KN
63.18
80.10
00.07
71.17
75.90
80.05
85.72
65.55
71.22
75.04
90.19
91.75
91.23
81.89
KNNS
63.78
80.00
57.30
71.70
76.34
79.67
83. 53
65.35
70.82
74.94
90.19
$1.45
01. 10.80
81.09
64.77
79.95
54.35
71.94
71.37
79.32
$6.00
157.00
70.64
74.26
19
$1.20
01. 14
80.R7
KNNoe
64.85
79.85
77.57
78.
$6.33
50,
71.10
74.33
$0.08
91. 11.00
79.59
Fretrain on pare- 4est da10)
K
85.57
50.30
00.51
21:75
06.72
07.53
53.30
11.31
17.79
KN
57.05
12.79
KNNG
20.45
55.35
13:05
97.40
890
50
88.47
19.97
53.00
95.57
13:05
97.38
91.
KNNae
19.50
52.53
95.94
44.82
19.77
50.60
Pretrain on pare-dest dasty
Finetune RVI-CIIP
MSP
63.00
00.55
30:35
77:01
81.31
00:21
77.002
65776
37.30
05.18
MaoLagi
79.97
28.84
80.35
77.35 36
Eaergy
61.27
43.61
19.81
79.13
78.85
98-15
30.33
Maltesson
43.64
96.36
44.99
95:35
40.73
MahsccaMars
94.58
32.97
96.83
36.00
95.35
37.84
98.35
59.91
92.08
Io
30.77
71:51
90.57
75.19
64.21
17.51
35.14
90000
77.76
Longermetan
57.81
71.72
81.15
17.34
58.14
77.12
KNNN
58.77
71.67
61.31
74.30
01.65
76.17
52.75
81.06
75.
KN
$3.94
82.77
67.22
63.95
Fretruia pure
23.35
93.20
$00.00
100.00
KNN
90.24
100.00
KNN
90:41
97.81
51.16
61.55
Pretroin EmtggeNtt Finetune on
(1D)
71.15
MaxLoggt
64:45
Energy
K
10:17
17.35
$4.40
10:700
93.70
KNNNa
92.61
90.85
54.52
95.05
RecNet-00
KNNNG
28
94.49
39.27
92.78
$4.41
92.09
KONN
27:21
39.78
94.22
53.
91.10
KNNNaoo
71.92
28.08
94.97
90.26
95.79
68.00
56.40
90.30
Pretroin
08.30
42:21
77.29
51.41
21387
91.15
99300
43.47
75.88
69.56
31132
39.58
12.30
95.17
KNN
41.00
96.7%
$1.50
29.22
92.27
99.00
62.29
64.32
$3.56
39.30
14.10
97.97
KNNNo
39.51
76.55
61,81
31.01
92.05
99.
63.55 35.55
$3.56
58.51
16.30
97.61
KNNNn
67
38.60
77.83
$1.49
32.64
91.84
99.
59.000 86
76
62. 62.95
83.56
$7.50
19.50
97.23
KNNas
78
97.57
70.05 34
$0.36
35.19
91.51
99.228
$9.79
78:
62. 18
30
36.70
25.
96.27
Prefroin EmtggeNet Finetune RVL-CDIP (1D)
MISP
$3.90
87.00
85:21
$3,15
73.56
79:72
61.40
15.355
05.125
$5.34
$3.95
74.35
92.49
MaxLogt
56.09
90.22
15.50
87.00
22.64
84.30
44.22
93.01
62.16
55. 62
72.60
84.16
29:10
95:20
Energy
57.41
90:11
77.01
86.00 00
23.44
84.30
93.06
62.55
58, 46
73.97
$4.25
25.
95.69
Maltisuen
99.96
27.23
99.000 09
49.97
43%
99-96
24.62
999
16. 90
100.00
60.31
100.00
28.99
Maln
99.06
27.44
09.00 05
49.56 56
99.72
43.
24.86
99.67
36.43 16.44
100.00
60.54
100.00
29.29
02.42
KNNN
60:27
0012
66.90
90.76
29:65
39:TS
47.67
92.87
56.12
00085
42.47
94.23
7.20
95:35
KNNN
60.32
90.01
61.37
91.31
48.83
910.5
40.00
92.53
55.13
91.00
30.14
95.56
5.80
98.33
Swithue
KNNNA
62.62
62.22
89.75
89.60
$4.98
$6.44
91.35
91.56
50.70
50.34
88.
89:55
47.63
48.52
92.18
92.30
54.35
53.58
90.30
90.64
26.03
30.14
95.54
95.72
11.80
13.90
97.97
97.65
KONN
65.77
89.65
56.57
91.63
51.39
98.34
47.63
92.24
54.49
90.47
35.62
95.15
15.90
97.32
Preiroin on ImageNet
KNNN1o
99.15
45.37
$6.00
79.44
32.45
30105
19:33
46.30
19.25
65.35
34.65
90.34
0.40
195.75
KNN
99.19
44.11
56.3 39
50.35
33.45
92.19
99.50
44.79
79.79
65.36
27.40
95.62
0.50
99.73
KNNN0
99.23
42.39
$7.99
$1.66
36.76
91.59
99.50
43.07
$1.50
64.68
43.84
94.57
0.80
99.63
99.19
41.45
99.02
$2.63
40.60
91.05
99.50
42.14
$2.10
64.32
$2.05
93.49
1.20
99:53
KNN
09.23
60.61
90.83 58
$3.30
45.41
90.40
99.00
41,26
$3.78
03.89
05.75
92.17
2.10
99.35
Prefroin en NmtgeNNt Finebune on RVL
CVV- (TD)
MSP
297.57
$1.38
72.25
31178
09.35
15.01
71122
$5.32
07:12
$5.97
50.30
ManLogit
63.35
89.20
68.40
$9.58
69.55
84.35
61.08
$9.94
65.60
59.02
57.55
89.41
45.40
93.04
Energy
62.22
89.21
70.34
$8.43
70.26
$4.37
60:75
90,00
65.89
$9.01
59.50
89.47
49:30
93.03
Mahtouee:
92.05
67.35
92.35
15.45
92.09
62 29
92.30
66.35
92.20
05.54
97.26
55.92
85. 50
72.82
Mahu
92.05
67.35
92.33 20.34
15.45
92.09
62 99
92.10
66.35
92.20
65.54
97.26
$5.92
85, 60
72.82
91.00
KNNN
68.10 10
$3.99
$4.90
92.30
53.44
$5.05
58.1 19
91.34 34
58.06
90.17
38.36
95.00
22. 90
95.75 71
KNNNa
67.61
85.95
49.01
92.85
51.53
89
53. 50
91.11 16
16.68
90:55
41.00 11.10
94.44
25. 40
96. 35
VIT
KNNN
67.20
06.10 10
81.91
83.90
41. 511
42.5 54
03.10 19
03.15
$3.96
55.71
85. 4%
87.75
53.77 75
50.11
90.00
90.64
56.30
64.64
50.3
00.12
45.21
42.47
02.50
07.60
70
34.00 00
95.76
95.27
KNN
66.07
85.96
47.27
92.73
58.57
$6.99
60:43
90.50
58.08
59.33
46.58
92.12
35.55 $5.50
94.72
Peetrain on Net
KNNS
98,90
41.95
90.96
77.15
34.87
90.69
99.40
43.21
31.03
62.76
$4.79
94.27
10.80
98.47
RNNN20
98.94
40.54
91.67
77.20
36.82
91.71
99.44
39.85
$1.72
62.32
64.38
93.57
12.70
95.25
KNNS
99.07
35.75
92.51
76.99
40.00
91.17
99.52
35.14
$2.80
61.26
75.34
92.47
15.90
97.87
10N
99.11
37.43
99.23
76.56
43.35
90.68
99.56
36,93
$3.82
60:40
82.19
91.52
18.90
97.09
KNN
95.11
35.99
94.27
75.93
66.92
90.11
99.56
35,50
$4.96
59 40
$9.04
90.46
22.90
97.01
30

---

Under review conference paper at ICLR 2023
Table 12: OOD detection performance for document classification. All models are pretrained IIT-CDIP. For LayoutLM models, we adopt the checkpoints from the Huggingface model hub. For UDoc, we pretrain the model our side. All models are finetuned RVL-CDIP ID data.
OOD
GOD
Method
Procudaaino
FP1855
AURRCC
FPR5
AUROC:
AUROC
FPRU5
FFRUS
AUROC
FP399
FP3095
AUROC
Maxl
24.42
Lagont Molus
Mahatcon
05.01
90.44
38.25
Energy
LigonLMMe
36.56
KMN
21.03
KNN
Energy
Maluscons
KNN
KNN
KNNN
44.11
42.09
92.79
31


Reviews:
[{"review_id": "tGZRv5fe5f6", "reviewer": "Reviewer_oNdB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides analysis of out-of-domain detection for different tasks in document understanding domain, including document classification and information extraction. Authors create a benchmark dataset for out-of-domain detection and study the performance a 4 main categories of models for document based on text, image, text+layout, and text+layout+image.", "summary_of_the_review": "Overall, this paper needs improvements to meet the bar of ICLR. Specifically, the in-domain vs out-domain data needs to be more rigorously defined and experiments designs be modified to make it easier to understand the impact of model architecture vs the data domain, thus making the conclusions more reliable.", "strength_and_weaknesses": "Strengths: \n* This paper provides a first benchmark dataset for out-of-domain detection for document understanding.\n* It provides comprehensive experiment studies on the different categories of multimodal transformer based models for OOD detection.\n\nWeaknesses:\n* The distinction between the in-domain and out-domain types of OOD data is loose. There should be a more rigorous way of defining this two categories based on some concrete similarity metrics.\n* The experiments design is not convincing. For example, the impact of fine-tuning on improving the OOD detection is not surprising, especially as you have defined the in-domain OOD data based on the classification accuracy in the first place.\n* The conclusions drawn from the experiments are not rigorous enough (e.g. Figure 5). For example, in zero-shot OOD detection, the architecture of the model seems to play a more important role than the OOD data being in-domain vs out-domain.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lEOTwcObZRE", "reviewer": "Reviewer_Dbcz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary: The authors aim to provide a systematic and in-depth analysis on OOD detection for document understanding models. They deduce that spatial information is critical for document OOD detection. The authors also propose a simple yet effective special-aware adapter, which serves as an add-on module to adapt transformer-based language models to document domain. They perform experiments to show that their method consistently improves ID accuracy and OOD detection performance compared to baselines. ", "summary_of_the_review": "Based on the weaknesses in experimental setup, I vote reject for ICLR 2023. ", "strength_and_weaknesses": "Strengths\n1. The authors position the paper as a analysis of OOD for document understanding domain. Given the practical implications and prevalence of documents, this is important work.\n2. The authors evaluate recent transformer based document understanding models for ID and OOD behavior and try to made deductions based on experiments. \n \nWeaknesses\n1. The authors inherently start with an assumption that OOD detection is multi-modal and that single modality cannot solve OOD. They do not show experimental justification on this assumption. Why is a single modality not sufficient? \n2. Miss citing relevant document understanding papers see below. In fact DocFormer has the sota performance on RVL-CDIP dataset (main considered dataset in this paper) and is not cited, nor discussed.\n3. Page 3 - the notation does not seems correct. If threshold gamma decides ID vs OOD, it is possible a model f, wrongly predicts a label for ID test-data. the authors do not dis-ambiguate this scenario in notation.\n4. Page 4 - the authors miss gradient based OOD detection. Logit and distance based are not the only broad research directions for OOD.\n5. Fig. 2 (a) is not entirely correct. e.g. in Spatial+Vision+Language have spatial embedding layers. The authors do not make it clear how this is different from their spatial-aware adapter and in which way it benefits over the embedding approach.\n6. The deductions made in Sec 3.1 seem faulty and has some un-answered assumptions and . e.g. for receipts the authors claim “improvement of fine-tuning is less significant for out-domain OOD data.” It’s possible that the pre-training data had several receipts which led to ViT exhibiting this behavior. Also why was this experiment only done using vision-only model? its possible text/multi-modal exhibit different behaviors\n7. Page 7, Zero-shot performance. The experiment seems faulty. The IIT-CDIP data the authors use, have the data which the authors deem OOD although in un-labelled form (IIT-CDIP has letters, forms, etc.) For strict OOD zero-shot performance, shouldn’t the authors remove such data? Please comment.\n\n[1] DocFormer - end to end document understanding using transformers ICCV 2021\n[2] more recent document understanding papers also not cited. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "963WYH4kOC", "reviewer": "Reviewer_4z2u", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors analyze the characteristics of spatial information in document understanding and propose a framework to leverage spatial information for document OOD detection. They start from an analysis on the effects across different OOD types and emphasize the importance of spatial information for document understanding. Following the observation, the authors propose to employ an add-on module and adapt transformer-based neural language models to document domains. Analysis and experiments on RVL-CDIP and IIT-CDIP demonstrate their observation and the improvement of Spatial-aware models over the conventional ones.", "summary_of_the_review": "In sum, I would recommend “8: accept, good paper” based on the extensive analysis and experimental results. Although there are some flaws in presentation and clarity, I believe the authors should be able to address the issues in the final version.\n\nEdit: After the reviewer discussion, we decide to change to the score to \"6: marginally above the acceptance threshold\".\n", "strength_and_weaknesses": "Strengths\n* In-depth analysis shows the motivation and the potential of leveraging spatial information for document understanding.\n* Extensive experiments not only demonstrate comprehensive comparisons, but also indicate that spatial-aware models outperform conventional models.\n\nWeaknesses\n* Writing needs to be significantly polished. The paper is not very easy to follow. Plus, there are many typos like “special-aware”.\n* The authors do not mention the plan of open-sourcing their implementation of spatial-aware models and analysis scripts for the reproducibility.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject