ADOPD: A Large-Scale Document Page Decomposition Dataset
Authors: Jiuxiang Gu
Xiangxi Shi
Jason Kuen
Lu Qi
Ruiyi Zhang
Anqi Liu
Ani Nenkova
Tong Sun
Venue: ICLR 2024 poster (2024)
Keywords: Document Understanding
Dataset
Segmentation
Detection
OCR
Captioning
Primary area: datasets and benchmarks
TL;DR: A Large-Scale Document Page Decomposition Dataset

Abstract: Research in document image understanding is hindered by limited high-quality document data. To address this, we introduce ADOPD, a comprehensive dataset for document page decomposition. ADOPD stands out with its data-driven approach for document taxonomy discovery during data collection, complemented by dense annotations. Our approach integrates large-scale pretrained models with a human-in-the-loop process to guarantee diversity and balance in the resulting data collection. Leveraging our data-driven document taxonomy, we collect and densely annotate document images, addressing four document image understanding tasks: Doc2Mask, Doc2Box, Doc2Tag, and Doc2Seq. Specifically, for each image, the annotations include human-labeled entity masks, text bounding boxes, as well as automatically generated tags and captions that have been manually cleaned. We conduct comprehensive experimental analyses to validate our data and assess the four tasks using various models. We envision ADOPD as a foundational dataset with the potential to drive future research in document understanding.

Published as a conference paper at ICLR 2024
ADOPD: A LARGE-SCALE DOCUMENT PAGE DECOM- POSITION DATASET
Jiuxiang Gu
Xiangxi Shi Tong Sun'
Jason Kuen'
Lu Qi
Ruiyi Zhang'
Anqi Liu
Ani Nenkova'
Adobe Research
"Oregon State University
3 UC, Merced
4 *Johns Hopkins University
Figure I: Overview of the ADOPD dataset showcasing densely annotated images of various docu- ment types and layouts. Each column presents the original image alongside visual entity masks and annotations of text bounding boxes, organized from top to bottom.
ABSTRACT
Research in document image understanding is hindered by limited high-quality document data. To address this, we introduce ADOPD, a comprehensive dataset for document page decomposition. ADOPD stands out with its data-driven ap- proach for document taxonomy discovery during data collection, complemented by dense annotations. Our approach integrates large-scale pretrained models with a human-in-the-loop process to guarantee diversity and balance in the resulting data collection. Leveraging our data-driven document taxonomy, we collect and densely annotate document images, addressing four document image understanding tasks: Doc2Mask, Doc2Box, Doc2Tag, and Doc2Seq. Specifically, for each image, the annotations include human-labeled entity masks, text bounding boxes, well automatically generated tags and captions that have been manually cleaned. We conduct comprehensive experimental analyses to validate four tasks using various models. We envision ADOPD as a foundational dataset with the potential drive future research in document understanding!
data and assess the
INTRODUCTION
Document understanding has been invigorated by the introduction of large-scale document datasets (Zhong et af., 2019, Mondal et al. 2020 Cheng et al. 2023). supporting a variety of document-related tasks (Mathew al. 2021 Mathur et al. 2023). However, document datasets still fall short compared to data resources in more established fields (Gu et al. 2018), in which advances have been so great that models and solutions can be incorporated in real-world applications. A case in point is the field of image decomposition, where progress was fueled by datasets like MSCOCO (Lin et al. 2014) and Pascal VOC (Everingham et al. 2010). Building document page decomposition dataset of comparable quality is essential to advance document understanding research.
"Correspondence to: jiqu@acobe.ccm com
Projecc page: httpt::t/////tt..tt pectps://adop6222// gi thub. 10

---

Published as a conference paper at ICLR 2024
We construct ADOPD by addressing two important questions: (1) How do we gather document data, and what rypes of documents should be included in the dataset? Table [ compares ADOPD with earlier datasets for document layout analysis (Mondal et al., 2020 Smock et al.) 2022 Landeghem et al. 2023 Saad et al., 2016 Most datasets are sourced from PDFs, with limited document types. Models trained on such homogeneous data are unlikely perform well on different types of documents, so a top prority when collecting ADOPD is to maximize the diversity of documents types in it. (2) What elements should be annotated in document images for page decomposition? Documents, with their varied forms, can be interpreted differently based on an individual's 'ss background. Document understanding encompasses intricacies such as visuals, text, and layout. For instance, poster with a form may visually seem like form, yet its text could classify it as a science or education book. The complex nature of document data poses challenges in hierarchically structuring it, a critical aspect for successful vision datasets like ImageNet and MSCOCO. Meanwhile, accurately describing the content of documents is highly valuable, but it is also more challenging than natural image captioning.
We explore the fundamental question: How can we obtain a reasonable taxonomy of doc- ument types? Pre-defining fixed taxonomy solely based on human knowledge is not prac- tical. Instead we assume an open taxonomy and make use of a data-driven taxonomy dis- covery method, gradually assembling the tax- onomy through large-scale data exploration. Relying solely on manual annotation of doe- ument types. which requires reading and un- derstanding the document content, is also not practical. Therefore, we leverage the powerful zero-shot capabilities of large pretrained mod- els such CLIP (Radford et al. 202I) and
Ciibal Tegs Siete Kalense Prev Crve, And Clast Ney Kowwiiki
Page
Doc2Tag
Trackee Faunrrcbus
BIG HISTORY:
BIG HISTORY:
DaclMask
Doe2Box
Dec25et
Caption 1h so bock oeve The thle Thodlin Prinec Hhy Nastryy Sapain Now Aur Park Hi and crangr unt to spair comst Ther''s whlla rabura kodl lour ribtcber mad: S.bad by laaidd Grineis Dovi Sttce Extta Quacdecter, and Audiy Acroager" The cover dadggoondd a Nas peitiig cetaa Goling with soar and and you beccee awe miing porsos be oeter the coeer thy teasyraph of derece poos
Figure 2: ADOPD page decomposition illustration.
Large Language Models (LLMs) (Floridi & Chiriatti 2020) to assist in data selection and analysis. We couple the language model with methods for oft-sf-iistiibuttio (OOD) detection (Gu et al. 2023) for outlier data selection, complemented by a human-in-the-loon (HITL) approach to achieve data diversity. Each ingredient in our proposed approach- LLM, OOD and HITL-is is imperfect, but together they support the selection and annotation of diverse data at scale, within a reasonable budget.
Fig. M illustrates the diverse document types in ADOPD, comprising visually and textually rich documents, posing an annotation challenge despite its advantage. For visually rich docu- ments like posters and diagrams, entity masks capture relationships between visual elements effectively. Conversely, for text-rich documents such as letters and articles, text bounding boxes more suitable for marking key textual ele- ments. To accomodate both types, we segment each document into entity masks and text regions, and provide two types of descriptive labels for each document image. Fig. 2 showcases the four document page tasks: entity segmentation (Doc2Mask), text detection (Doc2Box), tagging (Doc2Tag), and captioning (Doc2Seq).
Table 1: Comparison of document datasets.
Dataset
Year
Size
Anno (Type)
Category
PubLayNet
2019
360K
Bbux
A (1)
Bank
2020
500K
Bbox
(1)
IR---A--13K
2020
13k
Bbox (4
DucLayNet
2022
80.9
Bbux (R
(6)
M'D
2023
9.1k
Bbox (A2A)
L (?)
Polygon (#85)
Text Bbox (222)
ADOPD (Ours)
2024
120k
Caption (1)
(>1000)
Tag
In sum, ADOPD is large-scale diverse document page decomposition and understanding dataset, designed to support future research in document domain. In this paper, we:
present ADOPD, comprehensive dataset for document page decomposition, encompassing four distinct tasks: Doc2Mask, Doc2Box, Doc2Seq, and Doc2Tagg
propose a data-driven approach for constructing document taxonomies during data collection and safeguard the ADOPD through outlier detection and human-in-the-loop.
conduct extensive experiments and analysis on ADOPD, demonstrating its effectiveness and generalization capabilities for document understanding research.
'The symbol D indicates automatic annotations, + represents human annotations, and d signifies LLM
assistance. & indicates that the document source is digital PDF, while @ indicates document images.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
Document Datasets. As shown in Table 1 several recent document image datasets have been introduced. PubLayNet (Zhong et al., 2019) comprises images and annotations generated through the automated alignment of PDF' with XML formats. DocBank (Li et al.) (020b) is created using LaTeX-generated PDF files and employs an efficient weakly supervised approach for annotation. DocLayNet (Piizmann et al. (222)) relies on human annotation rather than automated methods. This dataset encompasses six distinct document types and encompasses a total of 11 annotation categories. DoD (Cheng et al. 2023) is recently introduced dataset featuring approximately 9k modern document images, divided into seven subsets. It contains detailed annotations spanning multiple distinet categories. IITT-AR-13K (Mondal et al. 2020 is tailored for object detection in business documents like annual reports, containing annotated pages with standard layout elements like text, headings, lists, graphics, and tables. In summary, existing large-scale document image datasets mainly focus on PDFs, unlike the varied scanned or photographed images encountered in real-worll scenarios. This limited distribution of data can bias trained models. Additionally, publicly available datasets often cover only a narrow range of document layouts and categories.
Document Models. The document domain has witnessed the emergence of foundational models (Li et al. 2020a Prasad et al., 2020), driven by advancements in deep learning. Despite rapid progress in document understanding models, the scarcity of powerful models trained on high- quality, large-scale document data remains significant challenge. Earlier document layout analysis methods (Ouwayed & Belaïd 2012; Lee et al., (019)) relied heavily on rule-based and heuristic algorithms. However, their applicability was limited to simple document types, resulting in poor generalization performance. In addition to of document pretraining models (Huang et al. 2022 Li et al. 2021: Gu et al. 2021; Tang al. 2023 Kim et al. 2022). These models are typically pretrained on the ITT-CDIP (Lewis et al. 2006) dataset and evaluated various document benchmarks. Despite the remarkable performance of these models on benchmark datasets, it is critical to acknowledge that most current image-based document datasets are predominantly composed of a narrow range of document types, failing capture the heterogeneity of real-world documents. Moreover, the restricted data diversity in these benchmark datasets constrains the development and evaluation of document models.
models, researchers have proposed a range
3
ADOPD DATASET
ADoPD
Web Iragges
Pre-Labekel
Data
Fabeled
Duta
Model- Zoenany
Model Ansiaed
Figure Model-assisted data collection and annotation pipeline for ADOPD.
ADOPD stands out among document datasets it is constructed using diverse document images found on the web. Sec. 3.1 introduces document page decomposition tasks. Sec. 3.2 presents a data-driven approach to discovering document taxonomy for data collection and analysis. Sec. 3.3 employs models to assist with human annotation, addressing challenges posed by diverse data.
3.1
TASK DEFINITION
Fig. 2 illustrates the document page decomposition task defined in this paper, which encompasses four subtasks: Doc2Mask, Doc2Box, Doc2Seq, and Doc2Tag.
The Doc2Mask task entails segmenting visual entities in document images in a class-agnostic manner. The "entity" in this context denotes thing (instance) mask stuff mask. For example, in Fig. D an entity represents a meaningful and coherent region (e.g., banner, figure, logo, etc). The Doc2Box task calls for identifying text region-of Interests (RoI) within a document image, regardless of their specific types. The term "box" refers text Rol (e.g., paragraphs and titles, etc).
3

---

Published as a conference paper at ICLR 2024
The Doc2Seq task involves generating captions for document images, requiring the model to analyze visual elements and structured text. Given the complexity of document images, the model must effectively comprehend visual, textual, and layout information produce detailed captions.
The Doc2Tag task is akin to image tagging. specifically multi-label image recognition, where the objective is to assign multiple semantic labels to an image. In Doc2Tag, two levels of tagging are utilized: one based on the overall image content and another on specific local regions.
3.2 DATA-DRIVER DOCUMENT TAXONOMY DISCOVERY
In standard classification scenario, we deal with a given dataset denoted as Diull where X' represents the input space, and y = {1,.... K} is the label space. The classification model, denoted as f= g g h, consists of a feature extractor h : X' + 1dd and a classifier 9 R - K which maps the input's feature embedding K real-valued numbers called logits. In practice, establishing a guiding taxonomy associated with K is crucial for effective data collection, enabling us to manage and assess the diversity of the collected data. However, determining an appropriate value for K in documents is challenging due to the diversity of documents. We draw inspiration from pretrained models such as CLIP, GPT-4, etc, which have been trained on large-scale datasets and can serve as knowledgeable "experts" for data selection. Despite the benefits of pretrained models, the predictions from such models are not always reliable. E.g., LLM tend to suffer from hallucination problems (Bang et al. 2023). Hence, incorporating safeguards into data collection is essential. Fig. provides overview of our data collection process, which will be detailed in the subsequent sections.
Can Large-Scale pretrained Models Facilitate Data Collection? Given a document image z Diull. we can extract document information using pre- pre-existing models as follows:
{z, SOCR:
Scaption: Satribute: Stahel) = (l(x), [oCR(x), [INT(r); [Tgg(T), [cup(x())) (1)
{SCaption Sing LLM( SoCR, Scaption SAtribute: Stabel Prompt)
where 2 E RRP is obtained through an image feature extractor l1(.). The sequence SOCR consists of words and their coordinates, extracted by OCR tool fOCR (oCc The caption Scaption is generated by the captioning model (1zr()). Tags 5 Attribbte are produced by the image tagging model (+). Labels Label are generated by the CLIP model fcLI constrained by y. Integrating multimodal infor- mation, as expressed in Eq for document reasoning poses a significant challenge. As demonstrated
in Eq 2
we harness the power of LLMs and formulate prompts to predict tags (Stiug) and captions (S Caption for document images. The ablation study of these prompts is explored in the Appendix.
How to Safeguard Data Collection? Despite the impressive zero-shot capabilities of LLMS for sequence reasoning, prediction errors and uncertainties may still arise. Some failure cases can be addressed with stricter prompts. Even so, fully relying on LLMs for data selection poses heavy risks.
Fig. A illustrates our data selection diagram, strengthened by outlier detection. For each batch of sampled web images sDeetecte),, we de- fine it as mix of in-distribution (ID) (D) psendo-in) and OOD (Dpeeudo out) data. In D Prseudo-in all sam- ples belong to taxonomy classes we have already explored, while 2 Ppecudooout comprises samples from document types we haven't I explored yet. Alg. D Nutlinee the process where we integrate outlier detection for data collection and taxon- omy discovery. Given the dataset pool denoted as Dj and indicates the time step, we initially select a batch of data, denote as Dí Deelected ip- from Drall Based on the current taxonomy D", we first partition yr into 100 clusters using the K K-means algorithm (Jiang et al. 2024). Afterwards, we sample D Piseudo-in from D' D'stected-inn based on the K clusters, corresponding to step ! in Fig. 4 Specifically, we randomly select sub-category v from cluster lo as the representative category. We then use CLIP to sample n' documents classified under D Discuutoin comprises total of 100 n' document images. This selection process ensures a balanced sampling operation within the current taxonomy y
D
fill
D selested-ia in
y
D' Ppesudo-in
Dj sellected
D' Precuso-cut
+ AJo
Duiteled
D
pesmdo-ia
D sclocsod
3
Figure 4: Data selection diagram for ADOPD.
vi- The pseudo input ID) data
WWttoout specific indication, L.LM in this paper refers GPT-4
4

---

Published as a conference paper at ICLR 2024
Sampling from ID data can lead to biased distributions because models trained on such data may silently fail when faced with OOD inputs. Therefore, enhancing the diversity of ADOPD by incorporating hard negative examples may result in an overall improvement in diversity. Therefore, we explicitly sample a OOD subset D pacudo-out from the current candidate pool Di saalocted-in \ DD psendo-inn corresponding to step 3 in Fig. To obtain D pseudo-outt we employ K --means to segregate outliers
from D selected-in Specifically, we extract image features Z' for D Ditto U DDD Selecrd-in where D DDi "peeudo-in psendo-ou out this context, Z' = #1 with k E [o, 100) representing the set of K-means centroids estimated from selected U.D
paeudo-inn
The outlier score is estimated as the Euclidean distance
between it and the nearest centroid:
s' (z)
min ke 1000
|| - 12,
E (D(
selected-in
D pesudo-in
(3)
where Di D
pseudo-out contains data points with outlier scores ranked in the top n² across K clusters.
Given the selected ID and OOD data, we have Dr Dielected Dr Prseddo in UDr D poeudo which ready for annotation (step ( in Fig. 4.. Before an- notation, we update yx-1 by using the newly selected data D Deelecsed- Here, we employ the ap- proach outlined in Eq. - leveraging the LLM to predict the presence of new labels and ob- tain the updated taxonomy y". We use prompt- based methods to predict document tags by considering four aspects: visual (P visual), ) tex- tual (9 (exuua)) layou! ( byyou)) and multimodal (IP multimodal): Each aspect addressed through unique input combinations in the prompt. Ad- ditional details about the prompts can be found in the Appendix. After obtaining outputs, we implement two safeguards to filter out failures. Firstly, we design a prompt-basee based summarizer
Algorithm 1: Data-Driven Taxonomy Discovery
Input 3 Din.. e.
Output :Expanded Tixonomy y while True do
Collect Pidtssin from Dia ;
Select Piseabii from Pidecedaa based 3 1;
① Generate image embeddiing for aokened femaoiii
① Calculate s ((x)) E (Dishsteiin
Dt pesaloia)
Select outlier data D Pisust--ari
foreach x: D/ penat-iin U posabb
@ Predict new lahels using four prompters;
@ Lppdate y" with the newly predicted labels;
Refine y with human annotator:
if [[y'] > then
Stop:
else
- 1+1:
(9 summary) using LLM obtain 10 tags by summarizing the tags predicted through the four prompt strategies. Secondly, after the label generation by LLM, human annotators review and climinate labels that are confusing or irrelevant the document.
3.3 MODEL--ASIITTR DATA ANNOTATION
Data Collection. The images in ADOPD are sourced from the Laion-HR Laionn High Resolution), which comprises high-resolution web images, including multilingual document images. Laion- HR provides foundation for multi-lingual multi-modal ADOPD. We leverage pretrained models with humans in the loop to collect and filter data. The process includes the following steps:
Model-Assisted Data Selection: We first select images based on Laion-HR's metadata by ap- plying criteria such as pwatermark < 0.8 and punsafe < 0.5. Then, we construct a document discovery dataset using natural image datasets (e.g., ImageNet, etc) and document datasets (e.g., DocLayNet, etc). We then finetune a DiT-based binary image classifier (Li et al.] 2022a) to identify potential documents (probability > 0.8). Subsequently, we apply an OCR tool (Du al.. (021). and retain those with a word count exceeding 10. Although metadata provides predictions for watermarks, in order to improve accuracy, we additionally train a watermark detection model to filter watermarked images. We compute MD5 hashes and Hamming distances between images to exclude duplicates, even if document images in Laion-HR have different URLs. Fig. 10 in the Appendix shows the percentage of data selection.
Human Selection and Sensitive Verification: Based on our taxonomy pretrained CLIP model for zero-shot tagging. Human annotators then select safe and valid images for all categories. We do not rigidly specify that images must be print-format documents, but instead suggested the annotators to choose those that resemble documents. Annotators are tasked with filtering the dataset for potentially sensitive information.
3 obtained by Alg. I we adopt
Note that the taxonomy y gradually change with the growth of data collection.
5

---

Published as a conference paper at ICLR 2024
Data Annotation. The annotation process of ADOPD prioritizes the core principle of understanding the document's 's structure and layout. We avoid imposing overly rigid constraints on annotation
Model-Assisted Manual Annotation: In the early stage, utilize pretrained CropFormer (Qi et al. [023) to generate pseudo entity masks, Annotators follow guidelines to adjust the masks by adding, modifying, deleting needed. After annotating sufficient amount of data, CropFormer is retrained with the new annotations and serves as the seed model for data preprocessing in the subsequent stage. Through this iterative process, our model progressively reduces annotation costs while simultaneously increasing annotation efficiency. Fig. 12 in the Appendix illustrates the effectiveness of model-assisted annotation. During the annotation process, we provide document captions (Spptioon) and tags (Sigg) to aid annotators in understanding the document.
Multi-Task and Multi-Lingual Annotation: ADOPD stands out from other document datasets for its multi-task and multi-lingual characteristics. Our primary focus is on English and CJK (Chinese, Japanese, Korean) documents, with 60k document images in English and the remaining in the other languages. We reserve a private test set for the competition. Each dataset has four tasks introduced in Sec. 3.1] Specifically, for Doc2Mask annotation, we refrain from imposing semantic constraints on labeling entities, therefore encouraging annotators to come up with open-ended names or descriptions that are accurate (e.g., "doc-in-doc", "banner", "infographic", "natural image", etc). As our task focuses on document entity segmentation, we do not incorporate label information in segmentation evaluations. For Doc2Box, we have stricter rules which require annotators to comprehend words and group them according to their semantic meaning. The annotation files follow the MSCOCO annotation format.
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Baseline Models. We experiment the subset of ADOPD, with training and validation sets comprising 50k and 10k images, respectively. (1) Doc2Mask: we evaluate two frameworks: Mask2Former (Cheng et al. 202T) and CropFormer (Qi et al. 2023). to identify which is best stuited for the document page decomposition task. We perform ablation studies on these frame- works using different backbones, such Swin Transformer (Swin) (Liu et al. 2021), Hornet (Rao) iet al. 2022).. and ViT (Parmar et al., 2018). (2) Doc2Box: we similarly benchmark three models: Faster R-CNN (Ren et al., 2013). Deformable-DETR (Zhu et al.) 202T), and Cascade Mask-RCNN (MR-CNM (Cai & Vasconcelos, 2019). We also enhance Cascade Mask-RCNN by incorporating pretrained ViT backbones, specifically DINOv1 (Caron et al., (22TT) and Dinov2 (Oquab al. 2023) with ViT-Adapter (Chen et al. 2022). (3) Doc2Seq: we build an encoder-decoder model using pretrained ViT and GPT-2 (Radford et al. 2019) fine-tuned on 80k image-caption pairs for training and 20k for validation. The captions are generated using prompts specified in Eq. 2 Acknowledging the gap between LLM-generated and human annotations, we collect an extra 5k dumann-nnottan validation set for further comparison. (4) 2oc2Tag: validate our taxonomy discovery using the CLIP 111-G/14modee and report the OOD performance on RVL-CDIP (Harley et al. 2015).
We build Doc2Mask using the Detectron2 (Wu et al., 2019) and Doc2Box with MMDetection (Chen) ct al. 2019). All experiments are run on NVIDIA A100-80G8 GPUs Following standard prac- Ghiasi et al. 2)21), we employ input resolution of 1024x 1024, achieved by re-scaling and padding the shorter side of the image. Doc2Mask (CropFormer and Mask2Former) and Doc2Box (Faster R-CNN, Cascade (ask-RCNN) are trained for 15 epochs with a batch size of 32 on 8 GPUs to achieve full convergence. We train Deformable-DEr -EERR for 30 epochs due to slow convergence issues. We build other models (Doc2Seq and Doc2Tag) with Huggingface Transformers framework (Wolf et al., 2020). For Doc2Seq, we train it for 50 epochs on 8 GPUs with total batch size of 800. Finetuning CLIP ViT- on Doc2Seq data takes 100 epochs on 8x8 GPUs.
Evaluation Metrics We evaluate Doc2Mask and Doc2Box with the mean average recall (mAR) and mean average precision (mAP) metries. This assessment considers ten overlap thresholds ranging from 0.5 to 0.95 in increments of 0.05 AmAP@0.550.955 For OOD evaluation, we use metrics including the Area Under the Receiver Operating Characteristic (AUROC), False Positive Rate at 95% Recall (FPR95), maximum concept matching (MCM) score (Ming al. 2022), and accuracy
"This research's data collection and annotation were completed in October 2023.

---

Published as a conference paper at ICLR 2024
(ACC). For Doc2Seq, the BLEU @n (B @n) Papineni et al. et al. 2015). METEOR (M) (Denkowski & Lavie 2014) and ROUGE (R) (Lin 2004) for evaluation.
2002), CIDEr (C) VVddanta
4.2 DOCUMENT PAGE DECOMPOSITION TASKS ANALYSIS
Comparing the Model Architectures. Table 2 compares Mask2Former and CropFormer models on Doc2Mask. CropFormer outperforms Mask2Former with similar backbones and pretrained datasets. CropFormer's superiority stems from its integration of image crops alongside full image imput, enhancing mask prediction with detailed information. This highlights the model's s ability to handle multi-view and local image information, especially in the context of document images.
We compare various object detection mod-
Table 2: Results on Doc2Mask with varied back-
els in Table 3 including Faster R-CNN, Deformable- Darinabble--ETT. DETR, and Cascade MR-CNN. While Befformaale-DETTe improves, it doesn't t outperform anchor-based detectors like Faster R- CNN and Cascade MR-CNN significantly. De- spite achieving a higher mAR, the limited mAP improvement may be due to the distinct data dis- tribution of text boxes, differing from general ob- jects in natural images with clear classification boundaries. Meanwhile, the Cascade MR-CNN, combining Mask R-CNN and Cascade R-CNN, achieves the highest mAP. It enhances instance segmentation performance and aids text detec- tion, especially for words requiring pixel-level feature representation.
bones (T, B, L) denoting tiny, base, and large types.
Mask
Quality
Backhone
Pretrain
mAP
APso
APs.
mAR
EntitySeg
31.80
37.16
32.33
34.0
Swiny
ImageNet
28.95
34.36
29.51
31.0
MaskFFormer
Swing
EaiiyySgg
32.81
38.14
33.17
33.3
ImageNet
30.21
36.30
31.18
32.5
Hormet..
Eatiyy
34.39
40.09
34.95
36.9
ImageNei
32.96
38.22
33.30
35.2
ViTs
SA-IB
35.59
41.05
36.35
VITL
35.81
40.27
36.53
37.8
EatitySg
35.46
41.56
35.60
38.5
Swing
ImageNet
37.71
45.30
38.20
41.0
Croplormer
EmiiySeg
36.03
42.30
36.21
39.3
Swin:
TmageNet
37.73
44.62
38.49
40.7
Hornet,
EatilySog
35.03
40.00
35.73
37.6
ImageNet
36.06
41.84
36.69
38.7
ViTa
35.87
41.92
36.73
38.4
SA-IB
ViTg
39.56
45.72
40.33
42.4
Comparing Backbones and Pretraining. Table 2 also investigates the impact of vision backbone pretrained various datasets. References EntitySeg, ImageNet, and SA-1B indicate pretraining on the respective datasets. SAM (Kirillov et al. (023) pretrained on SA-1B outperforms the Swin/Hornet models trained on ImageNet or EntitySeg. This can be attributed to two factors: firstly. SA-1B is sufficiently large (around 1 Billion). Secondly, while Swin/Hornet architectures are well-suited for segmentation, SAM is trained with pixel-level supervised learning. enabling it to acquire improved pixel-level representations crucial for document image understanding.
Table compares different backbones on Doc2Box. Dinov2p14 V excels with higher mAP yet slightly lower mAR, demon- strating the superiority of self- supervupee back- bones over pretrained alternatives. This is cru- cial for document analysis, given the absence of high-quality ImageNet-like pretraining data. Comparing Dinov lpg and D I suggests that fine-grained patches enhance document im- age features. Fig. 5b illustrates the results of Doc2Box using Dinov2p14+VIT Adapter-
Table 3: Performance comparisons on Doc2Box.
Box Quality
Method
Backhone
mAP APs, APr. mAR
Faster R-CNN
ResNetto
61.1
78.9
67.0
74.9
ResNet.o1
78.6
67.3
74.3
Deformable
ResNetgo
65.0 82.2
72.1
81.6
ResNef
65.5 82.8
72.7
81.6
ResNetso
64.7 80.9
71.0
Re
65.3 71.7
68.7
79.1
Cascude MR-CNN
Dinoclast VTT ViTadque
63.6 80.4
69.6
76.3
DinoxIpit VIT Adgas
63.2 80.3
69.5
76.2
Dinov2pus* VIT NiTada
67.0
82.7
73.2
77.8
Evaluating Generalization Ability. In Table (a), we compare the model trained on ADOPD with those fine-tuned on Eng, Combined with Fig. 5a it is evident that models fine-tuned on ADOPD can better focus on fine-grained document elements and make more reasonable predictions for document entity masks. Conversely, models pretrained on EntitySeg can predict some masks but tend to excessively detect elements present natural images (e.g., people, objects), while neglecting the document's 's inherent layout. Table 4 (b) validates the cross-dataset generalization pretraining advantage of ADOPD, specifically focusing on the evaluation set of DocLayNet. For a fair comparison, we consider only text detection without categorizing the boxes. Directly applying the model fine-tuned on ADOPD to DocLayNet data yields zero-shot results with high recalls. Furthermore, fine-tuning on DocLayNet with ADOPD pretrained backbones outperforms fine-tuning with ImageNet backbones. Note that DoeLayNet's 's testing is limited to its limited document types and cannot reveal anything about the generalization capability of ADOPD for other taxonomy types.
7

---

Published as a conference paper at ICLR 2024
Table 4: Ablation studies: (a) comparing the performance of models trained with ADOPD and models fine-tuned only on EntitySeg. The results in "(-)" represent the zero-shot outcomes for EntitySeg. (b) Cross-datased evaluation to assess the generalizability of ADOPD on DocLayNet. (b) Cross-dataset evaluation on DocLayNet.
(a) Results for with and without ADOPD,
Backbone
AP30
mAR
Zero-Shot
Finetune
Mask Former
Method
Backbone
ADOPD
ImageNet
ADOPD
Swing
58 228
(2(.225 33. 1711 88) 35. (5.33
MAP
mAR
mAP
mAR
mAP mAR
Hornet
34.3((15.83)
(0.04) (4.95(15.38) (6.9(29.0)
Faster R-CNN
ResNetso
0.9
58.5
43.0
55.8
44.5
60.
Croplormer
ReaNet.oi
1.0
56.6
46.0
58.5
47.0
60.7
Swing
3.03x13.46) 42. 19.441 56, 26.23(13112) (9.3524.5)
DeformableDETTR
ResNetco
2.2
80.4
75.4
88.9
Hornen,
35.03(15.688
(5.08(15.66/000002...33 40.00(21.399 (5.33135557333388 33.73 (37.6/26.9)
ResNetio
266
79.0
75.4
85.9
77.2
88.1
ROGERS
60%
M
(a) Mask Prediction Comparison: From top to bottom, we showcase the original image and predictions from the best models trained on ADOPD, V2, and SAM (SAIB), respectively.
FRODON COSTE OF BLYING AOOME
Natural Selectionts...
GOLOEn RETRIEVER COLORING BOOK! AAS
ERCHT DELIIO
(b) Document text detection visualization results, each image paired with its caption and tags.
Figure 5: Visualization of ADOPD images and results for Doc2Mask and Doc2Box.
Prodpt-Guided Context-Aware Captioning Benefits Vision-Language Modeling. Table 4.2 evaluates caption quality. We collect 5K test data evaluate the effectiveness of Doc2Seg. The @ represents the GPT-4 model, and t (Li et al. 2022b) and BLIP2-OPT-2.7b (Li et al. 2023) 1---38///TBes-i..-eee
are obtained from Huggingface model hub. ViTBase-
are line-tuned
on Doc2Seq. While GPT-4 captions achieve commendable CIDEr score, indicating consensus, a noticeable disparity persists between them and human annotations. Models fine-tuned on Doc2Seq can attain similar performance to GPT-4 on B @@n n. but show significantly lower CIDEr scores. In captions are notably longer than machine- machine-generated ones, impacting pearence-basedd evaluation. These findings highlight the challenge in document captioning due to diverse interpretations and varying caption lengths, complicating evaluation.
Fig. 6 (left),
To verify the benefits of prompt-guided captions, we fine-tune CLIP with Doc2Seq data and conduct two experiments: zero-shot evaluation on RVL-CDIP test set and supervised training on RVL-CDIP based on finetuned CLIP vision backbone. While finetuned CLIP improves zero-shot capability for specific data (e.g., Budget and Presentation), the overall enhancement is comparable to raw pretrained CLIP. We observe from Fig. [ (right) and Table 6 that finetuning the CLIP ViT backbone, initially trained on Doc2Seq. with a classifier layer for separate training on RVL-CDIP results in a noticeable improvement. This underscores the importance of caption rewriting for handling noisy data.
8

---

Published as a conference paper at ICLR 2024
Decanura Captia Word Coea Dicthedee
XVL CDIP Chacdiatio Perterance
Method
Test
B@1 1
B@
B83
B04
M
R
C
-
27.0
16.7
11.7
8.5
12.8
22.4
84.7
BLIPLage
12.4
8,5
6.4
5.0
9.3
22.6
18.3
A
344
BLIP2-OPT-2.76
4.3
3.0
2.6
25.1
16.5
12.3
7.6
5.3
3,9
9.0
20021.8
18.0
22.5
4.4
2,5
7.7
16.8
9.8
VIT PCP
2.3
5.8
13.9
4.4
ViT Buse Pis GPT2
69
23.4
4.7
2.7
8.0
17.2
11.0
Naather Weeds 20
17.3
6.0
2.4
1.1
6.0
14.1
5.3
Figure 6: Ablation study on captions.
Table 5: Ablation experiments on Doc2Seq.
Table 6: Performance of Models for Per-Class Classification
Model
Type
Letter Form Email|
SR
SP
SP
FF
NA
Bgt
Inv
Prsnt]
Que
Rsm
[Mmoo]
Avg
(Recal
DiT
Supervised 98 92.922100 98.76 76
99.45
98
59.
99.84
09.31
99.52
99.18
99.24
98.83
99.76
09.15
99.28
99.16
99.18
ViTG.14
Zero-Shot
98.52
98.76
79.78
91.89
53.46
76.51
87.64
84.82
32.81
99.49
33.07
99.35
54.81
93.75
90.46
98.23
79.58
VDDo
Zero-Shot
94.32
87.55
75.33
85.79
27.46
80.33 96.21 63.46
38.72
65.81
93.93
72.26
84.25
84.41
95.02
77.73
Por-Class
(Accuracy)
DiTuue
Supervised 92.41 86.83 98.97
96.13 94.63
187 95.22
94
96.68
92.77
92.73
92.73
94.00
87.38
190
97.67
99.14
993.36
ViTG
Supervised
78.26
85.87
VIDO. Supervised 90.87 84.48 96.98 95.34 93.76
82
93.51
93.00
95.61
89.81
89.62
92.85 84,29 90.23
96.45
92.62
91.38
The abbreviations are: Handwwitten (Ilw), Advertisement (Ad), Scientific Report (SR). Scientific Publication (SP), Specification (Spec), File
Folder (FF),, News Article (NA), Budget (Rgt), Invoice (Inv), Presentation (Prsnt), Questionnaire (Qnr), and Resume (Rsm).
Data-Driven Document Taxonomy Analysis. To verify Alg. [ we collect the ID dataset from both RVL-CDIP and Laion-HR based on the classes provided RVL-CDIP. We sample OOD categories such as "Magazine (M)", , "Comic (C)", "Guidebook (G)", "Yearbook (Y)", "Worksheet (W)", and "Open Book (OB)", etc, from y and collect the OOD data from Laion-HR. In the Appendix, Table 7 shows OOD detection results for two variants: predicting 16 and 50 centroids separately. The K-means method with 50 centroids excels in detecting outliers across all categories. Fig. (center) displays taxonomy expansion with HITL taxonomy cleaning. We start with an initial ID set with 10 classes selected from RVL-CDIP. At every step, we sample 10 detected outlier data. As the data increases, our outlier detection method successfully retrieves outliers for the majority of novel categories. Fig. (left, right) illustrates the distribution of "Comic" and ID data, where "Comic" is detected as an outlier in the first step. Red color indicates the detected outlier samples.
Illss Ieihhed HEBLD
Ialeur Lo
12l wit 12hm) -t (2000
Hatte otlingar
Figure 7: Visualization of outlier detection, taxonomy expansion, and OOD score distributions. Responsible AI Analysis. During data clean- ing, we conduct& a comprehensive Responsible Al analysis, tackling biases in sensitive areas such as nudity, sexuality, and violence, etc. We meticulously filter sensitive data with input from 15 diverse evaluators. Fig. disslays their geo- graphic distribution. If any evaluator deems image inappropriate, we label it as sensitive. After review, we remove 9.29% of potentially sensitive images, ensuring the majority of the 120K images remain non-sensiiive. This rigorous process guarantees a safer and less biased dataset, promoting fairness and inclusivity in our models.
Figure 8: The evaluators' geographic distribution.
CONCLUSION
This paper introduces ADOPD, a large-scale dataset for document page decomposition, and outlines systematic process including data collection, taxonomy analysis, model-assisted data annotation, and HITL processes. We conduct comprehensive analyses and detailed experimental comparisons across four tasks, demonstrating the value of ADOPD. It opens up numerous opportunities for future exploration and the development of foundational models for document understanding, aiming to catalyze advancements in document analysis.

---

Published as a conference paper at ICLR 2024
REFERENCES
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. In IJCNLP-AACL, 2023.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: High quality object detection and instance
segmentation. TPAMI, 2019.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and
Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li. Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint
2019.
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In ICLR, 2022.
Bowen Cheng. Alexander Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In NeurlPS, 2021.
Hiuyi Cheng, Peirong Zhang, Sihang Wu, Jiaxin Zhang, Qiyuan Zhu, Zecheng Xie, Jing Li, Kai Ding, and Lianwen Jin. M6doc: large-scale multi-format, multi-type, multi-layout, multi-language, multi-annotation category dataset for modern document layout analysis. In CVPR, 2023.
CLIP ViT-G/14. g- 14- laion2B s12B b42K 2023.
Clip vit-g/14.
-eee--ii-------
Michael Denkowski and Alon Lavie. Meteor universal: Language specific translation evaluation for any target language. In WMT, 2014.
Yuning Du, Chenxia Li, Ruoyu Guo, Cheng Cui, Weiwei Liu, Jun Zhou, Bin Lu, Yehua Yang, Qiwen Liu, Xiaoguang Hu, et Pp-oerv2: Bag of tricks for ultra lightweight oer system. arXiv preprint arXiv:2109.03144, 2021.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010.
Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds
and Machines, 30:681-694, 2020.
Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung- Yi Lin, Ekin D. Cubuk, Quoc V. Le, and Barret Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation. In CVPR, 2021.
Jiuxiang Gu, Zhenhua Wang, Jason Kuen, Lianyang Ma, Amir Shahroudy, Bing Shuai, Ting Liu, Xingxing Wang. Gang Wang. Jianfei Cai, et al. Recent advances in convolutional neural networks. Pattern Recognition, 2018
Jiuxiang Gu, Jason Kuen, Vlad Morariu, Handong Zhao, Rajiv Jain, Nikolaos Barmpalios, Ani Nenkova, and Tong Sun. Unified pretraining framework for document understanding. In NeurIPS, 2021.
Jiuxiang Gu, Yifei Ming, Yi Zhou, Jason Kuen, Vlad Morariu, Handong Zhao, Ruiyi Zhang, Nikolaos Barmpalios, Anqi Liu, Yixuan Li, et al. A critical analysis of dut-of-distribution detection for document understanding. In EMNLP. 2023.
Adam W Harley, Alex Ufkes, and Konstantinos G Derpanis. Eyaluation of deep convolutional nets for document image classification and retrieval. In ICDAR, 2015.
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutlmv3: Pre-training for document ai with unified text and image masking. In ACMMM, 2022.
10

---

Published as a conference paper at ICLR 2024
Wenyu Jiang, Hao Cheng, Mingeai Chen, Chongjun Wang, and Hongxin Wei. Dos: Diverse outlier sampling for out-of-distributin detection. ICLR, 2024.
Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeong Yeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In ECCV, 2022.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In CVPR, 2023.
Laion High Resolution. Laion. laion-bigh-resolutiii 2023.
Jordy Landeghem, Rubén Tito, Lukasz Borchmann, Michal Pietruszka, Pawel Józiak, Rafal Powal- ski, Dawid Jurkiewicz, Mickaël Coustaty, Bertrand Ackaert, Ernest Valveny, et al. Document understanding dataset and evaluation (dude). In ICCV, 2023.
Joonho Lee, Hideaki Hayashi, Wataru Ohyama, and Seiichi Uchida. Page segmentation using convolutional neural network with trainable co-occurrence features. In ICDAR, 2019.
David D. Lewis, Gady Agam, Shlomo Engelson Argamon, Ophir Frieder, David A. Grossman, and Jefferson Heard. Building a test collection for complex document information processing. SIGIR, 2006.
Junlong Li, Yiheng Xu, Tengchao Lv. Lei Cui. Cha Zhang, and Furu Wei. Dit: Self-supervised pre-training for document image transformer. In ACMMM, 2022a.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping Language-image
pre-training for unified languaee understanding and generation. In ICML, 2022b
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023.
Kai Li, Curtis Wigington, Chris Tensmeyer, Handong Zhao, Nikolaos Barmpalios, Vlad I Morariu, Varun Manjunatha, Tong Sun, and Yun Fu. Cross-domain document object detection: Benchmark suite and method. In CVPR, 2020a.
Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, and Ming Zhou. Docbank: benchmark dataset for document layout analysis. In COLING, 2020b.
Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu. Selfdoc: Self-supervised document representation learning. In CVPR, 2021.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In CVPR, 2021.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: dataset for vqa on document
images. In WACV, 2021.
Puneet Mathur, Rajiv Jain, Jiuxiang Gu, Franck Demnnncour, Dinesh Manocha, and Vlad Morariu. Docedit: Language-guided document editing. In AAAI, 2023.
Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of- distribution detection with vision-language representations. NeurlPS, 2022.
Ajoy Mondal, Peter Lipps. and CV Jawahar. liit-ar-13k 11kk a new dataset for graphical object detection in documents. In DAS, 2020.
11

---

Published as a conference paper at ICLR 2024
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. TMLR, 2023.
Nazih Ouwayed and Abdel Belaid. A general approach for multi-oriented text line extraction of handwritten documents. IJDAR, 2012.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In ICML. 2018.
Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S Nassar, and Peter Staar. Doclaynet: A large human annoattd dataset for document-layout segmentation. In SIGKDD, 2022.
Devashish Prasad, Ayan Gadpal, Kshitij Kapadni, Manish Visave, and Kavita Sultanpure. Cas- cadetabnet: An approach for end to end table detection and structure recognition from image-based documents. In CVPRW, 2020.
Lu Qi, Jason Kuen, Tiancheng Shen, Jiuxiang Gu, Weidong Guo, Jiaya Jia, Zhe Lin, and Ming- Hssan Yang. High-quality entity segmentation. In ICCV, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Yongming Rao, Wenliang Zhao, Yansong Tang. Jie Zhou, Ser-Lam Lim, and Jiwen Lu. Hornet: Efficient high-order spatial interactions with recursive gated convolutions. In NeurlPS, 2022.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS, 2015.
RSM Saad, RI Elanwar, NSA Kader, S Mashali, and M Betke. Bce-arabic-vl dataset: towards interpreting arabic document images for people with visual impairments categories and subject descriptors. PETRA, 2016.
Brandon Smock, Rohith Pesala, and Robin Abraham. Pubtables-Ime Towards comprehensive table extraction from unstructured documents. CVPR, 2022.
Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, and Mohit Bansal. Unifying vision, text, and layout for universal document processing. In CVPR, 2023.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In CVPR, 2015.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In EMNLP, 2020.
Yuxin Wu. Alexander Kirillov, Francisco Massa, Wan-Yen Lo. and Ross Girshick. Detectron2. 2019.
https://githuit hom.ttthhhcccoceeeeoooomeeeeeeee htwa:///cccccooooarrraacaeeeeeee
Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Publaynet: largest dataset ever for document
layout analysis. In ICDAR, 2019.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A. ABLATION STUDY OF OUTLIER DETECTION
Table 7 presents OOD detection results obtained using our K-means-based method, with two variants: predicting 16 and 50 centroids separately. The K-means method with 50 centroids excels in OOD detection across all categories, showcasing its effectiveness in identifying outlier data.
Table 7: Performance of outlier detection methods in dataset exploration.
Outlier Score
AUROC (%)T
FPR95
(9)4
Y
W
OB
M
C
G
OB
K. V-Meass (16)
95.99
99.99
59.99
99.99
99,85
99.98
0.01
0.11
0.00
0.00
0.67
0.08
RVL-CDIP
K-Means (50)
99.99
99.99
99.99
99.99
99.87
99.97
0.01
0.05
0.00
0.00
0.53
0.16
80.23
91.92
89.57
80.58
31.27
51.01
42.11
49.37
Means (16)
86.28
90.86
89.08
87.64
39.16
78.13
33.02
21.40
28.05
39.61
82.66
65.74
ADOPD
Means (50)
86.89
92.86
87.46
59.90
78.71
32.35
27.09
29.53
39.85
65.70
MCN
71.42
91.13
86.69
87.58
87.58
84.16
89.89
45.18
68.23
54.39
$6.66
60.76
A.1.1 ABLATION STUDY OF PROMPTS
Table 8: Human evaluation of prompt-guided context-aware captioning.
Prompt
Frampt
①. 4G Yiuure expes. is ponsninng dnsiigincagtin fn Josasero Puside comnieereoocee Jnciptonral feohouscce simage capt soer and neee imng dtilnses sstgges doceraes withh cunatal
27
2 oE Yourre on osperi gononing dostipine cagtinn in Jncanero. Punile consie reeasse of
33
<inage eaption: ionz eul more
isage.
imge siiielo stagee: The descusest las wosh cotrsce from inege:
koez wardas.. Ureas comprehamiiee and prodie captons deactiee the
keez wardav D Creas compretauuie and procue captacss to desetiee
@ &6 You in inpont geopaity cpviir drcener iragee
@ Vie eopert proeating decappas eqeion denmeess Procdd sorthesec
wordey The pootble itages: the doctrent:
iclaage cape wordes
The docmerr
uad conrent oforecing sposkal interution
captioe>
plane dosille docs-
Plass decche the docements crere rntrrnting the pooriddd intormateee.
D Cuesc compesanive and posbe capine to docile de Jocnses costos.
&
You eaper
decunn inclado some vinad ultiazgs
and cretasn
kooacis reforaae docsptin the docureet
capticos and mene
lor. roee pumsagys There as pooitti coane catiis for dae doctsncal.
saribace isegaa TTx decancei an seez collii soce aoraaays Pessc dozuil documeetsss costenl refoccaiigg lh pronicc islermatins. compueannce grocko captie
istiage captiass D Now, in yeur oes konds please descthee the doct- nene conrent interecnn the prosseed internatoon
Table 8 sshoss the results of human evaluation for various prompts. In each case, we randomly select 100 images for comparison and instruct human evaluators to exclude any incorrect samples.
As described in Sec. 3.2
we employ four prompts to generate annotated document labels, utilizing textual, visual, spatial, and multimodal information. The detailed prompts are listed in Table While the format of prompts focused on different aspects remains consistent, each prompt includes different input items, as indicated by check signals in Table 9 The bold text in the table represents inputs or variables related to prompt selection, as shown in Table 10
Considering the impact of input item order taxonomy generation quality, conduct experiments on various prompts to explore optimal input sequencing. Our investigation focuses on prompt sentences containing inputs (row 3-7 in Table 9. To simplify, we specifically examine rows 6 and 7, both representing high-level information for entire document images and intended to pair together, while the order between them remains undecided. In this study, human annotators evaluate prompt output labels and vote on them, assessing relevance between the document image and labels, eliminating orders resulting in conflicting or ambiguous labels. Subsequently, we tally votes for each order to determine the most favored arrangement.
1. Order of Visual and Textual Prompt Inputs:
We shuffle the order of input items for the visual prompt (3, 6, and 7) into four different arrangements while keeping 6 and 7 adjacent: 3-6-7, 3-7-6, 6-7-3, and 7-6-3. The investigation result is depicted in Fig. a). Similar to the visual prompt, for the textual prompt, we investigate four orders: 5-6-7, 5-7-6, 6-7-5, and 7-6-5. The investigation result is shown in Fig. (b). Based on the results of the visual and textual Prompts, we select 3-6-7 and 5-6-7 as the input order for our prompt. Additionally, we observe that it's preferable to input the description (row 6) before the pre-labels (row 7) and place it at the end of the list. We will maintain this order in the subsequent study.
2. Order of Layout Prompt Inputs: We investigate two different layout orders: 4-5-6-7 and 5-4-6-7, Considering that text space may influence layout comprehension, we classify a document with text area larger than 0.2 of the size as text-rich document. In our study, we present the results separately based on this criterion in Fig. 9 cc. The figure illustrates that when the text is rich, the order 5-4-6-7 receives more votes, while 4-5-6-7 is favored otherwise. Consequently, we choose to use the order 5-4-6-7 when the document is rich and 4-5-6-7 when the document is not as text-rich.
13

---

Published as a conference paper at ICLR 2024
JAARY
Figure 9: Human evaluation result for different order of Visual and Textual prompt inputs.
Table 9: Prompts across four aspects: textual, visual, layout, and multimodal.
Visual
Testual
Layout
Multimodal
You are an expert in sceing and understanding given document image.
Here the relevent information about the decument:
Visual elements may appear in document: visual [Pagn,
x
x
OCR layout information: [[OCR, Bboxs), [OCR1, Bbox,] 1, ...1 In
x
x
each layout item. the first numbers (xts Ma.
are normalized courdinatos, and
the last string the detected OCR text
The textual content the document
[OCRo, ,CR J.
x
7
7
6
A possible course description of this document: image caption
/
Pussible overall documeat calegories: [Pre-label,, Pre-labch,
7
You will primarily focus on USRD RESOURCE to provide 5 lahels related the should
/
conception "CONCEPT" documents your own word.
signilicantly representative of this document within 5 words.
an caample oulput: [Labelo, Label;, ...] labels should not contain any specilic content shown in the docammnt.
Order of Mutimodal Prompt Inputs: Multimodal prompts predict labels using all provided information. Building on previous findings, we standardize the order of rows 6 and 7, placing them at the bottom of the inputs. For the remaining inputs, we shuffle them to create a total of 6 different orders: 3-4-5-6-7, 4-3-5-6-7, 4-5-3-6-7, 3-5-4-6-7, 5-3-4-6-7, and 5-4-3-6-7. As shown in Fig. 9 the results suggest that the order 3-4-5-6-7 yields the best performance among all summarize them into 10 categories
orders. After obtaining labels with different information, to classify the document. The detailed prompt is shown in Table II
Table 10: Utilized information on various settings for prompts.
Modality
Information
Visual
The posential visual elements are considered alongside other mentioned details such as descriptions and possible categories.
Texuual
textual content summarized while taking into account other mentioee derails such as descriptions and possible categonies
Layout
The layout information and dasign of the document are considered alongside relevant details such as descriptions, visual elements,
textual content, and possible categorics.
Multimodal
All the aforementoned information, including descriptions, visual elements, textual content, possible categories, and layout
information, taken into consideration.
Table II: Prompts for summarizing document labels.
Prompt Sentence
1
You are a languistic expert that can well understand the difference between phrases.
same documentt image from annolators, They on visual, textual, layou, multimodal aspects of the image.
You will receive
of labels
3
You should come out 10 most common conceptions from the list with in words for each.
4
The labels given by the visual-focused annotator [Label]]] 1 Viscei
Label Visuel
labels given by linguistic-focttt focused annotator are: [Label] Pextuss
Label Destual
6
The labels given by the layout focused anmotator are:
[LabelFputin Label ...]
The labels given by multimdal- focused unnotator are: Ioltiiel JabelBultinnll
Lerr example for auiput: [Labelo, Label,,
A.2
DATA ANNOTATION PROCESS ANALYSIS
A.2.1 IMAGE ANALYSIS
Fig. illustrates our analysis of the document image data, showcasing the high resolution of our dataset. During the annotation process, notice that despite having high resolution, some document images' text appears blurry. We require annotators to skip labeling such images to ensure clear visibility of text in all document images.
14

---

Published as a conference paper at ICLR 2024
pobe
LAKD HKL(755)
Vald Image
Valid Doc
Doc (> Words)
Laken HUR(UTSM)
Doc Eo Like (3IM) - Doc Ln OCK Wonds 10 3M
Zaion HR(175M) Doe-ZheLikk (1.2M) Dee-Zh-OCR-Waah > (3963))
Lains (5M( Das-JaLike (24M) Dre.aa.CR-WWadss 10 (4243)
Figure 10: Diagram illustrating data selection proportions.
Figure 11: Distribution of the document image sizes for ADOPD.
15

---

Published as a conference paper at ICLR 2024
SCAME ULADE
Motion 2021 CALENDAR
Bodies
Bodies in Motion
2021 CAL ENDAR
2021 CALENDAR
EMPOWERING THE INDUSTRY
EMPOWERNG
INDUSTRY
Owaeee
Gnacl
Guade
Akk W.8
mistesss america
mishess americo
mishrss amertoo
HOT ROD WILD
HOT ROD
WIID
COOKIN
INFINITY
INFINIT
PINITY
OOOD0D
169
169
AMERCAS BEST MORTCAGE
AMERCAS BEST MORTOAGB MIERCAS BEST PORTGAGE
WUPH
OPH QUTZ
OPHEUN
OPN
Pre.abel ADoPD (Pa)
Lase-Lab Marla
Pre-Labd
Part-Labll Manks
ADDPD (Fo)
Figure 12: ADOPD (En) model-assisted annotation comparison. (Left) EntitySeg labels (prediction) and human annotations. (right) Doc2Mask (finetune on ADOPD) labels (prediction) and human annotations.
16

---

Published as a conference paper at ICLR 2024
FRUNT AODE BARO HAR
SUI'T MODE
SUNry MODE
ALOR
20
BIR 49F
18 4F
92 13URIK
92 SSURRA
Pref abe Madia
Pes Label Mada
Original
Labll ADuTD DXj
Paaslaall Madie
ADatD (7h)
Figure 13: ADOPD (Non-En) model-assisted annotation comparison. (Left) Doc2Mask (finetune on ADOPD-En) labels (prediction, Zh) and human annotations. (Right) Doc2Mask (finetune on ADOPD-En) labels (prediction, Ja) and human annotations.
17

---

Published as a conference paper at ICLR 2024
A.2.2 MODEL-ASSISTED ANNOTATION DATA VISUALIZATION
A.3
MULTI-LINGUAL DOc2BOX SAMPLES
Charllte CReAk A
FRANDA REO
Lo% PNdA
LONDON
Conduit
Lamiliio Goleebb
CANARA
raffera muiii Alkma
Mohiing WARdE
Zand Willfiimn
33
TAPL
518
T 7
26 20
40
Figure 14:
Doc2Box samples: English, Chinese, and Japanese document images, top to bottom.
Japanese's unique style poses challenges for Doc2Box compared to English and Chinese.
18


Reviews:
[{"review_id": "Co8ZnLXZde", "reviewer": "Reviewer_umJE", "summary": "To help the community develop techniques that process documents (like letters, forms, emails, news articles, resumes, and research papers), this work introduces a large dataset for several document understanding tasks. They annotate this data for document classification and for a few \"page-decomposition\" tasks, namely, Doc2Mask, Doc2Box, and Doc2Seq.\n\nThe underlying data is labeled and derived from Laion. However, the authors seem to have some concern about the diversity/balance of the types of data in there and/or the quality of the labels. Annotating this data is hard because of its scale. The authors introduce a rich pipeline to discover document taxonomies. The taxonomy developed in this way helps ensure that the final dataset is diverse and balanced. This pipeline is based on a human in the loop with information derived from several foundation models, namely, an LLM that orchestrates information from an image feature extractor, an OCR model, an image captioning model, an image tagging model, and a vision-language model. \n\nThe authors analyze the document taxonomy and find that it contributes \"to a more comprehensive and balanced dataset\". Then, they evaluate several models on their page-decomposition tasks, namely, Doc2Mask, Doc2Box, and Doc2Seq.", "questions": "See weakness 2.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This is clearly a very large and well-managed effort that contributes a resource that the community will benefit from. The authors are thoughtful and thorough in working to ensure the quality of the data.\n\n2. The proposed algorithm for hybrid data annotation is very rich and interesting. I'm not aware of any other successful orchestration of this number of models of various modalities, within a powerful human-in-the-loop iterative k-means clustering. There may be much of independent value in this process, beyond the benchmark itself.", "weaknesses": "1. The paper is particularly hard to follow! I cannot overstate this: it affects every section of the paper, from the abstract to the evaluation, so I'm not even 100% sure I understand the work. All sections contain a lot of unnecessary commentary (like comments on novelty or general motivations) that do not contribute a lot of value. At the same time, the background and organization are lacking. Please spend that space on background setup or technical details. For instance, half-way through the introduction, it was not really clear what Document Page Decomposition Dataset means (that's a lot of nouns in a sequence).\n\n2. What is the exact contribution in the paper, compared to the data source in Laion? Were the labels of Doc2Mask, Doc2Box, and Doc2Seq just used as-is from existing data or labeled in this work? It's a significant weakness in my opinion that I'm not sure how to answer this.\n\n3. The evaluation repeatedly claims that the authors \"carry out experiments using various baseline models and parameter configurations, confirming the effectiveness of ADOPD for document domain\". How does evaluating the models confirm the effectiveness of the dataset? It's very plausible in some ways, but could you be more explicit? How much of the data was confirmed to be of high quality and what kind of error analysis was conducted, etc., particularly for any labels (other than taxonomy, which is well-analyzed) generated automatically.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iFZ7FssnAC", "reviewer": "Reviewer_aArm", "summary": "This paper introduces a new document page decomposition dataset called ADoPD, mainly focusing on document tagging, segmentation, text detection and caption. Combining automatic procedure like machine/deep learning models and human annotator, this paper creates a much large and high-quality document dataset and still maintains low cost. Additionally, this paper conduct analysis of proposed dataset.", "questions": "1.\tThe caption annotation is just generated by BLIP-2 and there are not further correction or filter. I think this model-annotation might be meaningless for the area.\n2.\tThe section 4.2.3 Doc2Seq claims that 5000 samples are manually annotated, but after that, the paper says “utilized 80,000 samples for training, 20,000 for testing, and another 20,000 for validation“. It’s confusing how the manually annotation are collected and used, and why it has 20,000 validation and test samples – not consistent with 5,000 manual annotation.\n3.\tWill the dataset be open-source?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThis paper collects a large amount of data (120k), which has high resolution, wide source, and diverse appearance, the high quality data is beneficial for area. It offers segment, caption annotations which are rare in other document datasets.\n2.\tThe “hybrid data annotation” is novel and it helps to ensure the dataset is balanced and diverse. This proposed method alleviates the demand for human annotators and still maintains the high tagging quality for this dataset.\n3.\tThis paper offers quantitative evaluation for the diversity of data distribution, further confirmed the effectiveness of the “hybrid data annotation”.", "weaknesses": "1.\tThe detection and segmentation annotation quality are not well studied, they all come from pretrained model and corrected by human annotators. It’s better to give more detail about the procedure to correct the annotation. Especially there are some other document datasets like DocBank -- the box annotation is extracted from PDF by automatic tools and it should be much more precise than detection model.\n2.\tAlthough the proposed dataset is claimed to be suitable for caption, but the caption annotation is generated by BLIP-2 and there is no human correction. BLIP-2 is mainly trained for universal visual-language task and it might not suitable for document caption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uEKmX7HJnS", "reviewer": "Reviewer_BGf2", "summary": "This paper introduces a new annotated database to enhance automatic document comprehension and, more specifically, document page decomposition. The dataset was annotated multimodally to support 4 tasks: document entity segmentation, text detection, tagging, and captioning. The annotation was done in such a way as to combine the use of large pre-trained models and human expertise. Finally, a great effort has been made to maximize the diversity of document images.", "questions": "* This dataset would be very interesting to explore and obviously to use for pre-training. I think it would also be great to see what generalization capabilities a model trained on this dataset would have, once applied to another use case. This is not described in the paper, but do you plan to make this dataset public? If so, how do you plan to distribute it?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper presents a very significant and interesting work on data collection, showing a great diversity. Moreover, the use of numerous pre-trained models such as ViT, CLIP, or BLIP and the fusion of all the information predicted by these models within a single prompt is very interesting and rarely presented (and even used?) in research papers.\n* Even if this is not really shown in the results tables, it is clear that this dataset will certainly help research on documents, particularly thanks to the diversity of the images and the multiple annotations.", "weaknesses": "* While the use of 5 pre-trained models to pre-label images is quite innovative, it would also be interesting to indicate the cost of applying these different models to an image (and even to millions of images).\n* I found section 4.1 a little unclear. Although efforts have been made to explain the image selection process as fully as possible, I'm not sure that this methodology can be easily reproduced from these explanations.\n* In the results tables, the authors show how the different models compare once they've been trained on ADOPD. It would have been interesting to compare these models with and without fine-tuning on ADOPD to see the real impact of this dataset.\n* It would have been interesting to see results on other document datasets, starting from models pre-trained on ADOPD vs. on ImageNet for example. This would have shown the real gain of pre-training on a wide variety of document images compared to a large quantity of natural scene images.\n* The metrics used to evaluate the caption performance should probably be a bit more explained in the text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BK8gMgOlAe", "reviewer": "Reviewer_ZWQe", "summary": "### Summary:\nThe paper introduces \"ADoPD,\" a large-scale document page decomposition dataset designed for document understanding. This encompasses tasks such as document entity segmentation, text detection, tagging, and captioning. ADoPD is distinct with its novel document taxonomy, meticulously crafted through a data-driven approach that incorporates both large-scale pretrained models and human expertise. By merging outlier detection with a human-in-the-loop method, the dataset achieves a notable diversity. ADoPD offers deeper insights into document structures and significantly elevates techniques in document processing and analysis.\n\n### Major Contributions:\n1. **Systematic Exploration of Document Taxonomy**: The work presents the first systematic exploration of document taxonomy using a blend of machine learning models and a human-in-the-loop approach.\n  \n2. **Largest and Most Comprehensive Database**: The paper has developed the most extensive and comprehensive database for document image segmentation and object detection.\n\n3. **Thorough Dataset Analysis**: A profound analysis of the proposed dataset is conducted, accompanied by detailed experimental comparisons for entity segmentation and text detection tasks.\n\n4. **Quantitative and Qualitative Results**: The value of the dataset is highlighted through both quantitative and qualitative outcomes.\n\n5. **Advancement in Document Analysis**: The authors express the aspiration for ADoPD to serve as a catalyst in propelling research within the domain of document analysis.", "questions": "**Dataset Composition**:\n   Could you provide a more detailed breakdown of the types and sources of documents included in the ADoPD dataset? Understanding the diversity in terms of document genres, geographical origins, and linguistic variations would offer more insight into its applicability and robustness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### 1. Originality:\n\n- **Novel Dataset**: The introduction of \"ADoPD\" represents a fresh addition to the domain of document processing. Few datasets offer the same breadth in terms of document page decomposition.\n  \n- **Unique Taxonomy Approach**: The blend of data-driven approaches with human expertise in crafting a document taxonomy is a distinctive contribution. This combination brings together the best of automated and manual categorization.\n\n- **Human-in-the-loop Integration**: Incorporating human feedback in outlier detection for dataset diversity is an inventive methodology. This ensures that the dataset remains robust and versatile.\n\n### 2. Quality:\n\n- **Comprehensive Analysis**: The authors have delved deeply into the analysis of their proposed dataset, providing both qualitative and quantitative evaluations. Such rigorous examination is indicative of the dataset's quality and its potential applicability.\n\n- **Comparison with Existing Datasets**: While the main content delves deeper, the authors seem to be aware of the limitations of existing datasets, positioning ADoPD as a superior alternative. This indicates meticulous research and understanding of the current landscape.\n\n### 3. Clarity:\n\n- **Well-Structured Presentation**: The paper appears to be organized in a logical flow, starting from the introduction of the problem, moving to methodology, followed by results and conclusions.\n\n- **Illustrative Figures**: Based on the portions reviewed, figures like the overview of ADoPD annotations aid in visual comprehension, making the content more digestible.\n\n- **Explicit Problem Statements**: The authors have clearly laid out the challenges and questions they aim to address with their dataset, providing clarity of purpose.\n\n### 4. Significance:\n\n- **Filling a Gap**: With the increasing demand for automated document processing techniques, ADoPD addresses a significant gap in the domain, especially with its emphasis on diversity and comprehensive taxonomy.\n\n- **Potential for Future Research**: The introduction of such a dataset can catalyze further research in document understanding, segmentation, and object detection. It can serve as a foundational resource for subsequent works in the field.\n\n- **Broader Application**: Beyond academic research, the advancements proposed in the paper have potential real-world applications in areas like digital archiving, automated content extraction, and more.", "weaknesses": "### 1. **Depth of Comparative Analysis**:\n- **Weakness**: From the segments reviewed, while the paper introduces a new dataset, there seems to be a limited in-depth comparison with existing datasets.\n- **Recommendation**: A deeper comparative analysis highlighting the specific advantages of ADoPD over existing datasets would solidify its significance. Quantitative benchmarks against datasets like DocBank, if not already included in the deeper sections, would be beneficial.\n\n### 2. **Methodological Justifications**:\n- **Weakness**: The choice of the \"human-in-the-loop\" approach for outlier detection is novel, but the paper might not sufficiently justify why this method was chosen over others.\n- **Recommendation**: Delve deeper into the advantages and potential limitations of this method, comparing it with purely automated outlier detection techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)