AutoDocSegmenter: A Geometric Approach towards Self-Supervised Document Segmentation
Authors: Ankita Chatterjee
Anjali Raj
Soumyadeep Dey
Pratik Jawanpuria
Jayanta Mukhopadhyay
Partha Pratim Das
Venue: Accepted by TMLR ()

Abstract: Document segmentation, the process of dividing a document into coherent and significant regions, plays a crucial role for diverse applications that require parsing, retrieval, and categorization. However, most existing methods rely on supervised learning, which requires large-scale labeled datasets that are costly and time-consuming to obtain. In this work, we propose a novel self-supervised framework for document segmentation that does not require labeled data. Our framework consists of two components: (1) an unsupervised isothetic covers based pseudo mask generator which approximately segments document objects, and (2) an encoder-decoder network that learns to refine the pseudo masks and segments the document objects accurately. Our approach can handle diverse and intricate document layouts by leveraging the rich information from unlabeled datasets. We demonstrate the effectiveness of our approach on several benchmarks, where it outperforms state-of-the-art document segmentation methods. Our code is available at  https://github.com/ankitachatterjee94/AutoDocSegmenter

Published in Transactions on Machine Learning Research (01/2025)
AutoDocSegmmntee: A Geometric Approach towards Self- Supervised Document Segmentation
Ankita Chatterjee
ankita. Sesita.chhtteree222kkiiiin..n
Deportment of Computer Science and Engineering Indian Institute of Technology Kharugpur
Anjali Raj
anjjdh.ruj@kkgpicnn.iil.. Sttkgp. ac. ac.in
Department Computer Science and Engineering Indian Institute of Technology Kharagpur
Soumyadeep Dey
desoumyyo@micrrooof..cmm
Microsoft, INDIA
Pratik Jawanpuria Microsoft, INDIA
Jayanta Mukherjee
jay Bese. Kitkgp. ac.
Department of Computer Science ond Engineering Indian Institute of Technology Kharagpur
Partha Pratim Das
ppd@ashoka edu. in
Department of Computer Science
Ashoka Universily
Reviewed on OpenReview: https: // opw.r/weepp....e...mm/ net/ forum? id= JBueijjinn
Abstract
Document segmentation, the process of dividing a document into coherent and signifi- cant regions, plays a crucial role for diverse applications that require parsing, retrieval, and calegorization. However, most existing methods rely on supervised learning, which requires large-scale labeled datasets that are costly and time-consuming this work, we propose novel self-supervised framework for document segmentation that does not require labeled data. Our framework consists of two components: (1) un- supervised isothetic covers based pseudo mask generator which approximately segments document objects, and (2) an encoder-decoder network that learns to refine the pseudo masks and segments the document objects accurately. Our approach can handlo di- verse and intricate document layouts by leveraging rich information from unlabeled datasets. We demonstrate the effectiveness of our approach on several benchmarks, where it outperforms statc-of-the-art document segmentation methods. Our code is available at com/ http:///wwo...aaacccce/ooee/to/a
obtain. In
1 Introduction
Document digitization transforms paper-based documents into machine-readable formats that can be ac- cessed and processed by various devices and applications. Document layout analysis (DLA) is a crucial step in digitization, as it aims to extract the information and structural components of a document, such as text, images, tables and graphs (Namboodiri Jain 2007 Lee al.) 2019). DLA enables key-value information extraction and localization, which essential for tasks such text recognition, retrieval, segmentation,

---

Published in Transactions on Machine Learning Research (01/2025)
PRimA Dataset
PubLayNet Dataset
Global Adviser
Ji Advssr
Adviser
SelfDocSeg mask MNV3sm
AutoDosSegmenter- MNV3sm
SelfDocSeg mask
AutoDocSegmenter- MNV3sm
SwinDocSegmenter
SwinDocSegmenter
MNV3sm
Figure 1:
Segmenting typical document images from PRImA (Antonacopoulos et. al. 2009 and PubTayNel
[Zhong et al. 2019 datascts using SelfDocSeg mask (Maity et al. 2023) with MobileNet Vet3small (Howard
et al. 2019) backbone (MNV3sm), SuiinDeeSegmenter (Banerjee et al. 2023 and our proposed approach,
AutoLoocSegmenter-NN MM3.e.
and classification. However, the performance of these tasks depends largely on the quality and type of an- notations provided for the documents. For instance, while rectangular bounding boxes may suffice for text recognition and classification, but segmentation requires more accurate polygonal boundaries to capture the fine details of the document objects.
Existing works (Sun et al. 2019 Biswas et al. 2021 have explored Cone:eed-based models such Faster RCNN (Ren et al. 2015) and Mask RCNN (He et al. 2017b) for document object segmentation. More recently, transformer-baseee
architectures have shown superior performance over ConwNets by exploiting global attention layers for document segmentation (Appalaraju et. al., 2021 Huang et al.] 2022). However, these supervised models rely heavily on the availability and quality of labeled data for training. Labeling documents with polygonal boundaries for segmentation task is a tedious and costly process, and most large- scale benchmark datasets such as PubLayNet (Zhong et al.) 2019)) and DocLayNet (Pfitzmann et al.) 2022) only provide rectangular boundaries for the document entities. Such coarse annotations fail to capture the complex shapes and layouts of document objects (Harley et al., 2015 Clausner et al. 2019). limiting the teneralization ability of the segmenlation models. Therefore, it is essential to explore unsupervised or self-supervised segmentation approaches that can leverage unlabeled data.
The self-supervised mothods for natural images rely on contrastive learning to distinguish different classes of objects (Chen et al. 2020). However, this approach is not suitable for document images as they contain multiple objects of various classes in a complex layout. Moreover, feature matching methods that use global and local views without object localisation lose information and fail to capture the document structure and content. Recent works Li et al. 2021a Huang et al. 2022) have used supersuun for pre-training the encoder of document segmentation models, by combining mask language and image modeling to learn generic textual and visual features. However, these methods require supervised fine-tuning with annotated data for the segmentation task.
A vision-based self-supervised learning framework for DLA can address the limitations of existing methods, provided that the visual features are learned from reliable pseudo layout masks. However, the existing framework of SelfDocSeg (Maity et al.) 2023) generates pseudo layout masks by applying morphological operations such erosion, which are sensitive to the variations in the document characteristics. For example, the gray level intensity of text and image objects, the kernel size and shape for crosion, and the threshold value for binarization can affect the quality and consistency of the pseudo layout masks. Therefore, a fully automated SelfDocSeg segmentation pipeline may produce inaccurate or noisy masks during the training stage (Fig. 5)- To overcome this issuc, SelfDoaSeg uses self-supervision as a pro-training stop and fine-tunes the model on a specific dataset in supervised setting, but this compromises the generalization capability of the model across different. document types.
2

---

Published in Transactions on Machine Learning Research (01/2025)
In this paper, we present self-supervised framework for document segmentation that does not require human annotations. It has two components: an unsupervised method to generate polygonal masks (isothetic covers) that approximate document objects, such as paragraphs, images, tables, etc, and a learning- based method to refine them using an encoder-docoder network. Our unsupervised method improves on Biswas et al. (2010) 's grid-based technique by applying local and global thresholding to capture intensity variations and reduce binarization effects. We also propose a polygon merging algorithm that integrates both thresholding results and removes redundant or overlapping polygons, while preserving the structure, aligument, and isothetic properties of the document objects. It should be noted that the output quality of isothetic cover based approaches is influenced by the type of document, quality of binarization technique used, noise and artifacts present in the document, etc. To counter the above drawbacks, we introduce a learning based approach which leverages these pseudo masks as reference. The network learns to correct the inconsistencies and generalize to different document types, using a large-scale dataset of diverse document images. Overall, our main contributions are as follows:
We present AuteDoeSegmenter, a self-supervised framework for end-to-end document- image segmen- tation that leverages isothetic covers pscudo masks. AutoDoeSogmenter can adopt any encoder- decoder segmeniation model and we show ils versatilily and effectiveness with trausformer-based and ConwNet-based encoders.
We evaluate AutoDocSegmenter PRImÁ (Antonacoponlos et al. 2009 DocLayNet, PubLayNet, and Mo-Doc (Cheng et al.) 2023) datasets and show that it outperforms the existing baselines in both within domain and cross domain settings. Overall, we observe that AutolDoSSemmentr is able to generalize on complex layout images which are not observed during the training stage.
Figure shows sample segmentation results from SelfDoeSeg (Maity et al. 2023 SwinDDeeSementer (Baner- jee et 2023) and our AutoDoeSSementtr models. Since the pre-trained SelfDoeSeg model was unavailable, we followed train a MobileNet V3-small backbone (MNV3sm) using SelfDoeSeg's pseudo mask generation method (Maity et al. 2023). We observe that our method can segment various types of document images such magazine and scientific report images. In contrast, SwinDoo.Segenneer and SelfDocSey struggle to handle the diversity and complexity in document layouts.
The rest of the paper is as follows. Section 2 discusses related works. We introduce our self-supervised framework, SoeemeeSeg,,,,er, and discuss its components Section 3 In Section 4 we present a compre- hensive analysis of the experiments conducted to validate our approach, along with a detailed discussion of the observed results. Section 5 concludes the paper.
2
Related Work
In this section we discuss classical as well recent deep learning based DLA approaches.
2.1 Heuristic Rule-based Methods
Rule-based methods are typically classical approaches, which perform document segmentation by using pixel- level information. Based on the sequence in which segmentation is performed, these can be categorized top-down, bottom-up, hybrid. In top-down methods (Kise et al.
1998 Journet et al. 2005), the entire document is successively split into various components at each step and further into definite regions depicting similar entities. Bottom-up techniques (Saabni & El-Sana 2011 Asi et al. 2015) are usually more effective than top-down methods. They begin by considering every pixel as a separate cluster and at successive iterations, similar pixels are grouped together until homogencous regions are formed. Hybrid tochnique Tran et al. 2015) combine both the above approaches to obtain fast and generalized segmentation methods.
2.2 Using Convolutional Architecture
Over the last decade, ConuNet have been widely used for DLA and in particular for document segmentation. Farly works for page segmentation (He al. 2017a Li et al. 2020b) explored CNNs for extracting effective
3

---

Published in Transactions on Machine Learning Research (01/2025)
Table 1: Comparing the training characteristics of the state-of-the-aae techniques with our proposed ap- proach.
Method
Text features
Image features
Pseudo Self-supervised Supervised Self-superviser
masks
pre-training
fine-tuning
training
Layout Parser (Shen et al.
2021
DoeSegTr (Bswwss
2022
Lvy Huang at al.
2022
X
UDoc (Gu et al. 2021
X
X
DiTRASE [T et al. 2022
X
V
X
X
MaskRCNN [e of al.
2017b
X
V
FasterRONN Ren el
2015
X
V
SwinDonSegmennter Banerjee al.
2023
X
BYOL Grill al.
2020
X
X
SellDocSeg (Maity cl al.
2023
V
AutoDocSegmenler (ours)
V
visual features. Several CNN based seginntattio approaches have focused specific document types such as historical documents (Chen et al.] 2015:
2017 Oliveira et al. 2018 newspaper articles (Almutairi & Almashan 2019), and scientific publications (Biswas et al. 2021 Yang & Hsu 2021). Researchers have also developed methods on recognizing and segmenting specific objects. For instance, DeepDeSRT (Schreiber et al. 2017) and CascadeTabbie (Prasad et. al. 2020 aimed at detecting and segmenting tables. Lin al. (Lin al. 2021) developed character (text region) detection technique based on RetinaNet (Lin al, 2017) and transfer learning. [Saha et al. (2019) used transfer learning on Faster-RCNN backbone to segment graphical objects in documents. To account for the domain shift in the documents present in training and test datasets, Li et al. (Li et al.] 2020a) proposed cross-domain document object detection. LayoutParser (Shen et al.] 2021 is a repository of pre-trained CNN models for layout detection, character recognition, and various other document processing lasks.
Recently, Maity et al. (0033) developed SelfDocSeg which performs self-supervised pro-training for document- image segmentation by combining Bootstrap Your Own Latent (BYOL) method (Grill et al. 2020) and focal loss the pseudo masks. The pseudo masks generated using morphological operations such erosion. However, erosion-based masks potentially have drawbacks such as: (i) they may not not cover the document objects entirely: (ii) they may distort the structure and shape of the document objects; and importantly (iii) they are sensitive to the resolution of the image and the size and shape of the erosion kernel. To overcome such issues, SelfDoeSeg finally fine-tune the network with labeled datasets.
2.3 Using Transformer Architecture
Transformer based models gained popularity in computer vision after the success of Vision Transformers (ViT) (Dosovitskiy et al. 2021). Li et al. (Li et al. 2022) proposed self-supervised document image trans- formers DiT and showed its effectiveness in various downstream tasks like document classification, layout analysis, table detection, and text detection (OCR). However, the pre-training for DiT is done on large- scale unlabelled document images and is not applicable to small scale magazine datasets (e.g., PRImA) with dataset-speciffc attributes (Banerjee et al.] 2023). StrucTexT (Li et al., 2021bb employed multi-modal transformers for understanding structured text. Although Strue TexT obtains good performance both segment and token levels, it gets confused when the textual contents are somantically related and closcly placed. Yang & Hsu (2022) employ OCR based text extraction perform segmentation PubLayNel dataset. LayoutLM (Xu et al. 2020) and LayoutLMu3 (Huang et al. 2022 perform joint learning of layout, visual, and text features for visual document understanding tasks.
Biswas el al. (Biwwas el al.) 2022) proposed DocSeg Tr using ResNel-FPN backbone on transformer. Since To employs self attention mechanism, it is able to achieve faster convergence on small-scale datasets and obtain competitive performance. SwinDocSeementer (Bancrjee et al. 2023) is state-of-the-art instance segmentation method on complex layout document images. It employs a supervised Twiinrransfrreer fea- ture extractor backbone. This is integrated with transformer-based encoder-decoder architecture which
4

---

Published in Transactions on Machine Learning Research (01/2025)
Fasiidae peremaing
Fiaa FPA Decoder combired Crewer baibbeni
Segrnemtii masts
Figure 2: Schematic diagram of our framework, AntoDocSegmenter. The isothetic covers are generated using five types of rules Section 3.1 and the isothetic masks extracted are used as reference to train an encoder-decoder model. Different encoder backbones (e.g.. MNV3sm, MiT--O, etc.) may be employed. We fix the decoder network to feature pyramid network (FPN).
fine-tunes the downsampled features from the backbone using the annotations in supervised learning paradigm. Some approaches (Li et al. 2021a Kim et al., 2022 use cross-modality encoder during self- supervised pre-training phase to recognise the types of documents and segment entities. Recent techniques (Appalaraju et al., 2021 Kim et al. 2022 Gu et al. 2021 2022 have also focused on general purpose features which are generated using unified pre-training on multiple downstream tasks. However, such mothods have inherent biases towards classes with more number samples and fail to perform during domain shift.
2.4 Distinguishing AutoDocSegmmetter with Existing Methods
The existing (partially) self-supervised methods (Huang et al. 2022 Maity et al., 2023 Banerjee et al. 2023
mainly employ self-supervision for pre-training and subsequently perform a supervised fine-tuning step for dataset specific segmentation. The supervised dataset is typically annotated using rectangular boundaries which vaguely distinguishes different objects present in the image. However, for a detailed document analysis, it is essential to capture the structure of these entities. On the other hand, our fully self-supervist, approach, AutoDocSegmenter, is able to annotate well as segment the document images, using polygonal boundaries to outline different objects present in the document. Empirically, AuloDocSegmenter generalize well to various document layouts without additional fine-tuning. We enlist the distinctions of our method compared to the existing doep-learning based document sogmentation approaches in Table []] We note that all the methods in Table except our AutoDocSegmenter have a supervised fine-tuning step.
3 Proposed approach: AutoDocSegmenter
In this section, we discuss our self-supervised framework for document segmentation. Fig. illustraee the main components and workflow of AutoDocSeemmeter. In Section 3.1 we discuss unsupervised pseudo masks generation mechanism. The pseudo masks are subsequently used to train the segmentation model, as detailed in Section 3.2
3.1 Robust isothetic covers using local and global information
Isothetic covers are axis-aligned polygons that encloses the objects in binary images. They capture object structure and geometry with low cost. Variation in document type and binarization quality may render the isothetic covers noisy. In particular, binarization is challenging because single/global (binarization) threshold may preserve all document objects, while multiple local thresholds may introduce noise arti-
5

---

Published in Transactions on Machine Learning Research (01/2025)
(b)
(c)
The directions of traversal depending on the oceupancy of the four grids: (a) type 1 vertex with exactly one grid occupied, (b) type 2 vertex where adjacent grid are occupied, (c) type 3 vertex with diagonal vertex occupied, and (d) type 4 vertex with exactly three grids occupied.
Figure 3:
facts. Hence, we propose novel isothetic cover algorithm using both local and global binarized images. In the following, we first detail the vanilla isothetic cover algorithm Biswas et al. 2010) and then present our improved isothetic cover algorithm.
Isothetic covers: To form polygons that enclose the objects with minimal area, an input image divided into grids using fixed-distance horizontal and vertical lines between them. The grids are traversed in a row- major order, and the vertices of the polygons are determined by the occupancy of the four grids surrounding each grid point, which indicates whether not objects are present within them. The method classifies the grid points as vertices of different types and angles using six criteria (Biswas et al. 2010
1. A grid point is not vertex and is skipped if none of the four grids are occupied.
2. A grid point is vertex of type 1 with 90 angle and a value of t if exactly of the four grids is occupied. The method turns to the occupied grid and continues along the same row or column
(Fig. 3 (a))-
3. A grid point is vertex of type 0 with 180 angle and value of if two adjacent grids are occupied. The method follows the edge between the occupied grids and moves to the next row column (Fig. 3 (b)).
4. A grid point is vertex of type -1 with a 270" angle and value of / 1 if two diagonal grids are occupied. The method turns to the unoccupied grid and advances along the same row or column
(Fig. 3 (c)).
5. A grid point is also vertex of type with 270 angle and value of 1 if three of the four grids are occupied. The method turns to the unoccupied grid and proceeds along the same column (Fig. 3 (d)).
6. A grid point is not vertex and is skipped if all four grids are occupied.
The detailed algorithm in discussed in the Appendix A.2 The isothetic covers generated without any backtracking and in linear time with respect to the perimeter of the polygon. Pseudo masks are generated by filling these polygons. We refer to this stage of training pipeline with the conventional isothetic covers (Biswas et al. 2010) as AutoDocSegmenter....
Binarization and Merging polygons: Binarization facilitates the extraction of isothetic polygonal covers from document images, by enabling clear distinction between foreground and background regions. However, the quality of binarization depends on the choice of the threshold value, which can affect the preservation of information the exclusion of noise. Different document objects may have different gray levels, making it challenging to find a single global threshold that separate them from the background. On the other hand, applying multiple local thresholds on small image patches may introduce noise artifacts such as bleod-through, speckles, or uneven illumination. We introduce a hybrid method for obtaining the isothetic covers from a document image, which combines the advantages of global and local thresholding techniques. Importantly, our method maintains the shape and orientation of the document elements, and produces isothctic polygons that are aligned with the axes and independent of the resolution. This facilitates effective polygon merging with minimal loss distortion of information.

---

Published in Transactions on Machine Learning Research (01/2025)
Shve Baliiet
(1)
PN)
(m)
Figure 4: Annotating documents using isothetic covers. (a)-(e) Results from PRImA dataset, (n-(g) Results from - dataset, (b)-(i Results from PubLayNet dataset, and (k)-(n) Results from DocLayNet. The algorithm is able to annotate complex layouts of in-line images, tables, circuits and graphs.
For each image (7), we generate two types of binarized images: 1) a binary image obtained from global thresholding (B,())), and 2) a binary image obtained from local thresholding (Bi()). For local thresholding, we divide the image into non-overlapping sliding windows of size w and apply binarization technique on each window independently, where w is computed as 1/5th of the minimum dimension of the image. We then apply the isothetic covers algorithm both the binary images to obtain sets of isothetie polygons that approximate the document objects. To merge the two sets of polygons, aims to preserve the information from both the global and local binarization. The selected polygons are then considered as the final set of isothetic polygons. We then use the region filled isothetic polygons as pseudo- segmentation masks to train our segmentation model. We refer to this stage of the training pipclinc with our modified hsothetic covers as AutoDocSegnenter-U Thus, while AutoDocSegmenlerr/ generales pseudo masks using the conventional isothetic covers (Biswas et al. 2010). AutoDocSegmenter- generates pseudo masks using the proposed (Algorithm D
propose novel Algorithm ! that
Fig. shows the polygonal masks for various document objects obtained using AutoDocSegmenter-- on PRImA, M6-Doc, PubLayNet, and DocLayNet datasets. We observe that the obtained (unsupervised) polygonal masks approximate the shapes of the enclosed objects. Thus, they provide a coarse but rea- sonable supervision for the (next) model to learn the boundaries of the documentt objects.
3.2 Segmentation Model
We adopt a feature pyramid network (FPN) decoder based segmentation architecture for document segmentation task. FPN is general-purpose decoder that work with various encoder backbones and
Algorithm Merging Polygons
S {x is point in Polygon P. of B2(1)}
S = x' x' is point in Polygon Pt of B((
P = Final set of merged polygons
S, = S then
P, + Pg Pi
else if S [ Sg
& Sknsl1000 0.5|S then
P1 P
else if S, C S & ssss 2 S. then
Pf + P
end if

---

Published in Transactions on Machine Learning Research (01/2025)
produce multi-scale feature maps from single imput image. FPN is suitable for our task as it can capture both fine and coarse details of the document layout.
Encoder block: Our self-supervised method does not rely on any specific encoder architecture or pre- training scheme, as only uses conventional learning of isothetic masks. Therefore, we can integrate our method with different types of encoder backbones, ranging from lightweight. to heavyweight models, such as EfficientNees (Tan & Le, 2019), Mobile.Ned V2 (Sandler et al., 2018), Ve (Howard et al. 2019), Mir Transformer (Xie et al.) 2021), ResNeSt (Zhang et al,j 2022), MobileOne (Vasu et al.) 2023). ResNeXt [Xie et al., 2017), RegNet (Xu et al. 2022). These encoder backbones extract feature maps at different levels of resolution and semantic richness from the input image.
Document segmentation is a challenging task that requires capturing both fine-grain and coarse features of diverse real-world documents. A single architectural choice for the encoder-decoder network may compromise the procision of the segmentation masks. To overcome this limitation, we adopt the Feature Pyramid wetwork (FPN) decoder, which produces multi-scale feature maps by fusing low-level and high-level features from the encoder backbone. The FPN decoder leverages the spatial resolution of the low-level features and the semantic information of the high-level features to enhance the accuracy and robustness of the document segmentation. In particular, the FPN decoder fuses the multi-scale features from the encoder backbone through a top-down pathway and lateral connections. The top-down pathway upsamples the highest level encoder feature map progressively by nearest neighbour interpolation and adds it element-wise to the corresponding encoder feature map, which is first reduced to the same number of channels by 1 convolution. This fusion process is repeated for all encoder levels, creating a set of merged feature maps. To reduce the anti-aliasing effect of upsampling each level a 3 convolution is applied. Finally, the refined feature maps are passed through common classification layer to obtain the final segmentation mask.
Decoder block:
Loss function:
Our model aims to segment document images using polygonal masks where the reference annotation is the isothetic covers generated in an unsupervised manner. We employ the Dice loss between the generated and reference masks. Let y represents the pseudo annotations generated using isothet covers and p represents the predicted segmentation generated by the model. Then, the Dice loss is computed as
2yp + 1
L(y.p) = 1- y + ++
I+p+I calculates value between 0 and 1, with indicating no similarity between the two segmentation, and 1 indicating complete similarity. Thus, the loss become when the masks are completely similar. The model is trained using Adam optimizer.
The formula 2up+1
4 Experiments
We evaluate the proposed AutoDocSegmenter approach on different benchmarks datasets by (a) comparing it against state-of-tho-arr document segmentation methods (Section 4.2p and (b) performing ablation studies with varying backbone feature extractors (Section 4. polygon merging thresholds (Section 4.3.2) bina- rization techniques (Section 4.3.3). grid size of isothetic covers (Section [1..4), image sizes (Section 4.3.5
and document object types (Section 4.3.6 We begin by detailing our experimental sctup.
4.1 Experimental Details
Datasets: We evaluate our method on four popular document segmentation datasets: PRImA {Antona- copoulos et al. 2009), DocLayNet (Pfitzmann et al 2022), PubLayNet (Zhong et al. 2019 and M Doc
[Cheng et al. 2023). PRImA has 382 and 96 images for training and testing, respectively, with polygonal an- notations for each entity. DocLayNet and PubLayNet have rectangular annotations and contain 69 375/6449 and 335 703/11 11 245 images for training/test ing, respectively. M is a recent dataset with only a test set of 2724 images. Please refer to Appendix A.I for more details.
Evaluation Metric: We use the mean average precision (mAP) metric to compare segmentation models, which averages the intersection over union (ToU) scores of predicted and groundtruth masks across documents
8

---

Figure 5: Comparison of pseudo masks generated by SeliDocSeg Maity et al. (023) and AutoDocSegmenter with the modified AutoDocSegmenterr-U (a) This is an example for a document image where the AutoDocSegmenter-I fails to capture texts at the bottom written in bold while the AutoDocSegmenter--t Published in Transactions on Machine Learning Research (01/2025)
The Science Of Addiction
SelfDocSeg Autooooeeegmnterrl
AutoDocSegmenter-
(8
SelfDocSeg masks
AutoDocSegmenteri
AutoDocSegmenterrJ
(b)
covers these regions. (b) SelfDocSeg does not mask the table region in the image unlike AutoDocSegmenter C which precisely covers it.
and IoU thresholds from 0.5 to 0.95 steps of 0.05. This follows the Microsoft COCO benchmark protocol [Banerjoe al. 2023).
AutoDocSegmenter training and evaluation: As discussed in Section 3.1 the first stage of our pipeline generate pseudo masks for each object without using any supervision. For both isothetic covers as well as the proposed modified isothetic covers (Algorithm D. we employ Otsu's thresholding technique (Otsu 1979 for global and local thresholding, as it can automatically find the optimal threshold that minimizes the intra- class variance of the pixel intensitics. The model trained with the Adam optimizer and a learning rate of 0.001 for 50 epochs, and the learning rale is lowered by a factor of 10 every 10 epochs. All document images are resized 256 pixels. We do utilize any annotations from the datasets during training. We assess the model's segmentation quality by comparing its results with the groundtruth labels from the datasets.
4.2
Experimental Analysis
In this section, we compare our AutoDocSegmenter approach with the existing approaches in terms of the pseudo-mask generation quality as well as the overall segmentation generalization performance.
4.2.1 Comparison of pseudo masks
To evaluate the effectiveness of our dual binarization and polygon merging algorithm, we first compare the masks generated by AutoDoeSegmenter-- with the existing isothetic covers algorithm (AutoDocSegmenter-I] Biswas et al. 2010) and crosion-based SeliDocSeg Maity et al. (2023) masks in Fig.
Fig. 5 a) shows that AutoDocSegmenter1 fails to segment the bold text at the bottom of the image, as it relies on gray level intensities that vary across the document. The erosion- based masks produced by SelfDoeSeg (Maity et al. (2023) create a binary version of the original image, where text elements are morged together. This does not provide useful segmentation masks for training models, loses the distinction between different text regions. While the erosion window size of SelfDooSeg be adjusted to improve the

---

Published in Transactions on Machine Learning Research (01/2025)
Table 2: Comparing the mean average precision (mAP) of AutoDocSegmenter-- and SelfDocSeg masks on PRImA, DocLagNet and PubLayNet datascts.
Masks
PRImA DocLayNet PubLayNet
SeliDocSog mask (Maity et al. 2023
66.67
77.0
79.90
AutoDocSegmenter-U
82.20
82.90
84.40
Table 3: Comparing the mean average precision (mAP) when each model is trained on PublayNel, and evaluated on PRImA, DocLayNet and M -Dc datasets.
Method
PRImA DocLayNet
LayoutLMv3 (Huang et al. 2022)
4.35
2.30
5.30
SwinDocSeguentter (Banerjee el al. 2023
43.80
10.00
33.40
MuteDDosSegmmenter--AA11111111 (ours)
89.57
83.21
75.22
AutoDDoskegmmnttr---AA1V333 (ours)
85.46
83.43
74.26
AutoDDoSeeguenntee---MM111--- (ours)
89.65
83.80
75.56
MuboDeoSsgnmeeiiee--MMMVVSSS (ours)
87.76
83.66
75.24
text segmentation, does not affect the segmentation of other objects. In contrast, AutoDocSegmenter segments the document by tracing the boundaries of the objects, resulting well-defined masks for each text region. The difference in the masks is more evident in figures, tables and graphs, where AuloDocSegmenter U groups each object parts, seen in Fig. b))
single unit, while SelfDocSeg and AutoDocSegmenter-- split them into multiple
We also evaluate the pseudo masks from SellDoeSeg (Maily et al. 2023 and AutoDocSegmenter--U hods with respect to the given annotations in Table 2 We observe that AutoDocSegmenter- outperforms SelfDoc- Seg by extracting higher quality pseudo-masks on all the three datasets. Overall, our AutoDocSegmenterr preserves the structure and alignment of the document objects while ensuring that they remain axis-parallel and resoltiioniindependent This enables precise capturing of the structure of the objects with minimal information loss or distortion. On the other hand, morphological operation- SelfDocSeg Maity et al.) 2023h does not capture the shape of the object precisely and distorts with varying intensities.
4.2.2
Comparison against existing approaches
We next compare AutoDocSegmenter against recent document segmentation approaches two settings: (1) methods are trained on a large unannotated corpus (PubLayNet) and evaluated on unseen datasets (PRImA, DocTayNet, and M6-Doc), and (2) traīning and test sets belong to the same dataset. In this section, we report results of AutoDocSegmenter with lightweight encoders (MiT--BO and MobileNet V3-small) V3 which offer significant advantages in terms of parameter and computational efficiency. These are important preroquisites for mobile applications (e.g., Microsoft's M365 and Office Lens apps, Adobe Scan app, etc.) that demand fast and reliable document layout analysis. Thus, such variants of AutoDoeSSemmnter enhance its applicability in real-world scenarios. In Section 4.3 we also evaluate the performance of AutoDocSegnenter with various feature extractors.
Setting We evaluate the generalization ability of our method and the basclines in the scenario where the training and test sets come from different sources. This is realistic situation where one has access to large amount of unlabeled images for training and wants to apply the segmentation model to specific dataset (e.g,, new user data) that may have different characteristics. The baselines we compare with are Lo3 (Huang et al.) 2022) and SwinDoesSegmeneer (Banerjee et al. 2023), whose pre-trained models (on PubLayNel dataset) are publicly available. Table 3 shows the performance of different methods when they are trained on PubLoyNet and tested on PRImA, DocLayNet or 6
We observe in Table 3 that our AutoDocSegmenter approach achieves the best results on all the three test datasets, outperforming both state-of-the-art baselines by large margin. This indicates that
10

---

Published in Transactions on Machine Learning Research (01/2025)
Table 4: Comparing the mean average precision (mAP) of our proposed method with state-of-the-at tech-
niques on PRImA, PubLayNet and DocLayNet datascts. M denotes number of parameters in millions.
Method
M PRImA DocLayNet PubLayNet
Layout Parser (Shen et al.
2021
64.70
86.70
DocSegTr (Biswas al.)
2022
42.50
90.40
Layo (Huang
2022)
133
40.30
95.10
UDoc (Gu et al. 2021
272
93.90
BATE Li et al. 2022
87
93.50
MaskRCNN He et al.
2017b)
63.7
73.50
91.00
FasterRONN Ren et al.
2015
19
73.40
90.20
SwinDocSegmenter [aanereee et al.
2023
223
54.39
76.85
93.72
BYOL Grill et al.
2020
94
28.70
63.50
79.00
SclfDocSog (Maity et al.
2023
52.10
74.30
89.20
AutoDooSeginntte--MMiii T-O (ours)
5
79.12
87.20
84.50
AutoDDoeSegmennter<--MWVV 333m (ours)
4.1
80.72
87.14
85.10
AutoDDeSSgmennnee--MMT--- (ours)
5
80.51
82.50
85.20
AutoDooSegmontcc--MNN\333 (ours)
4.1
81.47
85.97
85.20
approach generalizes well to different lypes of documents and captures the layout information effectively. Both LayoutLMv3 and SwinDocSSemeeter are large-scale transformer- based models. We note that both the above basclines struggle to adapt to the diverse domains and styles of the documents, and may suffer from overfitting to the PubLayNet dalaset.
We also note from Table that AuteoDocSeegmenter-Miii-- has slight edge over AutoDocSegmmente MNV3sm. MiT-BO is based on the transformer architecture, which can capture the long-range dependencies and the spatial relations in the documents, while MNV3sm is based on the mobile network architecture, which can reduce the computational cost and the runtime memory usage.
Setting 2.
We next evaluate the models when the training and test sets belong to the same datasct Table In addition LayoutLMu3 and SwinDDeSegmenter, we also report the results obtained by Layout Porser (Shen et al. 2021), SelfDocSSg (Maity et al.) 2023), DocSegTr (Biswas et al. 2022 MaskRCNN (He) et al. 2017b), and FasterRCNN (Ren et al. 20155). For LayoutLMu3 and SutinDocSemmnnee,
obtain the
results using the pre-trained weights provided by the authors while for other baselines, we report the results as published in (Shen et al. 2021 Biswas et. al.] 2022: Zhong et al., 2019). Overall, AutoDocSegmenter
demonstrates superior performance across datasots. In particular, we make the following observations:
PRImA dataset contains complex and diverse document layouts with non-rectangular annotation boundaries. The results in Table show that AutoDocSegmenter is able to handle the variations and challenges of different document shapes and structures.
DocLayNet dataset consists of document images with rectangular annotation boundaries and various document structures. AutoDocSegmenter outperforms the basclines on this datasot, indicating its robustness and adaptability to different layout styles.
PubLayNel a large dataset, consisting of research document images with non-overlapping rect- angular annotation boundaries and relatively simple and regular layouts. LayoutLMv3 is the best performing method this dataset.
In the model training phase with unsupervised annotations, we observe two kinds of outcomes: (a) the model mimics some of the annotation errors, such as careful outlining of in-line images, in the segmentation masks, and (b) the model deviates from some of the annotation errors, and thus discards some of the wrongly labeled areas. These outcomes indicate both the advantages and disadvantages of using an unsupervised annota- tion process. However, as shown in Table AutoDocSegmenter generalize effectively to various unseen
11

---

Published in Transactions on Machine Learning Research (01/2025)
Strange customs
INNOVATION LANGUAGE ACTION
S
Results Swimaneeagmentrr
Strange customs
INMOMAATOON -OS
S
Results from Duto SegmenterMM (ours)
Figure 6: Comparing the segmentation results from PRTmA datasets using AutoDoeSogmento--MiT- model with SuiinDeeeemeeneer Our algorithu captures the layout of the objects more precisely compared SwinDeeSeementer (Banerjee al. 2023).
datasets, which lowers its dependence on data-specific hyperparameters. Hence, we see that AutoDocSeg- menter achieves better performance on the PRImA dataset when self-supervised on the PubLayNet dataset (Table 3) than when sell-supervised on the PRImA dataset (Table 4..
Fig. shows some examples of the segmentation results from the PRImA dataset using our AutoDocSegmenter-UUMMe U-MiT-B model. We observed that our proposed approach can capture complex struc- tures, such as tables, figures, captions, and in-line images, which often missed by SwinDocSegmenter. This shows that AutoDocSegmenter handle various document layouts and content types, and produce accurate segmentation masks.
4.3
Ablation studies
To evaluate the impact of our training parameters, we now discuss ablation studies performed with the proposed AutoDocSegmenter framework under different settings.
4.3.1 Performance on various backbone feature extractors
We begin by evaluating the performance of AutoDocSegnenter--U using different CNN and transformer based backbone feature extractors (combined with the FPN decoder). We train the models using image size of 256. Table 5 compares the performance of AutoDocSogmenter-- on various backbone fcature extractors on two datasets: PRImA and DocLayNet. In addition to mAP metric, we also report intersection over union (ToU) scores. We observe from Table that:
The transformer-based MiT-B4 performs worse than the much lighter MiT-BO backbone or the CNN-based backbones. This may indicate that requires more data and fine-tuning to achieve comparable results.
Among the CNN-based backbones, AntoDocSegmenter- DnS-eegmmeette---EEffitttttteee variants are among the best methods on both datasets. However, these backbones also have the largest number of parameters, ranging from 5.3M to 30M, which may limit their applicability on resource-constrained scenarios.
AutoDDoSSggnente----oooo U-Mobill variants, with MobileNotV2 and MobileNetV? sm backbones, show competitive performance on both datasets. The number of parameters in these backbones range from 2.5M to 3.4M, making them efficient and lightweight. for mobile scenarios.
12

---

Published in Transactions on Machine Learning Research (01/2025)
Table 5: Comparison of the performance of AutolocSeegmente on various backbone feature extractors on PRImA and DocLayNet datascts. M denotes number of parameters in millions.
PRImA
DocLayNet IoU
Backbone
M
IoU
mAP
mAP
EfficientNet-BB
Tan & Le
2019
5.3
0.65
84.11
0.83
97.56
FfficientNet-B33
Tan & Le
2019
12
0.65
86.20
0.84
98.80
EfficientNet-B55
Tan & Le
2019
30
0.69
82.42
0.87
99.20
MobileNetV2 Sandler et al. 20188)
3.4
0.65
80.01
0.86
96.37
MobileNet sm (Howard et all
2019
2.5
0.67
81.47
0.84
96.30
MobileOnc-SO (Vasu et al.
20233
2.1
0.42
42.70
0.70
70.20
RegNetXX-200
(Xu et al.
2022
2.7
0.42
42.70
0.70
70.20
ResNeSt-14
Zhang et al.
2022
8
0.67
84.08
0.84
96.87
ResNeSt-26
Zhang et al.
2022
15
0.63
75.73
0.83
97.10
ResNeSt-50 Zhang et al.
2022
25
0.64
79.26
0.88
98.20
ResNeSt-101 /Zhang a...
2022
46
0.62
82.83
0.89
98.62
ResNeXt-50 (Xie et al. 2017
22
0.65
75.70
0.89
98.30
MiT-BO Xic et al.
2021
3
0.67
80.51
0.81
93.64
MiT-B4 Xie et. al.
2021
60
0.66
80.33
0.70
70.20
Table 6: Analysing the performance of AutoDocSegmenter--U with different overlapping threshold while merging on PRImA and DocLoyNet datasets.
PRImA
DocLayNet
Threshold (%)
IoU
mAP
IoU
mAP
25
0.71
77.60
0.78
85.00
50
0.74
82.20
0.79
82.90
75
0.67
72.70
0.79
85.20
Other CNN-based backbones, such as MobileOne-SO, RegNetX-200, ResNeSt variants, and ResNeXt- 50, have mixed results on both datasets.
In the following, we perform additional studies on AuuoDDcSeggnentee-...MTTT--- and AutoDocSegnenttr- MobileNet V3 small as they are among the lightest models with transformer-base and CNN-based backbones, respectively, and achieve reasonably good generalization performance.
4.3.2 Performance with different polygon merging thresholds
AutoDocSeementerrU U uses a polygon merging technique to mitigate the influence of binarization thresh- olds on the overall quality of pseudo masks. The polygons generated using local and global thresholding are merged there is an overlap of more than a given threshold. Table D analyses the performance of AutoDooSSegmenter-T using different overlapping threshold values on PRImA and DoeTayNel datasets. We observe that the default threshold value of 50% is a robust choice across datasets. Setting it low or too high may result in over- or mnder-segmentation, observed in the PRImA datasct.
4.3.3 Performance with different binarization techniques
Document images often have regions with different gray levels, which pose challenge for foreground- background separation using a single threshold value. For instance, documents with text and images, varying background colors, or noise effects such as shadows and low lighting, require different thresholds for different parts of the image to preserve the object shapes and details. A global threshold may fail to capture the contrast brightness variations across the image Yan et al. (2005), while a local threshold based on fixed window size may miss some objects introduce artifacts due to the local background variation Wanas et al.
13

---

Published in Transactions on Machine Learning Research (01/2025)
Verbatim
$36. 357
26,00 0000
$360
9780
a
Figure 7: Pseudo masks generated for (a) the input image after binarisation using (b) Otsu's global thresh- olding, (c) Otsu's local thresholding and (d) Otsu's 's local+global thresholding. In (b), we observe that the global thresholding approach fail to differentiate between the foreground and the background in the right part of the document because of different grey-level of the background. In (c), we observe that the local thresholding approach unnecessarily captures very fined grained details, which would easily be influenced by noise. It also fails to capture objects near the bottom right of the document. In (d), the joint global and local approach combines the advantages of both the approaches and gives the best result.
2006 To overcome these limitations, we explore an approach (Otsu's global + local thresholding) that adapts the threshold value to the global and local characteristics of the image. Fig. 7 shows the robust performance of this method over global or local thresholding alone on complex document images.
In Table 7 we compare the binarization performance of Otsu's global local thresholding technique with the following baselines: (a) Global thresholding technique, where we set the threshold value to 147; (b) Otsu's global thresholding method (Otsu 1979); (c) Adaptive mean thresholding, which is local thresholding algorithm with window size fixed at 150; and (d) Otsu's local thresholding technique, where the threshold for every window is computed using Otsu's algorithm. In Table 7 we observe that Otsu's global + local thresholding performs competitively against. the baselines while retaining the benefits of both local and global thresholding approaches. We note that our datasets contain limited number of complex images, which reduces the advantage of combining global and local thresholding techniques that can handle diverse document types. Therefore, we observe similar results from global thresholding and the combined method.
4.3.4 Varying grid size of isothetic covers
The quality of the isothetic masks is influenced by the grid size used for generating the isothetic covers. A suitable grid size should correspond to the scale and the variety of the document layout, preventing over- segmentation or under-segmentattin of the layout elements. To measure the influence of the grid size on the segmentation performance, we calculate the intersection over union (IoU) and the mean average precision (mAP) of the isothctic masks with respect to the groundtruth masks.
Table 7: Analysing the performance of different binarization techniques in AutoDocSegmenter-l
PRImA
DocLay
Binarization method
loU
mAP
loU
mAP
Global thresholding
0.73
81.40
0.78
82.40
Otsu's global thresholding
0.74
82.40
0.79
86.00
Adaptive mean thresholding
0.72
79.80
0.77
82.00
Otsu's local thresholding
0.66
77.30
0.76
81.00
Otsu's global + local thresholding
0.74
82.20
0.79
82.90
14

---

Published in Transactions on Machine Learning Research (01/2025)
Table 8: Analysing the performance of AutoDocSegmenter U with different grid size for isothetic covers on PRImA datasct.
Encoder
Grid size
IoU
mAP
AP@50
AP@75
10
0.42
42.70
42.70
42.70
12
0.42
42.70
42.70
42.70
MIT-BO
15
42.70
42.70
42.70
18
0.67
78.30
77.50
78.50
10
0.55
56.71
55.80
57.50
12
0.60
63.35
61.90
64.20
MNV3sm
15
0.64
70.94
69.50
71.80
18
0.65
73.80
74.60
76.50
Table Analysing the performance of AutoDocSegmenter with different image sizes on PRImA dataset.
Encoder
Image size
loU
mAP
AP@50
AP@75
1024
0.67
80.36
79.60
80.50
MO
512
0.69
79.53
78.80
79.70
256
0.67
78.30
77.50
78.50
1024
0.67
80.31
79.20
80.90
MNV3am
512
0.66
77.53
76.20
78.10
256
0.65
75.80
74.60
76.50
Table [ shows the results for the PRImA dataset using the MiT-BO and MNV3sm backbones. We observe that a grid size of 18 achieves the best performance for both encoders, attaining the highest IoU and mAP scores. Smaller grid sizes result lower ToU and mAP scores due excessive splitting of the isot hetic covers. Likewise, find an appropriate grid size for DocLayNet and PubLayNet as and 10, respectively, by following the same procedure.
The proposed method relies on selecting an appropriate grid size for each dataset, which may vary depending on the characteristics of the document images, such as resolution, density, and diversity of layout elements. Therefore, we conduct hyperparameter tuning to determine the grid size that optimizes performance on have observed that the optimal grid size approximately
th of the minimum dimension of the image. Based this observation, set the grid size to be 1 of the minimum dimension of the image in all of our experiments.
the validation set. Empirically,
100
4.3.5 Varying image size
We next evaluate the effect of different image sizes on the segmentation performance of AutoDocSegmenter using the PRImA dataset. We use MiT-BO and MNV3sm encoder networks to extract features from the input images, and compare the results using four metrics: IoU, mAP, AP@50, and AP@75. Table g summarizes our findings. Table g reveals that the segmentation performance is influenced by the image size, and that the optimal size varies depending on the encoder network. For both encoders, the highest resolution of 1024 x 1024 pixels achieves the best results for most metrics, indicating that more details and features of the document layout are captured and exploited by the encoders. However, this also increases the memory and computational demands of the model. In this work, we consider image resolution of 256 x 256 for experiments (except in Table this setting provides reasonable trade-off between generalization performance and efficiency for resource constrained scenarios (e.g., mobile applications).
4.3.6 Performance on document object types
We also analyse the performance of AutoDocSegmenteer-U U-MiTT and AutaDoeSegnenter--UMMn U-MMVV on various document objects, such as, paragraphs, figures and tables. The training phase is self-supervised and
15

---

Published in Transactions on Machine Learning Research (01/2025)
Table 10: mloU scores of AutoDocSegmenter-- on paragraphss figures and tables (PRImA dataset).
Model
Paragraph Figure Table
AutoDDocsygmentee---MM111-
86.40
60.67
72.96
AutoDesSegmonte---MMNWW...
85.72
65.90
71.88
Table 11: Comparing different AuloDocSegmenttrr approaches on PRImA and M datasets.
Model
PRImA M6 -Dc
AutoDooSSegmenteer-
82.40
71.50
AutoDDoSSeguennerr-MM11--
89.57
75.22
AutoDDosSeegnnnneer---WWVWWaaW
85.46
75.25
AutoDoocSegrnetteer-.
82.20
72.90
AutolDocSegmmenter----MMMa
89.65
75.56
87.76
75.24
our model predicts only the segmentation mask for each document image without labeling the underlying objects. The PRImA dataset, on the other hand, has labels corresponding to each object in the ground truth annotation. Hence, we perform connected component analysis on the masks generated by our approach and label each component based on the maximum overlap of the component with the ground truth. We report the average mloU scores obtained for each object in Table 10 We observe that the mloU scores for paragraphs are high followed by tables and figures. This is because the mask of figures in PRimA dataset have diverse and complex shapes compared to paragraphs and tables. Morcover, for some complex document images, the ground truth does not cover all the objects while our model is able to detect them (e.g., shown in A.3
4.3.7 Comparing variants of AutoDocSegmenter
To study the offectiveness of Algorithm and the sclf-supervised training stage, we evaluate the performance of the unsupervised algorithms AutoDocSegmenter-I and AutoDoeSegmenter-U and their self-supervised variants with MiT-BO and MNV3sm backbones on PRTmA and M6-Doc. These two datasets closely re- semble real world documents consisting of complex layouts with varying background textures. The self- supervised learning methods are trained PubLayNVet (i.e., we follow the generalized setting 1 discussed in Section 4.2.2 Table 11 reports the performance of all the six methods. We observe that AutoDocSegmenter T achieves comparable or better performance than AutoDocSegmenter-.. Similarly, the sclf-supervised AudoDeSSegnnntee--MMiM..-- and AutoDoeSegmnnnee----MNN3 retain this advantage over their self- supervised AutoDocSegmenter--I counterparts.
5
Conclusions
The scarcity of large-scale datasets with precise polygonal annotations of document objects remains a sig- nificant obstacle developing document segmentation models. Our work introduces AutoDocSegmenterr, a self-supervised approach utilizing isothetic covers as pseudo masks to train an encoder-decoder model. AutoDocSegmenter effectively handles diverse and complex document layouts, producing accurate segmen- lation masks and generalizing well to unseen datasets. Additionally, it supports lightweight encoder ar- chitectures, offering notable parameter and computational efficiency. Overall, develop end-to-end self-supervieed document segmentation method for real-world applications.
Limitations and Future Works: Although AutoDocSegmenter can handle complex document layouts, it only segregates the foreground information from the background and segments various objects present in the document as binary masks. As is a self-supervised approach without any supervisod fino-tuning, our method cannot identify classify the objects into different categories. We plan investigate this unsupervised classification problem as future work.
16

---

Published in Transactions on Machine Learning Research (01/2025)
6 Broader Impact Statement
We present a self-supervised technique of document segmentation which can handle complex layouts effi- ciently and can be integrated with both lightweight and heavyweight encoder architectures based on the available resources. Overall, our work reduces the dependency on human annotations for segmentation task, which is a tedious task. To the best of our knowledge, this work does not pose any negative societal impact.
References
Abdullah Almutairi and Meshal Almashan. Instance segmentation of newspaper elements using mask r-cmn. In IEEE International Conference On Machine Learning And Applicalions, pp. 1371-1375, 2019.
Apostolos Antonacopoulos, David Bridson, Christos Papadopoulos, and Stefan Pletschacher. A realistic dataset for performance evaluation of document layout analysis. In International Conference Document Analysis and Recognition, pp. 296-300, 2009.
Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Manmatha. Docformer: End- to-end transformer for document undorstanding. In IEEE/CVF International Conference on Computer Vision, pp. 993- 18010 2021.
Abedelkadir Asi, Rañ Cohen, Klara Kedem, and Jihad El-Sana. Simplifying the reading of historical manuscripts. In International Conference on Document Analysis and Recognition, pp. 826-830) 2015.
Ayan Banerjee, Sanket Biswas, Josep Lladós, and Umapada Pal. Swindocsegmenter: An end-co-end unified domain adaptive transformer for document instance segmentation. In Document Analysis and Recognition, pp. 307 325, 2023.
Arindam Biswas, Partha Bhowmick, and Bhargab Bhattacharya. Construction of isothetic covers of a digital object: combinatorial approach. Journal of Visual Communication and Image Representation, (4): 21000, 2010.
Sanket Biswas, Pau Riba, Josep Lladós, and Umapada Pal. Beyond document object detection: instance- level segmentation complex layouts. International Journal on Document Analysis and Recognition, 24 (3):269 281, 2021.
Sanket Biswas, Ayan Banerjee, Josep Lladós, and Umapada Pal. Docsegtr: An instance-level end-to-end document image segmentation transformer. orXiv preprinl ar arXiv:2201. 11438, 2022.
Kai Chen, Mathias Seurel, Marcus Liwicki, Jean Hennebert, and Rolf Ingold. Page segmentation of historical document images with convolutional autoencoders. In International Conference on Document Analysis and Recognition, pp. 1011 1015, 2015.
Kai Chen, Mathias Seuret, Jean Hennebert, and Rolf Ingold. Convolutional neural networks for page segmen- tation of historical document images. In International Conference on Document Analysis and Recognition, volume 1, pp. 085-970, 2017.
Ting Chen, Simon Korublith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pp. 1507-1607.
PMLR, 2020.
Hiuyi Cheng, Peirong Zhang, Sihang Wu, Jiaxin Zhang, Qiyuan Zhu, Zecheng Xie, Jing Li, Kai Ding, and Lianwen Jin. M6doc: A large-scale multi-format, multi-type, multi-layout, multi-language, multi- annotation category dataset for modern document layout analysis. In IEEE/CVE Conference on Computer Vision and Pattern Recognation, pp. 15138-151441 June 2023.
Christian Clausner, Apostolos Antonacopoulos, Tom Derrick, and Stefan Pletschacher. Icdar2019 competi- tion on recognition of early indian printed documents-reid2019, In International Conference on Document Analysis and Recognilion, pp. 1527-1532. 2019.
17

---

Published in Transactions on Machine Learning Research (01/2025)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Rep- resentations, 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33:21271 21284, 2020.
Jiuxiang Gu, Jason Kuen, Vlad Morariu, Handong Zhao, Rajiv Jain, Nikolaos Barmpalios, Ani Nenkova, and Tong Sun. Unidoc: Unified pretraining framework for document understanding. Advances in Neural Information Processing Systems, 34:39-50, 50, 2021.
Zhangxuan Gu, Changhua Meng, Ke Wang, Jun Lan, Weiqiang Wang, Ming Gu, and Liqing Zhang. Xylayouthm: Towards layout-aware multimodal networks for visually-rich document understanding. In IEEE/CVE /CF Conference on Computer Vision and Pattern Recognition, pp. 4583-4592. 2022.
Adam W Harley, Alex Ufkes, and Konstantinos G Derpanis. Evaluation of deep convolutional nets for doc- ument image classification and retrieval. In International Conference on Document Analysis and Recog- nition, pp. 991 995, 2015.
Dafang He, Scott Cohen, Brian Price, Daniel Kifer, and C Lee Giles. Multi-scale multi-task fen for se- mantic page segmentation and table detection. In International Conference on Document Analysis and Recognition, volume 1, pp. 254-261, 2017a.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In IEFE international confer- ence on computer vision, pp. 2961-2969, 2017b.
Andrew Howard, Mark Sandler, Grace Chu, Liang- Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In IEEE/CVP Interna- tional Conference on Computer Vision, pp. 1314-1324 2019.
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layouthmv3: Pre-training for document ai with unified text and image masking. In ACM International Conference on Multimedia, pp. 4083-4091, 2022.
Nicholas Journet, Véronique Eglin, Jean- Ramel, and Rémy Mullot. Texl/graphic labelling of ancient printed documents. In International Conference on Document Analysis and Recognition, pp. 1010-1014 2005.
Goewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyoong Yim, Wonscok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding trans- former. In European Conference Computer Vision, pp.
2022.
Koichi Kise, Akinori Sato, and Motoi Iwata. Segmentation of page images using the area voronoi diagram. Computer Vision and Image Understanding, 70(3):370-38) 382, 1998.
Joonho Loe, Hidcaki Hayashi, Wataru Ohyama, and Sciichi Uchida. Page segmentation using a convolutional neural network with trainable co-occurrence features. In International Conference on Document Analysis and Recognition, pp. 1023-1028, 2019.
Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, and Furu Wei. Dit: Self-supervised pre-training for document image transformer. In ACM International Conference on Multimedia, pp. 3530 3539, 2022.
Kai Li, Curtis Wigington, Chris Tensmeyer, Handong Zhao, Nikolaos Barmpalios, Vlad I Morariu, Varun Manjunatha, Tong Sun, and Yun Fu. Cross-domain document object detection: Benchmark suite and method. In IEFF/CVF Conference on Compuder Vision and Pattern Recognition, pp. 12915- -29244, 2020a.
18

---

Published in Transactions on Machine Learning Research (01/2025)
Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu. Selfdoc: Self-supervised document representation learning. In IEEE/CVE Conference on 2021a.
Computer Vision and Pattern Recognition, pp.
Xiao-Hui Li, Fei Yin, and Cheng-L.in Liu. Page segmentation using convolutional neural network and graphical model. In Document Analysis Systems: IAPR International Workshop, pp. 231 245, 2020b.
Yulin Li, Yuxi Qian, Yuechen Yu, Xiameng Qin, Chengquan Zhang, Yan Liu, Kun Yao, Junyu Han, Jingtuo Liu, and Errui Ding. Structext: Structured text understanding with multi-modal transformers. In ACM International Conference on Multimedia, pp. 1912 1912-1920, 2021b.
Guo-Shiang Lin, Jia-Cheng Tu, and Jen-Yung Lin. Keyword detection based on retinanet and transfer learning for personal information protection in document images. Applied Sciences, 11(20):9528, 2021.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming Hc, and Piotr Dollár. Focal loss for dense object detection. In IEEE/CVF International Conference on Computer Vision, pp. 2980 2988, 2017.
Subhajit Maity, Sanket Biswas, Siladittya Manna, Ayan Banerjec, Josep Lladós, Saumik Bhattacharya, and Umapada Pal. Selfdocseg: A self-supervised vision-based approach towards document segmentation. In International Conference on Document Analysis and Recognition, 2023.
Anoop M Namboodiri and Anil K Jain. Document structure and layout analysis. In Digital Document Processing: Major Directions and Recent Advonces, pp. 29 48. 2007.
Sofia Ares Olivcira, Benoit Seguin, and Frederic Kaplan. dhsegment: A generic doep-learning approach for document segmentation. In International Conference on Frontiers in Handwriting Recognition, pp. 7-12, 2018.
Nobuyuki Otsu. A threshold selection method from gray-level histograms, IEEE Transactions on Systems, Man, and Cybernetics, 9(1):62-66, 1979.
Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S Nassar, and Peter Staar. Doclaynet: A large human-annotated dataset for document-layout segmentation. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3743 1143-3751, 2022.
Devashish Prasad, Ayan Gadpal, Kshitij Kapadui, Manish Visave, and Kavita Sultanpure. Cascadetabnet: An approach for end to end table detection and structure recognition from image-based documents. In IEEE/CVF Conference on Computer Vision and Pottern Recognition workshops, pp. 572 573, 2020.
Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 28, 2015.
Raid Saabni and Jihad El-Sana. Languag--nddepeneent text lines extraction using seam carving. In Inter- national Conference on Document Analysis and Recognition, pp. 563 563-568, 2011.
Ranajit Saha, Ajoy Mondal, and CV Jawahar. Graphical object detection in document images, In Interna- tional Conference on Document Analysis and Recognition, pp. 51-58, 2019.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chieh Chen. Mobilenetv2: Inverted residuals and lincar bottlenecks. In IEEE Conference on Computer Vision and Pattern Recogni- lion, pp. 4510 4520, 2018.
Sebastian Schreiber, Stefan Agne, Ivo Wolf, Andreas Dengel, and Sheraz Ahmed. Deepdesrt: Deep learning for detection and structure recognition of tables in document images. Tn International Conference on Document Analysis and Recognition, volume 1, pp. 1162-1167, 2017.
Zejiang Shen, Ruochen Zhang, Melissa Dell, Benjamin Charles Germain Lee, Jacob Carlson, and Weining Li. Layoutparser: A unified toolkit for deep learning based document image analysis. In International Conference Document Analysis and Recognition, pp- 131-146, 2021.
19

---

Published in Transactions on Machine Learning Research (01/2025)
Ningning Sun, Yuanping Zhu, and Xiaoming Hu. Faster r-cmn based table detection combining corner locating. In International Conference on Document Analysis and Recognalion, pp. 1314-1319, 2019.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105 6114. PMLR, 2019.
Tuan Anh Tran, In-Seop Na, and Soo-Hyung Kim. Hybrid page segmentation using multilevel homogeneity structure. In International Conference Ubiquitous Information. Management and Communication, pp- 1 6, 2015.
Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan. Mobileone: An improved one millisccond mobile backbone. In IEEE/CVF Conference on Computer Vision and Pattern 2023.
Recognilion, pp.
Nayer M Wanas, Dina A Said, Nadia H. Hegazy, and Nevin M Darwish. A study of local and global thresholding techniques in text categorization. In Australasian Data Mining Conference, 2006.
Enze Xic, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Sogformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34:12077-120 12090, 2021.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In TEEE Conference Computer Vision and Pattern Recognition, pp. 1492- 1500, 2017.
Jing Xu, Yu Pan, Xinglin Pan, Steven Hoi, Zhang Yi, and Zenglin Xu. Regnet: self-regulated network for image classification. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pre-training of text and layout for document image understanding. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1192- 1200, 2020.
Feixiang Yan, Hong Zhang, and C Ronald Kube, A multistage adaptive thresholding method. Paltern recognition letters, 26(8):1183 1191, 2005.
Huichen Yang and William Hsu. Transformer-based approach for document layout understanding. In 2022 IEFE International Conference on Image Processing, pp. 4043 4443-4047, 2022.
Huichen Yang and William H Hsu. Vision-based layout detection from scientific literature using recurrent convolutional neural networks. In International Conference on Pattern Recognition, pp. 6455-6456 0366,46222 2021.
Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Haibin Lin, Zhi Zhang, Yue Sun, Tong He, Jonas Mueller, R Manmatha, et al. Resnest: Split-attention networks. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2736 2746, 2022.
Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Publaynet: largest dataset ever for document layout analysis. In International Conference on Document Analysis and Recognition, pp. 1015-1022, 2019.
A Appendix
A.1
Data Description
We evaluate our method on four popular document segmentation datascts: PRImA (Antonacopoulos et al.
2009,, DocLayNet (ffitzmann et al. 2022). PubLayNet (Zhong et al. 2019), and Mo-Doc (Cheng et al. 2023
PRImA datasct (Antonacopoulos et al. 2009) has 382 training and 96 testing samples, respectively with
the image dimensions varying between 2500 to 3500. The annotations of this dataset provide polygonal boundaries for each entity present in the document.
20

---

Published in Transactions on Machine Learning Research (01/2025)
However, for DocLayNet Pfitzmaan al. 2022 and PubLagNet (Zhong et al 2019), the bounding boxes are rectangular in shape. DocLayNel has 69375 training and 6489 testing images, respectively. All the images are of fixed dimensions of 1025 x 1025. On the other hand PubLayNet, is a large-scale dataset with image dimensions ranging from
750 to 950 and contain 335703 training and 11245 testing samples, respectively.
-o- (Cheng et al. 2023) is a recently released dataset which introduces diverse set documents including scanned, photographed and PDFs of scientific articles, books, magazines, newspapers and notes in English and Chinese languages. It has been used for testing the effectiveness of our generalised model. The test set comprises of 2724 images of dimensions roughly varying from 2034 2877 to 2034 2916.
A.2 Isothetic Covers Algorithm
The isothetic covers follow row-major traversal of the document to outline the regions where objects are present based on convex hull algorithm. The convexity of the point determines the type and angle associated to define the direction of movement. detailed algorithm is presented as follows:
Algorithm 2 Path Traversal of Isothetic covers
91 Top-left grid occupied
92 Top-right grid occupied
93 Bottom-left grid occupied
= Bottom-right grid occupied
Type of verlex, la = Angle of vertex
if (91 & ((92&1931911) ||
(g2 (1988)9)1)))) (gs & ((91&192(9))) || (ga ((918192898)) then
+-1 tu + 90
else if (g18 & ((98894)) || (gslegs & ((g))))) then
t
ta + 180
else if (g1& & ((92&195)) || (go8tgs & !(91&191) then - -1
la + 270
else if (918 & 1gx) &gggtg & !gs) (92&gg3& & !91) &1g8g9 & (92) then t -1
ta + 270
else if (g) &1884))88)4))) (99 then
continue end if
The type of grid and angle associated with it depends on the occupancy of the four adjacent grids. When either one or three the grids are occupied with text, image, etc., we denote the type, = 1 and tg = 90". Therefore, the direction of movement takes a 90 turn and traverses along the boundary of the occupied grid. Similarly, the direction of movement remains the same when two adjacent grids are occupied. There is no effect the traversal when all or none of the grids are emply. We show more examples of AutoDocSegmmetter- C from DocLagNet dataset in Fig.
A.3 Comparing AutoDocSegmenter-U with ground truth masks
We show the difference between groundtruth masks and the masks generated using AutoDooSSegmener-- in Fig. for PubLayNet datasct. PubLayNet consists of clean images with simple layout. Hence, it is casier to compare the masks generated using human annotation with the isothetic covers.
We observe that for documents with text-only format or sequential paragraphs, the AutoDocSegmenter- masks are identical that of the given annotations. However, for tables, instead of rectangular box denoting the entire table, AutoDoeSegmenter-u precisely outline each row and column using polygonal boundaries. Similar results observed for images well.
21

---

Published in Transactions on Machine Learning Research (01/2025)
conditiona
Figure 8: Examples of AutoDDocSegmenter U from DocLayNet dataset.
Results from groundtruth masks
Results from AutoDooSSementer-
Figure 9: Comparing the ground truth annotations with our AutoDocSegmentor- masks for PubLayNet dataset.
As our proposod lcarning method trained using AutoDocSegmenter-U masks, the model learns to mimic the characteristies of polygonal covers and generates similar predicted masks for various objects present. in the document. Most complex layout images annotated using rectangular boxes left non-annotated in some cases. As AutoDoocSemmentrr- U is independent of the given groundtruth, it is able to detect complex shaped objects present in the documents. Fig. 10 shows an example of our method applied to a magazine image, where we can see that our network can segment the figure with reasonable precision, even though the training dataset does not contain any tight annotations for the document objects.
22

---

Published in Transactions on Machine Learning Research (01/2025)
Global Adviser SEMP 65 Ehol sesa
Global Adviser
Moontain Magic. An
Mountain Mapccin
religion cornes to life
coneess to
(a)
(b)
Figure 10: Result of typical magazine image from PRImA dataset using our learning technique. (a) The real annotation provided for the document. (b) The predicted mask generated using our proposed lcarning Our proposed method have been able to outline the complex
technique with AutoDeeSennenter-MMWWS boundary of the image while it is kept non con-annotated in the given dataset.
23


Reviews:
[{"review_id": "Cxt1irI8Bf", "reviewer": "Reviewer_v7LM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The proposed method is the first full self-supervised approach for the task, exploiting a traditional technique as supervision instead of human annotations.\n- The method achieves state-of-the-art results in document segmentation when compared with many baseline methods in both within-domain and cross-domain settings.\n- The method is independent of architectural choices and can be used on any encoder backbone.\n- Not only is data efficient, but AudoDocSegmenter is also compute efficient both at train and at test time due to its use of small architectures\n\nWeaknesses:\n\n- Since baselines differs in backbone and training recipes, it could be beneficial to report number of parameters for each method and, if possible, the train size for the pre-training and fine-tuning. Aside from giving more context for comparison, it would also strengthen the claim that the proposed method is efficient.\n- Missing ablation regarding different merging strategies for local and global thresholding, e.g., use different overlap thresholds, using weighted average, etc.\n- Missing ablation on binarization techniques, i.e., replace Otsu's method with other approaches\n- Analysis on the model's performance on specific document object types, i.e., performance on tables, figures, paragraphs, to demonstrate where the approach improves more.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents AutoDocSegmenter, a self-supervised framework that effectively segments documents into meaningful regions, without relying on labeled data. It introduces an unsupervised pseudo-mask generation procedure to create isothetic covers for self-supervised learning. Compared to prior work, it does not depend on a two-stage pipeline where the second requires annotated data, but solely relies on self-supervision to work in within-domain and cross-domain settings. In both evaluation scenarios, the proposed method achieves impressive results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Add number of parameters, pre-train and fine-tune samples for methods in the main tables.\n- Ablate merging techniques and binarization strategy.\n- Analyse the performance of the model on specific object types.\n- (minor) There are a couple of typos in the text. A common one regards the use of the \"pseudo\" term, which is used inconsistently, e.g., \"pseudo-masks\" and \"pseudo masks\", etc.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vF44wJwMvm", "reviewer": "Reviewer_1HGU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The proposed method can be used in various document-related applications, in diverse real-world scenarios.\n- The empirical results are quite good on several benchmarks, and the authors provide several ablation studies to analyze the characteristics of the proposed method.\n\nWeaknesses\n- The main contributions of this work are two folds: an automatic pseudo mask generation algorithm, and an encoder-decoder document segmentation network trained using the generated pseudo masks. However, the quality of pseudo mask itself is never directly evaluated in the paper, except some examples in Figure 8. All the experiments evaluate the segmentation model. Please provide more evaluation results demonstrating the high quality of pseudo masks. For example, it would be great to compute segmentation metrics by directly comparing the pseudo masks and ground-truth masks.\n- The authors stated in Section 1 that SelfDocSeg is sensitive to hyperparameters such as the kernel size and shape for erosion. However, according to Table 5, the proposed method is also quite sensitive to the grid size, which reduces the robustness of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Labeling documents is quite a costly process, so most large-scale datasets provide simplified annotations, which compromises the performance of models trained on them. This paper proposes an automatic algorithm to generate pseudo polygonal masks capturing the complex shapes and layouts of diverse document layouts and objects. The algorithm improves on a previous grid-based approach by merging isothetic masks generated from two types of binarized images, each derived from global and local thresholding, respectively. Additionaly, the authors train an encoder-decoder segmentation network using the generated pseudo polygonal masks. The resulting model shows competitive performance on several document segmentation benchmarks and generalizes well to unseen document layouts. The framework is also adaptable, supporting a range of encoder architectures, including both convnet and transformer based models.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the above weaknesses and the following minor points:\n- In Algorithm 1, what does S_g == S_l exactly mean? and Pg || Pl means applying the logical or operation?\n- In Section 3.2, the value \"0\" for the dice loss function indicates complete similarity, not \"1\". Also, how can the loss in Equation (1) become 1?\n- In Figure 6, the captions for (a) and (b) should be swapped.\n- The authors said they fixed the grid size to be approximately 1/100-th of the minimum dimension of the images for a particular dataset in 4.2.1 while they also said they performed hyper parameter tuning for finding the optimal grid size. Which is correct?", "broader_impact_concerns": "I don't believe there are ethical concerns about this paper. Please discuss the limitation of the proposed method and add the Broader Impact Statement section in the final draft.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xsZfOKfU6Z", "reviewer": "Reviewer_pYgA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths of the paper include:\n- Clear explanation of the proposed algorithm for unsupervised generation of pseudo-masks;\n- Good experimental section with approach tested on sufficient number of datasets and in both in-domain and out-of-domain settings. Many previous approaches are used to compare the proposed method to;\n- Good experimental results both in terms of metrics and visual quality;\n- Decent ablation with many backbones and hyperparameters tested. Also it is good to see comparisons of the masks generated by the algorithm to the ground-truth ones with the discussion of fail modes.\n\nElements that require attention:\n\nIt is not clear what the difference between AutoDocSegmenter-I and  AutoDocSegmenter-U is. One can assume from the text that AutoDocSegmenter-I is a version of approach which generates masks based only on the proposed approach of constructing and merging isothetic covers, i.e. without training a neural network-based segment, while AutoDocSegmenter-U is a version of approach which uses a neural network segmentor trained on pseudo-masks generated by AutoDocSegmenter-I. If that is true, it is unclear why there are still two versions of AutoDocSegmenter-I approach in Tables 2 and 3, i.e. AutoDocSegmenter-I-MiT-B0 and AutoDocSegmenter-I-MNV3sm, as  MiT-B0 and MNV3sm refer to different backbones for the neural network segmentor. If AutoDocSegmenter-I is not that, then it is unclear what it is (and what the difference between AutoDocSegmenter-I and AutoDocSegmenter-U is), and another question arises: do we actually need to train any segmentation model on top of pseudo-masks generated by proposed isothetic covers algorithm? Will that be any better than just using proposed isothetic covers algorithm for any new document? What is a motivation behind training a network on top of generated pseudo-masks?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel self-supervised approach to the task of document segmentation. There are two main novel contributions:\n1. A proposed algorithm for unsupervised generation of pseudo-masks for further training of segmentation model. The algorithm is based on well-known approach for construction of isothetic covers. The novelty lies in that the authors propose to build two sets of isothetic covers for each documents using different binarisation strategies, and then merge two obtained covers;\n2. An idea of training an end-to-end document segmentation model based on unsupervisely collected pseudo-masks using algorithm from (1).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Clearer explanation of what algorithms AutoDocSegmenter-I and AutoDocSegmenter-U are and the difference between them. Add motivation and justification for training a segmenting neural network on top of pseudo-masks generated by isothetic covers algorithm.", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers really appreciate this paper.

One reviewer proposes to comment on an ablation discussed with the authors. This is encouraged. 

Apart from the paper the paper is ready from publication.
Decision: Accept (poster)