RDLNet: A Novel and Accurate Real-world Document Localization Method

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 08:23:14

Abstract: The increasing use of smartphones for capturing documents in various real-world conditions has underscored the need for robust document localization technologies. Current challenges in this domain include handling diverse document types, complex backgrounds, and varying photographic conditions such as low contrast and occlusion. However, there currently are no publicly available datasets containing these complex scenarios and few methods demonstrate their capabilities on these complex scenes. To address these issues, we create a new comprehensive real-world document localization benchmark dataset which contains the complex scenarios mentioned above and propose a novel Real-world Document Localization Network (RDLNet) for locating targeted documents in the wild. The RDLNet consists of an innovative light-SAM encoder and a masked attention decoder. Utilizing light-SAM encoder, the RDLNet transfers the mighty generalization capability of SAM to the document localization task. In the decoding stage, the RDLNet exploits the masked attention and object query method to efficiently output the triple-branch predictions consisting of corner point coordinates, instance-level segmentation area and categories of different documents without extra post-processing. We compare the performance of RDLNet with other state-of-the-art approaches for real-world document localization on multiple benchmarks, the results of which reveal that the RDLNet remarkably outperforms contemporary methods, demonstrating its superiority in terms of both accuracy and practicability.

Main paper: 
 introduction: 
 In recent decades, there has been an increasing number of users starting to use mobile devices to take photos or videos of the paper documents. Document digitization has become a trend in our society [40] . Once documents are digitized, they will be easier to store, retrieve, and read. To achieve this, the document must first be accurately located in the image. Therefore, document localization plays a crucial role in the document digitization process. However, localizing documents in the natural scene images still faces many challenges in practical applications. Firstly, images can be taken from different distance and viewpoints, the documents in the image will have various sizes, positions, and perspective distortions. Additionally, in some cases, images are usually taken in a casual way. For instance, some captured images are often blurry, possessing different lighting conditions, low contrast, some occlusion in the document area and even similar overlapping background documents. Therefore, document localization in real world images remains a challenging problem.Although many advanced methods have achieved good results in the field of document localization, experimental verifications of most methods are based on existing opensource document detection datasets, such as SmartDoc dataset [4] and CDPhotoDataset [11] . But these datasets usually contains limited document types. Moreover, the pictures in these datasets are taken in the scenes which are relatively simple and do not contain comprehensive real-world situations. These complex situations are common in practical deployments and inevitable. However, to the best of our latest knowledge, there is not a comprehensive dataset that fully reflects complex scenarios of document images in the wild. Therefore, many methods have reached near-saturation localization accuracy on current datasets, but this does not fully prove that the challenges existing in the real-world have been well solved. That is why a more extensive and comprehensive real-world document dataset is needed to provide a more valuable and research-space benchmark for future research.As mentioned above, many previous advanced studies have achieved excellent results in the field of document detection, but they all focused on a single semantic-level document type and simple photography scenarios. And these methods do not fully demonstrate their generalization performance. In recent years, the emergence of large models has revolutionized the applications in many fields. In the application of image segmentation, the emergence of SAM [18] unifies the tasks of instance segmentation and semantic segmentation and shows superior performance. However, SAM is a segmentation model for general image scenes and can be used for document edge detection rely on its prompt design. Moreover, due to the large scale of the SAM model, it is not suitable for running on mobile devices such as personal computers or smartphones.To solve the above problems, we propose a novel Real-world Document Localization Network (RDLNet) and create a new complex and comprehensive benchmark dataset which is Real world Mobile Document dataset (RWMD) captured by mobile phones in the wild. First, in order to better transfer the powerful generalization ability of SAM in segmentation tasks to document localization tasks, we design a lightweight SAM encoder module. Second, after the document image is encoded by the light-SAM, we propose an advanced decoder module based on masked attention and object query. Third, we design three output branches based on the features from both the encoder and decoder modules. The three branches contain the classification branch, mask branch and corner points branch. Different from the previous methods of corner prediction based directly on decoded semantic features, we design an innovative branch based on the hybrid information of semantic features, mask features and object query vectors. Hence, the accurate document localization results can be directly obtained by the corner prediction branch. Document localization samples of different methods are presented in Figure 1 . Specially, our method can simultaneously output corner point predictions of foreground document and region prediction results that distinguish among different documents in one image. Finally, through the above proposed modules, our model's able to generate accurate document localization results with high efficiency, especially on the new comprehensive benchmark dataset we contributed with more distinct effect. In summary, the main contributions of this paper includes:(1). Light-SAM encoder: We introduce the light-SAM encoding module into the field of document localization, and exploit distillation method to constrain and reduce the parameters of the designed encoding module to improve model performance.(2). Masked attention and object query based decoder: We design an advanced decoding module based upon masked attention and object query to allow the model to form the global and local relations between document objects and image context to directly and separately output the predictions of different documents in one image or frame.(3). Triple-branch: We propose a novel triple-branch that integrates hybrid features to predict corner points, area masks and categories at the same time. By jointly optimizing the losses of the three branches, we can get more accurate prediction results. In addition, the number of corner points of the document can be modified as a hyper-parameter in our method.(4). New benchmark dataset: We create a new benchmark dataset which is called Real world Mobile Document(RWMD) dataset for document localization. This dataset provides rich document types and diverse shooting scenarios in the wild. RWMD fills the current lack of comprehensive challenging document localization datasets. And the RWMD dataset will be soon publicly available.(5). SOTA results: In the experiments, we compare the performance of our method and other current state-of-art methods in document localization on multiple datasets contain our new dataset RWMD. The results proved that the accuracy of our proposed method exceeded other methods with low inference time. 
related work 2.1 document localization datasets: 
 There are several public datasets used to measure the capabilities of document localization methods: SmartDoc dataset [4] , Extended SmartDoc dataset [10] , CDPhotoDataset [11] , MIDV dataset family (MIDV-500 dataset [1] , MIDV-2019 dataset [2] , MIDV-2020 [3] ). SmartDoc dataset was created for ICDAR 2015 SmartDoc competition. Each of these documents are printed using a color laserjet and captured by a Google Nexus 7 tablet [4] . The database consists of 150 video clips comprising 25,000 frames. Many research works have been carried out based on this dataset [4, 8, 15, 34] . However, SmartDoc dataset only contains smooth A4 paper format, and most backgrounds are in close similarity and are lack of diversity. Ricardo, et al. [10] created an extension of the original Smartdoc Dataset known as the Extended SmartDoc dataset. The authors inserted extra backgrounds into the original images with backgrounds acquired from various smartphone cameras with richer scenes than the original datasets. This dataset has a total of 29,522 images with a fixed size of 1728 × 2304 pixels. However, the data synthesis method of this dataset doesn't sufficiently increase the richness of the original dataset. The CDPhotoDataset [11] is composed of images of Brazilian identification documents, in different backgrounds of realworld environments, with non-uniform standards. It has a total of 20,000 images of documents with the respective document mask as ground truth. Since the document images in this dataset are mannualy merged into background images, this dataset cannot fully reflect the real-world document photo scene. The first dataset of the MIDV family is MIDV-500 [1] which contains 500 video clips of 50 identity documents, 10 clips per document type. The conditions represented in MIDV-500 thus have some diversity regarding the background and the positioning of the document, however, they don't include variation in lighting conditions, or significant projective distortions. To address the issues the dataset MIDV-2019 [2] was later published as an extension of MIDV-500, which featured video clips captured with very low lighting conditions and with higher projective distortions. The main disadvantage of the datasets of the MIDV-2019 is the scarcity of different document samples. All images in this dataset are made using the same 50 physical document samples. In order to make up for this shortcoming, MIDV2020 [3] photo dataset consists of 1000 different physical documents, all with unique artificially generated faces, signatures, and text fields data. Although this dataset contains some complex scenes, it only contains hard cards and therefore does not include some shape changes and flexible transformations.Research has been conducted on document localization since the start of camera-based document processing development [17] . The earliest methods were primarily designed to solve scanned documents. These documents are required to be placed flatly in a prescribed area, so the background is simple. Then the traditional segmentation methods [20, 27, 30] can locate the document easily. In the recent literature addressing document localization problems, these methods can be broadly divided into four categories: document boundary-detection-based methods, segmentation methods, keypoints detection methods, document content-based methods.Document boundary-detection-based methods: These methods continue the idea of traditional image processing methods and locates documents by detecting the edges of the document. Document edge detection methods usually include the following methods: edge extraction methods (such as Canny detector, Sobel operator, or Holistically-Nested Edge Detector [36] ), line detectors (like Line Segment detectors [13] , Hough transform). This type of methods is robust to corner occlusion, but requires a clean background and clear document edges. Document detection accuracy heavily depends on environmental conditions and shooting situations. Once the conditions change, such as low contrast and light intensity, the accuracy of document localization results will be significantly affected.Segmentation methods: These methods follow the flow of object segmentation methods [14, 21, 28, 32, 35, 39] , which treat the document as an object. This type of methods [7, 8, 37] uses dense prediction to output a classification result for each pixel to distinguish whether the pixel belongs to the document or the background. Then by grouping the pixels with the same label, the document is segmented. This type of method is robust to irregular documents. But these methods require further post-processing to get the orientation of the document. Moreover, these methods usually have large model sizes and require more inference time, making it difficult to meet the efficiency requirements for processing on mobile devices.Keypoints detection methods: This type of methods [15, 26, 40] usually assumes that the document has four corner points, and then converts the document localization problem into a corner positioning problem. It can be solved using corner detectors and key point detectors. If the used dataset has corner annotations, this type of methods can directly obtain the orientation of the document based on the predicted corner points coordinates. However, these methods will be ineffective for documents whose corners are occluded or have irregular shapes.Document content-based methods: This type of methods [23, 24, 31] transforms the document localization problem into the positioning problem of the text content area. These methods use the layout and pattern characteristics within the document to locate document areas, such as text line information, image or table edge information, and the layout of different text blocks. The advantages of this type of methods are that it can judge the orientation of the document based on the information of the text content. And it is conducive to extracting useful content information for subsequent recognition steps. However, these methods usually assume the background to be simple and flat in contrast to the document. Therefore, these methods often fail when the background is complex or confusing.There are also some literature that use hybrid methods, such as [29] which uses both corner prediction and edge prediction to locate documents. And [34] combines corner points prediction, line boarders prediction and document classification in a single lightweight network. Our method belongs to this kind of category combining the corner points prediction, instance-level segmentation, and classification.SAM [18] is a powerful model with strong generalization ability for segmentation tasks based upon user prompts. It's pre-trained on a broad dataset SA-1B consisting of 11 million images and 1 billion masks. Given a prompt like a single ambiguous point prompt on the document in an image, it's capable of generating the document's area mask, and with post-processing, corner points of that document can be obtained. Nonetheless, it's computationally costly since it contains at least 91 million parameters which is too large for mobile devices. 
method: 
 The pipeline of the proposed RDLNet is depicted in Figure 2 . In this section we elucidate distillation process we applied for the backbone, following the encoder-decoder structure we adopted and the loss functions for RDLNet's training supervision.SAM [18] is a powerful model for generic and referring segmentation. It is based on the Vision Transformer(ViT) [12] . The ViT is a stack of Transformer blocks, which is used to extract features from the RGB input image 𝐼 ∈ R 𝐻 ×𝑊 ×3 of size 𝐻 × 𝑊 with three channels.Concretely, the input image 𝐼 is divided and flattened into a sequence of non-overlapping patches 𝐼 ′ ∈ R 𝑁 ×𝐶 , where 𝐶 = 𝑃 2 × 3 and 𝑁 = 𝐻𝑊 𝑃 2 , assuming the patch size is 𝑃 × 𝑃. They are then linearly projected to obtain a sequence of 𝐶 ′ -dimension feature vectors, which are fed into the Transformer encoder with each block (𝑙) consisting of multi-head self-attention (MSA) and feedfoward network(FFN) conducting operations below:EQUATIONwhere MSA is used to capture the global dependencies among the patches, while the FFN is used to model the local dependencies within each patch.For enabling knowledge transfer from computationally expensive and large models to smaller ones, we propose a multiplex distillation method for acquiring a light-SAM image backbone. We first obtain a large SAM model pre-trained on a large-scale dataset, e.g., SA-1B [18] , then we distill the knowledge from the large model to a smaller one. The multiplex distillation process is conducted by diminishing the multiplex relation map gap L 𝑚𝑑 among teacher and student models' intermediate layers' features denoted as 𝐹 𝑇 ∈ R 𝑁 ×𝐷 𝑇 and 𝐹 𝑆 ∈ R 𝑁 ×𝐷 𝑆 respectively and minimizing the KL divergence L 𝑘𝑙 between the output of the teacher model 𝑝 𝑠 (x) and the output of student model 𝑝 𝑡 (x). The distillation loss is defined as follows:EQUATIONwhereF 𝐾𝐿 denote the KL divergence loss, with 𝜏 as the temperature parameter. Hence, the overall loss function for distillation is: EQUATIONTo fully harness information from the backbone, we design a crosslevel encoder that utilizes the low-level to high-level features from backbone intermediate layers. They are projected into a new feature space and concatenated to form the feature map 𝐼 ′ projEQUATIONwhere Conv denotes the convolutional operation, 𝐼 ′ 1 , 𝐼 ′ 𝑙/2 and 𝐼 ′ 𝑙 are the feature maps of the first, middle and last layer of the backbone output respectively. For spatial dependencies capturing, we take advantage of deformable attention [41] to guide the encoding process with important regions from the image and bridge visual tokens' relations more effectively.Enriched with positional information via positional embedding𝐼 ′ proj ← 𝐼 ′ proj + 𝑃 𝑒 , 𝑃 𝑒 ∈ R 𝐻 16 × 𝑊16 ×𝐶 , the input for each encoder layer conduct:EQUATIONwhere DMSA denotes the deformable multi-head self-attention. To prepare input mask for decoder attention mechanism and filter out background objects, we also project the last layer output from the backbone with convolutional operation and obtain prior mask 16 with mask embedding 𝜓 and object queries 𝑄 ∈ R 𝑁 𝑜 ×𝐶 ′ : 𝑀 𝑝 ∈ R 𝑁 𝑜 × 𝐻𝑊EQUATIONRecent studies [9, 33] suggest global context features eventuate in the slow convergence of Transformer-based models, as it takes numerous training epochs for attention mechanism to learn and attend to local foreground regions. Thus, we propose a masked decoder and mask some of the background areas based upon prior mask prediction with Eq.6 to filter global context features for attention operations as shown in Figure 3 . With masked cross attention, we compute the result matrix via:EQUATIONHere, 𝑀′(𝑙 -1) 𝑝is the previous reshaped and repeated prior mask 𝑀 𝑝 for decoder layer 𝑙, each decoder layer exploits mask prediction from the last layer while the first decoder layer utilizes mask acquired from Eq.6 to initialize 𝑀 ′(0) 𝑝 . Q (𝑙 ) , K (𝑙 ) and V (𝑙 ) are the query, key and value matrices respectively. x (𝑙 -1) is the output of the previous decoder layer. The object queries after decoding are then fed into the triple embedding branches of point, mask and class for prediction. In particular, class embedding branch endues our model with the ability to distinguish among different types of documents on the instance level, e.g., top document and overlapped document or document with different categories such as paper and magazine etc. Contrasting to conventional methods like [8, 14, 16 ] that demand heavy post-processing, e.g., polygon approximation, to attain final document border or corner points predictions, RDLNet directly output polygon corner points with its point embedding. In the case 16 where 𝑁 ℎ is the number of heads for multi-head masked attention operation. Note that we switch the order of self-attention and cross-attention so that local areas can be attended solely in both attention operations for faster training convergence rate and performance. of documents with irregular shapes or complex document targets, we also make use of the mask embedding branch as an auxiliary approach for a precise approximation on their edges.𝑀 ′ 𝑝 ∈ R 𝑁 ℎ ×𝑁 𝑜 × 𝐻𝑊To supervise the training of RDLNet, 𝑁 𝑜 decoded object queries are used to generate a set of 𝑁 𝑜 predictions { ŷ𝑖 } 𝑁 𝑜 𝑖=1 via point, mask and class embedding consisting of point coordinates q, raw mask logits m and class probabilities p𝑖 (𝑐). Following [6] , the optimal permutation { ŷ𝜎𝑖 } 𝑁 𝑖=1 is determined by bipartite matching between the ground truth and the predictions.In lieu of using focal loss [22] , we use the binary cross-entropy loss:EQUATIONand dice loss [25] :EQUATIONfor our mask loss. Besides, we propose a distance loss for point branch prediction supervision,EQUATIONwhile foreground document class predictions are supervised via standard cross entropy:EQUATION)Ultimately, the total loss is composed of the weighted sum of the above losses:EQUATION)with 𝜆 𝑖 as adjustable hyper-parameters.We train RDLNet with AdamW optimizer with weight decay set to 10 -4 under the initial learning rate of 10 -4 with a learning rate drop by a factor of 10 -1 every 40000 iterations. Experiments are carried out on 1 NVIDIA A800 GPUs with the mini-batch size of 8 for 160000 training iterations in total. Loss weights are empirically set to 𝜆 1 = 2, 𝜆 2 = 2, 𝜆 3 = 5 and 𝜆 4 = 5. The input images are resized to 1024 pixels for both width and height. To the best of our knowledge, there is no existing publicly available dataset comprising of documents captured by smartphones in real conditions with various types of document and complex scenarios. We create such a comprehensive benchmark dataset RWMD for document localization. The dataset will soon be made public, and we hope that it will serve the scientific community for more document localization methods to boom. The images in this dataset are captured by eight different mobile phones including iPhone13, Samsung Galaxy S21 Ultra, HUAWEI nova 7 SE, HUAWEI P30, HUAWEI P50 Pro, VIVO S17e, VIVO X21A, and Xiaomi 8. The RWMD dataset contains a total of 2009 images which are divided into nine categories. Moreover, each category contains documents in different styles with different real background, distortion, rotation, perspective transformation, contrast, light intensity, occlusion and other real situations. The samples of different types of documents with various scenarios are shown in Figure 4 . These document pictures taken in the real world contain a variety of real scenes, which pose great challenges to the document localization algorithm. In order to meet the needs of different methods, the RWMD annotation documents contain a variety of information, including masks of different document areas, class labels of different document instances corresponding to the masks, and corner points coordinates of the document. If the image contains multiple different documents, in order to be closer to the actual usage scenarios of the document localization algorithms on the devices, we distinguish the main document and the background document based on the area size and position of different documents in the image. As presented in Figure 5 , different document areas are marked separately, the main document is given the maximum class label value 2 and the corner points coordinates are also marked.  
experiment: 
 In this section, we evaluate the performance of our proposed method on five datasets: SmartDoc dataset [4] , Extended smartdoc dataset [10] , CDPhotoDataset [11] , MIDV2020 dataset [3] , and RWMD dataset. The existing four datasets ( SmartDoc dataset , Extended smartdoc dataset, CDPhotoDataset, and MIDV2020 dataset) have already been described in the introduction section and RWMD dataset is introduced in the previous section. They are all divided into 75% for training and 25% for testing, separately. To illustrate the effectiveness of our method, we compare it with state-of-the-art document localization approaches from several aspects. Furthermore, we evaluate our proposed design modules through ablations on the RWMD dataset.To quantitative evaluate the accuracy of the proposed model, we compute the Jaccard index (JI), which is described in the SmartDoc competition report [4] and adopted in other works [8] [34] . Firstly, we remove the perspective transformation of the ground-truth area and the predicted area to get their correct situations 𝐷 𝑔𝑡 and 𝐷 𝑝 . Then for each frame or image, the Jaccard index (JI) is calculated as follows:EQUATION)5.2 Comparison with SOTAThe first experiment compares the proposed method with other state-of-art (SOTA) algorithms in the SmartDoc dataset. The images in the dataset can be divided into five categories(BG01-BG05) according to different backgrounds. Only BG05 is complex, with similar overlapping document and strong occlusions from other objects. As shown in Table 2 , our RDLNet outperforms the previous works on all sub-test scenarios from BG1 to BG5. The overall JI of RDLNet achieves 99.53% and exceeds other methods including HU-PageScan in [8] , which had achieved the highest overall JI 99.23% before. At the same time, we can observe that our method can achieve almost the same accuracy as BG1-BG4 on BG5, which proves the robustness of our method in more complex scenarios. 3 , which shows that our method still obtains the best accuracy performance when comparing with the other previous methods. In particular, although LDRNet achieves good results on the SmartDoc dataset, it is not suitable for the CDPhotoDataset. Because CDPhotoDataset contains multiple documents in one image that need to be detected, however, LDRNet can only detect a single quadrilateral document. Additionally, in order to compare the efficiency of different methods, we also compared the inference speed of our method with others. As Table 3 shows, our method exhibits an average speed of 0.0385s and 0.0408s per image on the CDPhotoDataset and Extended SmartDoc datasets, respectively, which is comparable to other fastest methods. All the experiment results demonstrate that our proposed RDLNet has great capabilities for locating documents in natural scene images accurately and efficiently.Although many preceding methods has obtained outstanding detection metric results with high JI on publicly available document datasets mentioned above, they are not capable of coping with problems of the real-world document detection and generalizing well, since the scenarios in the datasets above are relatively deprived in the diversity of background and don't contain sufficient complex document photography scenes existing in the real situations. In the third experiment, we evaluate our method and other SOTA algorithms on our new dataset RWMD. RWMD contains a rich [14] 0.8622 HU-PageScan [9] 0.7219 OctHU-PageScan [8] 0.6821 Efficient-SAM [38] 0.6726 SAM [18] 0.6988 U-net [28] 0.6380 LDRNet [34] 0.5261 SEECS-NUST-2 [16] 0.3751 RDLNet(ours) 0.9123 variety of common document types and real-world scenes, which is much more complex and better to reflect the real document scenarios. As described in Table 4 , our proposed method achieves the highest JI value and outperforms other methods. Notably, we also directly apply pre-trained Efficient-SAM [38] and SAM [18] , which are prompt-based segmentation methods, on the RWMD test dataset for document localization with a single center point prompt as prompt input and post-processed output masks as final predictions. It turns out SAM-series models don't support generic document segmentation and localization well due to its specific characteristics of prompt design. As a result, we acquired relatively low JI from SAM models. Figure 6 shows the examples of the comparison results between our method, maskrcnn, LDRnet and SAM on the RWMD test dataset. These results present that our proposed same time, we can also see that although our method has obtained a relatively high JI value, there is still a certain space for further improvement. This also shows that many existing document localization problems have not been well solved. Figure 7 shows some document localization results those with still problems. The area within the green line is the detected document area. The results also verify the significance of the RWMD dataset we construct in this paper for document localization methods measurement, and provide a more realistic and challenging benchmark dataset for future research.Evaluation of inference speed. Furthermore, in order to verify the efficiency of algorithms on mobile devices, we compare the test duration of different methods on mobile phone HUAWEI Mate 40. In Figure 8 , the vertical axis is the JI of the model while the horizontal axis is the inference time. Figure 8 illustrates that our proposed model RDLNet obtains the highest accuracy with low inference time. As can be seen from Figure 8 the inference speed of our model is second only to LDRNet, but the accuracy is much higher than LDRNet. Mask R-CNN obtains the accuracy second to our model, but the inference efficiency is far lower than our method.Table 5 : JI Results on MIDV2020 dataset Method Fine-tune JI Mask R-CNN [14] 0.8607 HU-PageScan [9] 0.6735 OctHU-PageScan [8] 0.7154 LDRNet [34] 0.4221 U-net [28] 0.3201 SEECS-NUST-2 [16] 0.3690 RDLNet(ours) 0.9266 5.2.5 Experiment 5: Generalization on unseen documents and background. To verify our model's generalization ability to new documents on unseen background, we cross-validate our method on the dataset which is not used for training. Specifically, we use the RWMD training set to train models and MIDV2020 test set to verify the detection effects. As demonstrated in Table 5 , our method is significantly better than other methods on unseen dataset. Especially, the accuracy of our method outperforms the second-place method Mask R-CNN by 15.04%. The results prove that our method has good generalization capability on unseen scenarios. In our model, we propose several novel modules, such as crosslevel encoder, multiplex distillation1 , and point-mask joint supervision. To verify the effectiveness of these modules, we carry out corresponding ablation experiments. Table 6 shows the verification results, from which we can observe that the proposed modules are essential for accuracy. 
conclusion: 
 In this paper, we introduced RDLNet, a novel model designed for robust document localization in real-world images. Leveraging a light-SAM backbone, RDLNet efficiently extracts hierarchical visual features and employs a decoder that harnesses mask attention and object queries to accurately predict document corner points, segmentation masks, and categories. The RDLNet exploits the light-SAM backbone to extract the cross-level vision features of the document images and jointly predicts the corner points coordinates, area mask and class of the documents following decoder based on masked attention and object query. At the same time, we create a new document localization dataset RWMD to fill the poverty of dataset for multiple document types and complex scenarios in the wild. Extensive experimental results have demonstrated that our model achieved the state of the art accuracy, while using low inference time. Additionally, the experiments on unseen documents and backgrounds highlight RDLNet's strong generalization capability, underscoring its practical applicability in diverse settings. We hope this research not only advances the state-of-the-art in document localization but also establishes a challenging benchmark that will likely drive future progress in the field. 
 
 Appendix: When distillation is removed, we train the proposed RDLNet from scratch without loading any pre-trained weights or parameters. 


Decision: Summary:

The paper addresses the increasing need for robust document localization methods due to the widespread use of smartphones for capturing documents. The authors propose a novel Real-world Document Localization Network (RDLNet) featuring a light-SAM encoder and a masked attention decoder, aiming to improve the generalization capability of document localization tasks. Additionally, the paper introduces the Real-world Mobile Document (RWMD) benchmark dataset, which covers 9 categories and contains 2009 images, demonstrating the effectiveness of RDLNet through extensive experiments on multiple benchmarks.

Strengths:
1. RDLNet introduces a novel triple-branch structure that simultaneously predicts corner points, area masks, and document categories, enhancing localization accuracy and efficiency.
2. The creation of the RWMD dataset, which includes various document types and challenging real-world scenarios.
3. Experimental results show that RDLNet outperforms state-of-the-art methods on multiple benchmarks, particularly in complex real-world scenes.

Limitations:
1. While the method achieves impressive results, the core components like the light-SAM encoder and masked attention decoder are not novel, which may limit the perceived innovation.
2. More details on data annotation consistency and quality control for the RWMD dataset are needed. Additionally, the scalability of RDLNet on larger-scale datasets requires further validation.
3. The framework's complexity, involving split learning, knowledge distillation, and encryption, might complicate implementation and maintenance. Furthermore, details about the distillation process and selection of hyperparameters should be provided.
4. The comparative analysis may not be entirely reliable without ensuring that all methods were evaluated under the same conditions, including the use of identical training datasets.

According to the comments, the final average rating leans to the positive side. Therefore, I will recommend this paper as Accept (Poster).

Reviews: review: paper_summary: paper_summary: The paper addresses the increasing need for robust document localization methods due to the widespread use of smartphones for capturing documents. The authors highlight the lack of open-source datasets that cover complex real-world scenarios. To address this gap, they created the comprehensive Real-world Mobile Document (RWMD) benchmark dataset, which covers 9 categories and contains a total of 2009 images. Additionally, the paper proposes a novel Real-world Document Localization Network (RDLNet), which features a light-SAM encoder and a masked attention decoder, aiming to transfer the generalization capability of SAM to document localization. The method claims to outperform state-of-the-art approaches on multiple benchmarks.
strength_weakness: strength_weakness: The paper introduces a novel triple-branch structure embedded in the proposed RDLNet, which simultaneously predicts corner points, area masks, and document categories, enhancing the accuracy and efficiency of document localization.

The authors conducted experiments and comparisons with other methods, demonstrating the effectiveness and robustness of the proposed RDLNet.

The authors constructed the Real-world Mobile Document (RWMD) dataset, which includes various document types and challenging real-world scenarios.
limitations: limitations: When presenting table values, it is common to list methods chronologically or in ascending order of results. The authors' table values are in reverse order. To indicate ranking order, it is recommended to re-order or include a ranking number in the first column.

In document localization, images are often captured casually, resulting in blurriness, low lighting, low contrast, or occlusion. Not only boundary-detection methods need traditional image processing. Smartphones recording metadata, like ISO, image size, lens type, and shutter speed, may contribute to improving localization accuracy.
paper_summary: paper_summary: This paper introduces a new method for real-world document localization called RDLNet and establishes a new benchmark dataset RWMD. The innovation of this article mainly lies in the design of a lightweight SAM encoder, a decoder based on masked attention and object queries, and the proposal of a three-branch prediction method. These methods perform exceptionally well in multiple benchmark tests, especially in document localization tasks within complex scenes.
strength_weakness: strength_weakness: 1. The authors propose a lightweight SAM encoder and a decoder based on masked attention, combined with a three-branch prediction method, to address multiple challenges in document localization.
2. The RWMD dataset contains a rich variety of document types and complex shooting scenes, providing valuable resources for future research.
3. The experimental results in the paper demonstrate the superior performance of RDLNet across multiple benchmark tests, particularly in robustness within complex scenes.
limitations: limitations: 1. While RDLNet shows significant improvement in accuracy, further evaluation is needed regarding the algorithm's complexity and inference time performance on mobile devices. It is suggested that the authors provide more detailed data on inference time and resource consumption.
2. The paper mentions the use of multiple distillation methods to obtain a lightweight SAM encoder. It is recommended to provide additional details regarding the distillation process and the selection of hyperparameters.
3. When describing the RWMD dataset, it is advisable to provide more information about the methods for data annotation consistency and quality control to ensure the dataset's high quality and reliability.
4. While RDLNet performs excellently in this paper, its performance on larger-scale datasets still needs validation. It is suggested that the authors provide more discussion and experiments on the algorithm's scalability.
paper_summary: paper_summary: The increasing use of smartphones for capturing documents in various real-world conditions has underscored the need for robust document localization technologies. Current challenges in this domain include handling diverse document types, complex backgrounds, and varying photographic conditions such as low contrast and occlusion. However, there currently are no publicly available datasets containing these complex scenarios and few methods demonstrate their capabilities on these complex scenes. To address the issues, the paper proposes a document localization method named RDLNet, which consists of light-SAM backbone multiplex distillation, cross-level encoder, and masked decoder. Besides, a large-scale real-world document localization dataset is proposed, which reflects more complex scenarios of document images in the wild. Experiments are performed on five datasets to demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1) The proposed dataset includes documents captured by smartphones in real conditions with various types of documents and complex scenarios, which is valuable to the community. 
2) The proposed RDLNet not only utilizes the knowledge from SAM but also produces various types of predictions with unified queries. 
3) The experiments are well conducted and the paper is overall written.
limitations: limitations: 1) The details are absent in Section 3, e.g., how the prompt is utilized for Light-SAM Backbone Multiplex Distillation.
2) Typos. L109: Real world -> Real-world. L433: HW/16 -> HW/16^2.
paper_summary: paper_summary: This paper proposed a real-world document localization method and a new dataset for a real-world document localization dataset. The proposed method consists of a light-SAM encoder and a masked attention decoder. A triple-branch predictions that contains corner point coordinates, instance-level segmentation area and categories of different documents, are adopted to accomplish the task. The experimental results on mulitple benchmarks show the effectiveness of the proposed method.
strength_weakness: strength_weakness: Dataset Contribution: The creation of a new and comprehensive dataset is a significant contribution. It addresses the existing gap in the field by providing a diverse set of document types and real-world scenarios, which is essential for training and evaluating robust localization models.


State-of-the-Art Performance: The method's performance on multiple benchmarks is commendable. Achieving state-of-the-art (SOTA) results in terms of both accuracy and inference speed is a testament to the effectiveness of the proposed approach.
limitations: limitations: Technical Novelty: While the paper presents a new method, the core components such as the light-SAM encoder and the masked attention decoder are not novel in themselves. This raises questions about the extent of the innovation in the proposed method.


Multi-Task Learning: The triple-branch prediction can be seen as an extension of multi-task learning, which is not a new concept. The paper could benefit from a deeper exploration of how this approach differs from or improves upon existing multi-task learning frameworks.


Fairness of Comparisons: The paper's claims of superiority over other methods are contingent on the fairness of the comparisons made. It is crucial to understand whether the compared methods were evaluated under the same conditions, including the use of identical training datasets. Without this information, the comparative analysis may not be entirely reliable.