Boosting Document Layout Analysis with Graphic Multi-modal Data Fusion and Spatial Geometric Transformation

Authors: Zhishan Li, Zhenhao Lu, Haoran Cai, Xubin Li
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Multi-modal Data Fusion, Document Intelligence

Abstract:
Document layout analysis is essential for Document Intelligence, playing a pivotal role in automated understanding and processing of document content. Most existing approaches within this domain are predicated on computer vision techniques that concentrate on image modality, despite documents containing both rich visual and textual information. While recent advances in multi-modal approaches begin to incorporate word embeddings to enhance recognition capabilities, they also incur a substantial computational burden.  Moreover,  the diversity of document structures demands models with great robustness, especially during the document editing process.  In this paper, we introduce pluggable and efficient data pre-processing strategies to boost the layout analysis performance. Firstly, we discover that element categories depend on relative relationships and propose a Graphical Multi-modal Data Fusion technique, which constructs a graph to establish connections between disparate textual segments. Secondly, in terms of structural diversity of documents, we devise a Spatial Geometric Transformation strategy to improve model robustness against layout alterations. Our methods operate during the pre-processing phase, which facilitates straightforward integration with existing models to achieve significant accuracy increase with negligible extra computations. Experimental results show that our strategies illustrate State-Of-The-Art performance across multiple document layout analysis datasets. We will make the code publicly available shortly.

Decision: reject
Reviewer scores: 6, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730204956482

thread_type:
official_review

<thread_object>
creator: reviewer_pmah

<content>
summary:
Current document layout analysis methods are mainly divided into two categories: one is based on computer vision technology, which focuses on image modality but ignores the textual modality in documents; the second is based on multimodal technology, integrating word embeddings to improve recognition accuracy, but inevitably increasing the computational burden after introducing textual modality. The authors found that the relative relationships among elements in the document affect the element categories, thus proposing Graphical Multi-modal Data Fusion technique, which is an image preprocessing module. Its main purpose is to construct a graph to establish connections between different disparate textual segments. At the same time, to enhance the robustness of the model, the authors designed Spatial Geometric Transformation strategy, which enhances the diversity of document structure in three dimensions: sentence, paragraph, and page. This strategy is also used in the image preprocessing stage. The authors skillfully leveraged "multimodal" information to improve accuracy without incurring much additional computation through their proposed strategies. Experimental results show that the authors' strategies have demonstrated state-of-the-art performance on multiple document layout analysis datasets.

strengths:
Graphical Multi-modal Data Fusion technique proposed by the authors effectively integrates multimodal information, combining image modality and textual modality into a Fusion Image that is input into the model. This clever use of "multimodal" information significantly enhances the model's accuracy without substantially increasing its computational load. In addition to this, Spatial Geometric Transformation strategy put forward by the authors introduces variations at the sentence, paragraph, and page levels to enhance the diversity of document structures. This not only improves the model's robustness but also further increases its accuracy.

weaknesses:
1. The authors claim that the proposed method not only facilitates integration with existing models but also achieves significant accuracy improvements with negligible extra computations. However, the author does not discuss why such methods can "negligible extra computations"; They only conducted ablation experiments to prove that their methods do not bring additional burdens.

2. There is a lack of experiments on other commonly used layout analysis datasets, such as PubLayNet and M6Doc. Furthermore, there is no comparison with the latest methods, such as M2Doc. Whether the proposed method can be applied to or integrated with the most recent approaches, such as M2Doc. Additionally, the performance comparisons and ablation studies are only conducted on DocLayNet, which is not convincing.

[1] Zhong X, Tang J, Yepes A J. Publaynet: largest dataset ever for document layout analysis[C]//2019 International conference on document analysis and recognition (ICDAR). IEEE, 2019: 1015-1022.

[2]Cheng H, Zhang P, Wu S, et al. M6doc: A large-scale multi-format, multi-type, multi-layout, multi-language, multi-annotation category dataset for modern document layout analysis[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 15138-15147.

[3]Zhang N, Cheng H, Chen J, et al. M2Doc: A Multi-Modal Fusion Approach for Document Layout Analysis[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(7): 7233-7241.

questions:
In the paper, it would be advantageous to highlight the performance improvements achieved with minimal additional computational cost, along with an analysis that explains the underlying reasons for these gains. This approach would not only more effectively underscore the key contributions of the work but also pave the way for future research in this area. Furthermore, for aesthetic purposes, it is suggested to shift the 'Parser / OCR' label in Figure 2(a) slightly to the left. To strengthen the credibility of the proposed method, it is advisable to conduct a comparative analysis against previous methods on more benchmarks, as well as to include ablation studies on additional datasets.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730354202325

thread_type:
official_review

<thread_object>
creator: reviewer_eyjb

<content>
summary:
The manuscript proposes two methods in the pre-processing stage of a document layout analysis (DLA) task. The proposed models fall under the category of multi-modal methods, as they combine semantic information (e.g. word embeddings) along with the vision-based information (e.g. spatial positions). While a few multi-modal methods exist, the primary contribution of the paper is on computational cost savings, while still achieving a better mAP score.

The first method is a Graphic Multi-modal data fusion (GMDF) stage, which constructs a graph from spatial and semantic relationships. The original document image is parsed through an OCR, and tokenized. The word embeddings are used to from word and sentence graphs. The fusion of the spatial and semantic graph in then fed to the backbone of the network.

The second method is a data augmentation technique. Various augmentations are produced based on sentence remixing, paragraph perturbation and crops are used to improve the generalization of the model.

The approach is evaluated on 4 DLA datasets (DocLayNet, D4LA, PubLayNet and Docbank), and against 2 detectors: FasterRCNN and CascadeRCNN. Compared to VGT (previous SOTA), the average mAP score goes up by 1.0 while reducing the FLOPS to half.

strengths:
The paper's has a few strengths in significance: computational cost, pluggable pre-processing step and a multi-modal approach to gather both semantic and spatial information. The proposed GMDF + SGT stages are able to perform as good as SOTA VGT method with nearly half the FLOPs. The proposed approaches are a pre-processing step, whose output is fed to the backbone of any DLA network. This makes the method applicable to a lot more use-cases. Lastly, using both textual and spatial information usually outweighs vision-only approach, and this method proves this again. 

In terms of experiments, ablation studies are performed, along with comparisons with both vision and multi-modal approaches. The comparison with VGT method is clear in terms of mAP and FLOPs.

weaknesses:
The primary weaknesses of the paper are insufficient experiments, reasoning of success, and originality. 

Originality: It is unclear how a text and spatial graph is novel. The paper itself talks about SOTA approaches that have done the same, but is not able to clarify what makes the GMDF stage novel. Creating a word graph, and constructing a sentence graph from it has been a known and well studied work. As graph-based methods have known scaling challenges, it is unclear why the given approach will flare better? The second stage SGT is a well-known data augmentation technique. It does leave open questions like -
- What is the effect of scaling on OCR parsing?
- Why augment here instead of abstracting it away from the architecture, and pulling it in training procedures

Reasoning of Success: On the lines of novelty, it is unclear why does the approach perform better? What are the areas it fails and why?

Experiments: Lastly, while the mAP scores are compared against a gamut of SOTA approaches, VGT is picked as the only one from the list for FLOPs comparison. How does the proposed approach compares to other SOTA techniques with similar mAP scores: Hybrid (V+BERT-3L), Hybrid (V+BERT-12L), GLAM+YOLOv5x6?
- How does the approach generalize to non-standard documents with unstructured layouts?
- How does it get affected by a bad OCR result? Do the semantic information start to hurt more than help then?
The data augmentations are also limited to scaling, translating and cropping. It does not seem to capture the real world issues like skewing, font changes, noise artifacts etc.

questions:
- As graph-based methods have known scaling challenges, it is unclear why the given approach will flare better? The second stage SGT is a well-known data augmentation technique. It does leave open questions like -
- What is the effect of scaling on OCR parsing?
- Why augment here instead of abstracting it away from the architecture, and pulling it in training procedures
- why does the approach perform better? What are the areas it fails and why?
- How does the proposed approach compares to other SOTA techniques with similar mAP scores: Hybrid (V+BERT-3L), Hybrid (V+BERT-12L), GLAM+YOLOv5x6?
- How does the approach generalize to non-standard documents with unstructured layouts?
- How does it get affected by a bad OCR result? Do the semantic information start to hurt more than help then?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730403773833

thread_type:
official_review

<thread_object>
creator: reviewer_ssuv

<content>
summary:
The paper describes a simple method for image pre-processing specially designed to obtain a better image representation for document layout analysis. Two different pre-processing strategies are proposed: on one hand, enriching image information with semantic information obtained through word embedding and the relations between words and sentences in the document. On the other hand, a data augmentation strategy specific for document layout analysis. The proposed strategies can be integrated with several existing DLA methods with a very reduced extra computation cost. Experimental results analyze the contribution of the two proposed pre-processing strategies and compare the performance with current SoA on standard benchmarks, showing SoA performance.

strengths:
- The proposed method is simple, efficient and can be integrated into several DLA methods. 
- Experimental results show that the Graphical Multi-modal Data Fusion module improves the results of Faster-RCNN and Cascade-RCNN, obtaining SoA results on the standard benchmarks.

weaknesses:
- Although the experiments show an improvement in the performance when the pre-processing step is applied, it is not clear to me which is the reason that brings that improvement, since the proposed fusion approach does not seem very intutitive to me. It is just the result of summing raw pixel values with some very highly compressed semantic word and sentence information, which can be very unrelated values. It would be valuable if you could explain the rationale behind the fusion of raw pixel values with semantic information.Moreover, semantic word embedding undergoes an aggressive projection from 768 dimensions to only 3, with the risk of high informaton loss. It is difficult to me to visualize which enriched information is added to the original image in eq. (8). Some discussion on this would be useful. 
- The ablation analysis is incomplete. The contribtuion of the two strategies is evaluated but the multimodal fusion module should be analyzed with a deeper detail. It should be analyzed the contribution of word and sentence graphs (i.e, what are the results using only the word graph or the sentence graph?). The contribution of the attention on the graph should also be analyzed (comparing results using the sentence/word features with and without attention on the graph) since one of the starting hypothesis is that the relations among elemetns in the document are relevant for DLA. 
- The contribution of the SGT is a bit marginal

questions:
- The fusion strategy only modifies the original image where there is text. What about non-text elements, such as tables or figures? What is the impact of the method on those regions? 
- In equation (8), is the token embedding w_m the result of attention on word graph? Or is it just the original token embedding without attention?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730657793753

thread_type:
official_review

<thread_object>
creator: reviewer_tgtr

<content>
summary:
This paper presents an architecture for learning a representation of document images. The presented model is addressed to improve the efficiency of Document Layout Analysis systems. It is presented as a pre-processing strategy that can be plugged in baseline methods for semantic segmentation. The paper taxonomizes existing methods for DLA as Vision-based and Multi-modal models. The proposed contributions are: First, the Graphic Multi-modal Data Fusion model uses Graph Attention Networks to generate image features at pixel level that combine word, sentence embeddings with attention scores between them as a structural representation. The second proposal is the  Spatial Geometric Transformation, that consists in several document operations that allow to augment the data used for training, boosting the layout features. Both strategies are integrated in pre-processing steps in the experimental setup consisting in FasterRCNN and CascadeRCNN as baselines.

strengths:
A multimodal system is presented, that combines visual information, semantic embeddings from words and senteces, and an attention mechanism capturing the relative relationships between text elements.

The Spatial Geometric Transformation is a simple but effective method consisting in basic operations that allow to augment the original data with plausible document new instances.

The proposed representation is experimentally shown as an improvement when it is used as a pre-processing step in classical baselines.

weaknesses:
The use of structural information, in particular graph-based representations is not novel. There are several works that combine visual, textual and structural features. In particular, the attention mechanism that is proposed is the classical implementation of the Graph Attention Networks (GAT).

In a layout, there are other components than just text. The strategy of modeling relationship between elements is baed on text words and sentences, but not considering other elements like figures, images, tables... The method seems to be highly sensitive to the detection of text words and lines, disregarding a more macroscopic representation.

questions:
How does your method differ from a classical Tranformer model, that actually captures the attention between words of a text? The attention mechanism implemented between sentences resembles a Graph Transformer architecture.

It is not clear to me how sentences are obtained. Are they "meaningful" sentences obtained by the BERT model? or just word lines? 

The extraction of edges in eq (3) is based in the 4 nearest sentences, in terms of their position. Have you considered other strategies, like a visibility graph?

======================== AFTER THE REBUTTAL

After interacting with the authors, and looking at the other reviewers' revisions, I will keep the score of the first review. I thank the authors for their clarifying responses, and the effort to consider the comments. The authors have solved most of my concerns. However, I consider that the proposed method and contribution still has room for improvement in a more solid work, and it prevents me to raise the score.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper describes pre-processing strategies to enhance document layout analysis as a crucial step in automated document understanding. The authors point out critical issues with existing image-centric approaches and propose a Graphical Multi-modal Data Fusion technique to connect textual segments by leveraging their relative relationships. Additionally, a Spatial Geometric Transformation strategy is introduced to improve  robustness across varying document structure. These methods aim to improve accuracy while adding minimal computational overhead and are compatible with existing models. Experimental results are given on multiple datasets.

Reviewers variously articulated the following strengths of the proposed approach:
+ **Simplicity and Effectiveness of the Spatial Geometric Transformer**: Spatial Geometric Transformation is a simple, but effective, way of combining basic operations to augment the original data with new, plausible document instances.
+ **Plug-and-Playability**: The representation proposed by the authors can be used as a pre-processing step in classical baselines, leading to improved results with used in combination with Faster-RCNN and Cascade-RCNN models.
+ **Efficiency**: The combination of Spatial Geometric Transformation and Graphical Multi-modal Data Fusion performs as well as the state-of-the-art, but at about half the computational costs in FLOPs.

However, reviewers also articulated a number of weak points that outweigh the strengths of the work:
+ **Insufficient analysis and Motivation for Performance Improvements**: While the performance and efficiency improvement is empirically quantified, there is little analysis or motivation given in terms of *why* the proposed approach yields the empirically observed results. Ablations were added in rebuttal, but a much deeper analysis is needed to justify the proposed approach -- especially in light of the relatively marginal novelty of the proposed system (see below).
+ **Reliance on Word Graphs**: The use of word graphs derived from OCR is questionable, as reviewers point out that there is much more to document layout than spatial arrangement of text. Sensitivity to missed text detection due to OCR failures is not addressed.
+ **Novelty**: Reviewers noted that the Spatial Geometric Transformation proposed is a standard Graph Attention Network, and that the combination of combining visual, textual, and structural information for layout analysis is very standard. As such, the proposed approach is a combination of well-known techniques and represents little novel contribution.

The consensus after the rebuttal period is that the work is in need of significant revision and reformulation before it can be considered for acceptance.