Vision-Enhanced Semantic Entity Recognition in Document Images via Visually-Asymmetric Consistency Learning
Authors: Hao Wang
Xiahua Chen
Rui Wang
Chenhui Chu
Venue: EMNLP 2023 Main (2023)
Keywords: Visually rich documents
Information extraction
Consistency Learning
Multimodality
TL;DR: An empirical, effective, yet simple approach for visually rich document information extraction using consistency learning

Abstract: Extracting meaningful entities belonging to predefined categories from Visually-rich Form-like Documents (VFDs) is a challenging task. Visual and layout features such as font, background, color, and bounding box location and size provide important cues for identifying entities of the same type. However, existing models commonly train a visual encoder with weak cross-modal supervision signals, resulting in a limited capacity to capture these non-textual features and suboptimal performance. In this paper, we propose a novel Visually-Asymmetric coNsistenCy Learning (VANCL) approach that addresses the above limitation by enhancing the model's ability to capture fine-grained visual and layout features through the incorporation of color priors. Experimental results on benchmark datasets show that our approach substantially outperforms the strong LayoutLM series baseline, demonstrating the effectiveness of our approach. Additionally, we investigate the effects of different color schemes on our approach, providing insights for optimizing model performance. We believe our work will inspire future research on multimodal information extraction.

Vision-Enhanced Semantic Entity Recognition in Document Images via Visually-Asymmerric Consistency Learning
Hao Wang', Xiahua Chen', Rui Wang and Chenhui Chu
'School of Computer Engineering and Science, Shanghai University, China
Department of Computer Science and Engineering, Shanghai Jiao Tong University, China Graduate School of Informatics, Kyoto University, Japan wang-hao@shu. edu. cn cxh1 cxh110029@gmail.. com wangrui 1 2@s edu. chu@i kc.
Abstract
ANOT DASORENT
Extracting meaningful entities belonging to pre- defined categories from Visually-rich Form- like Documents (VFDs) is challenging task. Visual and layout features such as font, back- ground, color, and bounding box location and size provide important cues for identifying en- tities of the same type. However, existing models commonly train a visual encoder with weak cross-modal supervision signals, result- ing in a limited capacity to capture these non- textual features and suboptimal performance. this paper, we propose a novel Visually- Asymmetric coNsistenCy Learning (VANcI.) approach that addresses the above limitation by enhancing the model's ability to capture fine- grained visual and layout features through the incorporation of color priors. Experimental results on benchmark datasets show that our approach substantially outperforms the strong LayoutLM series baseline, demonstrating the effectiveness of our approach. Additionally, we investigate the effects of different color schemes on our approach, providing insights for optimizing model performance. We believe our work will inspire future research on multi- modal information extraction.
ROON BLAE PANE CORBER JABLOO
Consgneo 93A1
DOU POLONOOOO
ETE MAXT 10 D0M.E--11000
Question
Answer
Question
Answer
Sippr 3REAA!
RS4NF
SPECITIS
Why are
INTERNATIOOAL
TARGET
To Meea leing
LIMITDD
CONKUNICCTION
Figure 1: Motivation of this work. Semantic entities of the same type often have similar visual and layout properties, such as the same similar font, background. color, and the location and size of bounding boxes, pro- viding important indications for recognizing entities and their types. Despite the importance of these properties, existing LMMs for information extraction in VFDs of- ten rely on a limited visual encoder that cannot fully capture such fine ene-ggained features. Therefore, this work focuses on incorporating these visual priors using colors into the task of IE for VFDs.
1 Introduction
present in the document, it is necessary to leverage multimodal information, including text, spatial, and visual data. Therefore, large-scale pretrained Mul- timodal Models (LMMs) (Gu et al., 2021; Li et al., 2021a; Appalaraju al., 2021; Xu et al., 2021; Huang et al., 2022: Wang et al., 2022a; Lee et al., 2022), which are models that can process multiple modalities of data, have emerged as the dominant approach in IE for VFDs in recent years. State-of- the-art LMMs integrate advanced computer vision models (Ren et al., 2015; He et al., 2016) within BERT-like architectures (Devlin et al., 2019) to leverage spatial and visual information along with text and learn multimodal fused representations for form-like documents. However, these representa- tions are biased toward textual and spatial modal- ities (Cooney et al., 2023) and have limited per-
Information extraction (IE) for visually-rich form- like documents (VFDs) aims to handle various types of business documents, such as invoices, re- ceipts, and forms, that may be scanned or digitally generated. This task has attracted significant atten- tion from the research and industrial communities (Xu et al., 2020; Garncarek et al., 2021; Gu et al., 2021; Li et al., 2021a.b). As shown in Figure 1, the goal of IE for VFDs is to identify and extract mean- ingful semantic entities, such as company/person names, dates/times, and contact information, from serialized OCR (Optical Character Recognition) output in the documents. Since a single modal- ity may not capture all the semantic information
Coorresponding author

---

formance, especially when the data contains richer visual information. This is because the visual en- coder in these models usually plays a secondary role compared to advanced text encoders.
Our contributions can be summarized follows:
We propose novel consistency learning approach using assmmmrrii inputs, called VANCL, which effectively incorporates prior visual knowledge into multimodal repre- sentation learning.
There are two problems with the visual encoder in previous LMMs. First, these models only impose coarse-grained cross cross-modal constraints during pre- training (e.g., text-image, word-patch, and layout- text aligmment (Xu al., 2021; Huang al., 2022; Wang et al., 2022a)) to enhance feature extraction from the visual channel, but this does not capture sufficient fine-grained visual features, leading to limited performance and underutilization of prior knowledge in vision. Second, the visual encoders in previous LMMs have weaker representational capabilities than those in the latest Optical Char- acter Recognition (OCR) engines because they do not consider the intermediate tasks such as text segment detection and bounding box regression, which are important for accurately localizing and extracting fine-grained visual features.
We demonstrate the effectiveness of VANCL by applying it to the task of form document information extraction using different LMM backbones. Experimental results show that the improvements using VANCL are substantial and independent of the backbone used.
We investigate how different color schemes affect performance, and the findings are con- sistent with cognitive psychology theory.
2 Problem Formulation
We treat the Semantic Entity Recognition (SER) task as a sequential tagging problem. Given a doc- ument D consisting of a scanned image I and a list of text segments within OCR bounding boxes B (b.....,by) we formulate the problem finding extraction function FIE(D (B,I)) - & that predicts the corresponding entity types for each token in D. The predicted sequence of labels is obtained using the "BIO" scheme (Begin, Inside, Other) and a pre-defined label set. We train our sequential tagging model based on pretrained LMMs and perform Viterbi decoding during testing to predict the token labels. Each bounding box b; contains set of M tokens transcribed by OCR engine and coordinates defined (#t .x?) for the left and right horizontal coordinates and (w/ vz) for the top and bottom vertical coordinates.
To address the issues mentioned above, inspired by recent works on consistency learning (Zhang al., 2018; Miyato et al., 2019; Xic et al., 2020; Lowell et al., 2021a; Liang et al., 2021; Chen et al., 2021b), we propose novel vision-enhanced training approach, called -ssally--symmetric CoNsistenCy Learning (VANCL). By incorporating color priors with category-wise colors as additional cues to capture visual and layout features, VANCL enhance the learning of unbiased multimodal representations in LMMs. Our approach aims to jointly consider the training objectives of the text segment (or bounding box) detection task and the entity type prediction objective, thereby bridging the preprocessing OCR stage with the downstream information extraction stage.
Approach
VaNCL involves standard learning flow and an extra vision-enhanced flow. The inputs are asym- metric in the visual modality, with the original doc- ument images input to the standard flow, while im- ages to the vision-enhanced flow are the synthetic painted images. The vision-enhanced flow also can be detached when deploying the model. During training, we encourage the inner visual encoder to be strong as the outer visual encoder via consis- tency learning. As result, VANCL outperforms existing methods while (1) extremely little manual effort to create synthetic painted images, (2) no need to train from scratch, whilst (3) no increase in the deployment model size.
Our model architecture for aismmlly-syymmetric consistency learning is illustrated in Figure 2. In- spired by mutual learning (Zhang et al., 2018), we start with a standard learning flow and an extra vision-enhanced flow, which are learned simul- taneously to transfer knowledge from the vision- enhanced flow to the standard learning flow. It is worth noting that the input images for the vision- enhanced flow colorful prompt paints, while the input images for the standard flow are original images. Therefore, the information contained in the visual inputs to the assion-enhanced flow and the standard model is asymmetric.

---

Preprocessing
2 Encoding
Consistency Learning
Labeling
VFDs
Multimodel channels
LMMs w color priors
Semantic Entity Recognizer
LMMs
textual
I
spatial
weight tuning
enhanced
weak inner encodar
color-naive representation
represantation
visual
fiee E
structured results
painting
y
Info asymmetric
coons
Lee 4
ground truth
CSTAT- SAL (S:
Ps
i
color potches
visual
color-eic
PORT LOADRIG (ONGL000"
robust encoder
representation
reverse alligiigg
color assigning
VISSO PALIETT" WT TSRRNOSNIIE
co/or scheme
standard learning flow vision-enhanced flow
< loss function
cletach
Figure 2: The overall illustration of the VanCL framework. It encourages knowledge transfer from the extra vision-enhanced flow to the standard flow through consistency losses.
3.1 Overall architecture
where L.e is the cross-entropy loss function and y* is the ground truth. P (B*,); θ) and (Bk (B); ; O, Ox) refer to the corresponding predicted probability distributions of standard and esshnn-enhanced models, respectively. B* I de- note the bounding box position information and the original image of the k-th document. I refers the synthetic image with color patches.
The proposed approach consists of two entity recog- nition networks: a Standard Learning (SL) flow to train the backbone LMMs and an extra vision- enhanced (VE) flow to transfer the prior visual knowledge. The two networks have the same struc- ture and share the parameter weights O in the back- bone LMMs. Note that the Vision-Enhanced flow has an extra visual encoder, and thus has addi- tional parameter weights O, during training. The two flows are logically formulated as follows:
The consistency loss defines the proximity be- tween two predictions. During training, there exists inconsistency due to asymmetric information in the inputs to the standard learning flow and the vision- enhanced flow. Concretely, ize the gap between two soft label signals (i.e., the prediction distributions) generated by the standard and vision-enhanced flows. The consistency loss is computed as:
Py = (x (x;); θ),
necessary to penal-
Py (vE((;
where Py and Py are the predicted probability dis- tributions output by the standard learning flow and the extra vision-enhanced flow, which are the la- tent outputs after softmax normalization (i.e., soft label). Note that the inputs for two networks are different, namely X and X, the former is the origi- nal document image, and the latter is the synthetic document image (with additional color patches).
Coons
Σ Q(P(()(B*,I'))
|DK
XCDK
(4) where Q is a distance function that measures the divergence between the two distributions.
The training objective contains two parts: super- vision loss Loup and consistency loss Lcons- The supervision losses are formulated using the stan- dard cross-entropy loss on the annotated images as follows:
The final training objective for visually- asymmetric consistency learning is written as:
Lanal Cyup(y" 0v) (P, Ps), (5)
Lsup
Σ Lev (P ())
y")
where is hyperparameter for the trade-off weight. The above loss function takes into account the consistency between hard and soft labels, which also reduces the overconfidence of the model.
DD
XCDK
+ Eco (P 1,); (3)

---

3.2 Painting with colors
ing. These are exciting and interesting findings in this work.
Visual-text alignment is essential for learning mul- timodal representations, but fine-grained alignment the bounding box level has not been adequately captured by previous models. Therefore, it is im- perative to explore methods for bridging the gap between text segment (or bounding box) detection and entity classification tasks.
4 Experiments
4.1
Datasets
We conduct our experiments on three datasets, FUNSD (Jaume et al., 2019), SROIE (Huang et al., 2019), and a large in-house dataset SEABILL (See Table 1). Details refer to Appendix A.1.
One natural solution is integrate label informa- tion using colors, which could effectively enhance visual-text alignment by representing entity-type information with color patches. However, the man- ual creation of these visual prompts would be ex- tremely consuminn and laborious. To tackle this problem, we adopt simple and ingenious pro- cess that uses OCR bounding box coordinates to automatically paint the bounding boxes with colors in the original image copies.
'Train
'Tss
Dateset
Doc
BD
Token
Doc
BD
Token
FUNSD SROIE SEABILL
149
21K
33K
50
12K
626
34K
347
19K
86K
3,562
250K
1.5M
953
74K
0.5M
Table 1: Statistics of the used datasets, including the numbers of documents (Doc), bounding boxes (BD). and tokens (Token).
Let D denote the k-th document, consisting of (B*, ,Th.. First, we make an image copy In for each training instance of VFD. Then, we paint the bounding boxes in the image copy with the colors responding to entity types according to the coordinates [<=] 2 (vl vf)] of bounding box b;- Hence, obtain new image I with color patches after painting. This process can be repre- sented as follows:
4.2 Backbone networks
We conduct our experiments using LayoutLM series backbones, family of transformer-basee well-pertained LMMs specifically designed for visually-rich document understanding tasks.
LAYOUTLM Vanilla LayoutLM model
1 Paint ROIAlign( (B* ) (
(6)
LAYOUTIM-BASE-UNCAASI (Xu et al., 2020) pretrained only using text and layout information. LAYOUTLM w/ img Given the vanilla LayouLLM model does not utilize the visual information, we integrate LayouLLM with ResNet-101 (He et al., 2016) to enable the capability of feature extraction from the visual channel!.
where ROIAlign obtains fine-grained image areas corresponding to bounding boxes within the region of interest (here refers to B*) I the newly gen- erated synthetic image that preserves the original visual information as well as the bounding box pri- ors with label information. We use these prompt paints to train the extra vision-enhanced flow.
We also make comparisons by substituting the backbone with LAYOUTTLMYY-BASE-UNAAAEE BASE-UNCASEDD (Xu et al., 2021) and LAYOUTLMV3-BASS (Huang et al., 2022).
AAYOUTLMYY/L.VVOUTL..
3.3 Dual-flow training
VANCL (this work) We initialize the VANCL from existing pretrained LayoutLM backbones and share the parameter weights of the LayoutLMs in the standard learning and vision-enhanced flows, including the text encoder (c.g., BERT), the inner visual encoder (e.g., ResNet, ResNeXt, and Linear), the position encoders. the Transformer layers, and the classifier. For the outer visual encoder, we use ResNet-101 as the default and also test non-pretrained CNN and ResNet.
Avoiding label leakage becomes major concern this task when directly training backbone LMMs with prompt paints using only the standard flow. Fortunately, the dual-flow architecture used in our model not only allows for detaching the vision- enhanced flow and discarding prompt paints when testing but also enables the use of arbitrary back- bone LMMS and any available visual encoders in the outer visual channel. This strategy avoids label leakage and enhances the visual feature-capturing ability of the original LMM through consistency learning-bas based strategy similar to adversarial learn-
Note is different from Xu et al. (2020) in which they use Faster-RCNN (Ren et al., 2015) as the visual encoder.

---

FUNSD
SROIE
SEABILL
Model
#aaram
P%
R%
F1%
P% 1
R%
F1%
P%
R% 1
F1%
BERT!
110M
54,69
61.71
60.26
90.99
90.99
90.99
66.70
68.77
67.72
ROBERTA
125M
63.49
69.75
66.48
91.07
91.07
91.07
64.35
67.76
66.01
UNILMV23
125M
63.49
69.75
66.48
94.59
94.59
94.59
BROS
139M
80.56
81.88
81.21
94.93
96.03
95.48
DOCFORMER
149M
77.63
83.69
80.54
LILT[EN-RJ
87.21
89.65
88.41
84.67
86.02
85.64
[IL
84,67
87.09
85,86
85.19
87.39
86.29
FORMNET
217M
85.21
84,18
STRUCTEXTS
107M
85.68
80.97
83.09
95.84
98.52
96.88
XYLAYOUTLM
83.35
90.75
91.59
91.17
LAYOUTLM
113M
75.97
81.55
78.66
94.38
94.38
94.38
86,93
89.16
88.03
LAYOUTLM img)
147M
79.68
80.74
95.53
95.80
87.50
88.68
VANCI_[AYOITI ingg))
+0M
80.78
81.89
81.33
96.32
96.67
96.50
89.06
90.24
89.65
LAYOUTLMV2"
200M
80.29
85.19
82.76
96.25
96.25
96.25
90,90
91.73
91.31
VANCL[LAYOUTLMV2]
+0M
82.95
83.29
83.12
97.45
97.58
97.51
91.61
92.17
91.89
133M
90.29
96,59
96.94
96.77
89.53
91.78
90.64
VANCI/LAYOUTLMV3)
+0M
91.76
92.95
92.35
97.09
97.30
97.20
90.64
92.07
91.35
Table 2: Precision, Recall, and Fl the FUNSD, SORIE, and SEABILI datasets. [$] denotes the backhone model used in VANCL. Bold indicates better scores comparing the model performance between the standard training approach and VANCL. #Param refers to the number of parameters for deployment. '(Devlin et al., 2019), ? (Liuu et al., ( (Bao et al., 2020), '(Hong et al., 2021), 2021), (Appalaraju et al., 2021), (Wang al., 2022a),? (Lee et al., 2022), ((i)) et al., 2021b), "(uu et al., 2022), (Xu et al., 2020), (Xu et al., 2021), 12 (Huung et al., 2022).
4.3 Experimental setups
isting LMMs for visually-rich document analysis and information extraction tasks with little or no significant modifications the network architec- ture. Please refer to Appendix A.3 for detailed case study.
We train all models based on the default parame- ters using the Adam optimizer with of (0.9, 0.99) without warmup. Our models set to the same learning rate of 5e 5. For the FUNSD and SROIE datasets, we set the Dropout to 0.3 to prevent model overfitting, while we reduce the Dropout to 0.1 for the SEABILL dataset. We set the training batch size to 8 and train all models on an NVIDIA RTX3090 GPU. We trained our models for 20 itera- tions to reach convergence and achieve more stable performance.
5.2 Impact of consistency loss
As for a comprehensive evaluation of each mod- ule, we conduct the additional ablation experiment by examining the impact of using consistency loss. The experimental results in Table 3 reveal that re- moving the consistency loss leads to decline in the model's performance to varying extents. This find-
5 Result Analysis
Model
FUNSD
SROIE
SEABILL
LAYOUTL img)
5.1 Overall evaluation
+ VANCL
81.33
96.50
89.65
+ VANCL-CL
79.36(] 1.97)
95.68(} 0.82)
19.1 1.46)
Table 2 shows the scores of precision, recall, and
LAYOUTLMV2
Fl on the form document information extraction task. Our models outperform the baseline models in terms of both LayoutLM (a model that only con- siders text and spatial features) and LayoutLM w/ LtgeTaaaouLL_MM222 M Mt (models that also consider visual features). It should be noted that LayoutLM w/ img incorporates visual features af- ter the spatial-aware multimodal transformer layers, while LayoutLMMe LM22 and LayoutL My33 incorporate visual features before these layers. suggests that VANCL could be easily applied to most the ex-
VANCL
83.12
97.51
91.89
VANCL-CL
81.52(1
1.60)
97.08(L 0.43)
89.2 89.28( 2.61)
LAYOUTLMv3
+ VANCI
92.35
97.20
91.35
VANCL-CL
90.93(1 1.42) 96.98( 0.22) 90.87() 0.48)
Table 3: Ablation experiment by examining the impact of using consistency loss on different backbones. CL means consistency loss is used.
ing demonstrates the significance of consistency loss in the model. Simultaneously. indicates that

---

X
fsL
Py
fsu
y
x
x
y
fvK
Pg
fve
Pg
fst.
Pg
(a) VANCE
(b) Mutual learning
(c) R-drop
Figure 3: Illustrating the architectures for (a) our approach VANCL. (b) mutual learning, (c) R-drop. means
forward operation and '-->" means loss of supervision.
without the driving force of consistency, the addi- tion of color information may actually increase the learning noise.
datasets.
5.4 Comparison with inner/outer consistency To demonstrate the effectiveness of our proposed method, we compare our approach versus R-Drop (Liang et al., 2021) and mutual learning (Zhang et al., 2018). R-Drop is a powerful and widely used regularized dropout approach that considers multiple model inner consistency signals. Fig- ure 3 illustrates the different structures of the three methods. Table 5 gives the results of incorporat- ing LayoutLMS with R-Drop and mutual learning. Compared R-Drop and mutual learning, we ob- serve a positive correlation between using VANCI. and the improvements on the FUNSD, SROIE, and SEABILL datasets. The results indicate that VANCL benefits from the vision priors.
5.3 Different divergence metrics
There many ways measure the gap between distributions, and different measures lead different consistency loss types, which can also affect the final performance of the model on form document extraction tasks. In this section, we ver- ify two types of consistency loss, densen-Shannol (JS) divergence (Lin, 1991) and Kullback-Leible (KL) divergence (Kullback and Leibler, 1951), on the effect of form document extraction. While the
Model
FUNSD
SROIE
SEABILL
LAYOUTLM(W/
80.20
95.80
88.68
+ +VANCL-JS
80.75
96.50
89.36
+VANCL-KL
81.33
96.32
89.65
Model
FUNSD
SROIE
SEABILL
LAYOUTLMV2
82.76
96.25
91.31
LAYOUTLM (w/img)
80.20
95.80
88.68
+VANCI-1S
82.21
97.33
91.89
+ VANCLLKL
83.12
97.51
91.58
+MUTUAL-LEARN
80.44
95.46
88,99
LAYOUTLMY3
90.29
96.77
90.64
+R- +R-DROP
96.02
89.13
+ +VANCL-JS
92.35
97.07
91.16
+VANCL
81.33
96.50
89.65
VANCL-KL
91.72
97.20
91.35
Table 5: Comparison of Fl scores with inner and outer consistencies using R- R-Drop (Liang et al., 2021) and mutual learning (Zhang et al., 2018).
Table 4: Effect of different consistency losses using JS and KL divergences on Fl scores. Bold indicates the best and underline indicates the second best.
5.5 Visualization analysis
VANCL model outperforms the baseline in most cases, regardless of the different backbone net- works and datasets, it still worth noting that the choice of consistency loss varies depending on the characteristics of the dataset, such as key type and layout style, and whether overfitting occurs due to the complexity of the model and the data size. As shown in Table 4, diffcrent consistency losses used for different datasets and backbone networks achieve optimal results. For example, when us- ing Jensen-Shannon divergence for Layoutl.Mv2, VANCL achieved optimal results on the SEABILL dataset, but second best on the FUNSD and SROIE
To more intuitively demonstrate the effect of VANCL on the obtained visual encodings in the visual channel, we visualize the encoding distribu- tions before and after VANCL training via t-SNE. Figure 4 shows the integration of visual informa- tion in both flows. This demonstrates that VaNcI. indirectly transfers the label prior distribution the standard flow by aligning the information in the standard and vision-enhanced flows, which im- proves the subsequent inference process. More detailed results the overall distribution visualiza- tion can found in Appendix A.4.

---

QUESTION
HEADER
OTHER
MICRO-AVG
color
F1
color
FT
color
FT
color
F1
1
#FF0000
83.88
85.51
#O00
56.31
#TTA500
77.36
81.33
2
#0000FF
84.42
#FFO00D
85,49
#00F1-00
57.92
#FFA5000
75.83
81.28
#FFA500
84.01
#O0FF0O
85.22
#FF0000
54.08
#0000FF
77.38
81.27
4
#CCCCCC
83.67
#999999
84,78
#333333
57.18
#000000
77.47
81.16
#I-F0000
83.52
#FF6699
84.88
#FF33366
56.02
#FF0099
80.75
6
#0000FF
83,98
#FF0000
84.59
#0099FF
56.40
#0066CC
76.42
80.76
7
#FF0000
83.43
#0000FF
83.62
#FFA500
55.73
#00FF00
76.30
80.23
8
#FFFFFF
83.45
#FFFFFF
84.21
#FFFFFF
61.40
#FFFFFF
75.58
80.38
SUPP.
2.542
3,294
374
2,356
8,566
Table Results using different color schemes on the FUNSD dataset. SUPP. denotes the number of supports for each entity type in the test set.
Ratio (F1)
Model
5%
12,5 %
25%
50%
100%
LAYOUTLM(w/ img)
71.74
80.67
83.72
85.97
88.68
+ANCC
76.33
82.09
84.83
87.14
89.65
Vaian-enhancadd
LAYOUTLMy2
80.63
84.48
88.12
89.47
91.31
+VANCL
84.75
87.42
89.34
91.18
91.89
VANCE
Table The impact of reducing the data size of the training subset the SEABILL datase.
After training
5.7 Low-resource scenario
Standard
To verify the effectiveness of VANCL in low- resource scenarios, investigate whether VANCL improves the SER task when training the model with different sizes of data. We choose LayoutLM and LayoutL.Mv2 as the test backbone and com- pare the results of VANCL with the correspond- ing baseline by varying the training data sizes. Specifically, we randomly draw a percentage p of the training data, where is chosen from the set (5%, 12.5%, 25%, 50%), from the SEABILL dataset. Results in Table 7 reveal two key obser- vations. First, VANCL consistently outperforms the LayoutLM baselines for different sizes of train- ing data. Second, the performance gap between VANCI and the baseline is much larger a low- resource scenario, indicating that VANCL can boost the model training in such scenarios.
Figure 4: The t-SNE visualization of token-level visual encodings of a particular type output by the standard (red) and vision-enhanced (grayish purple) flows.
5.6 Changing the color scheme
To verify the effect of different background col- ors on the model, we conduct experiments on the FUNSD dataset. In addition to the standard color scheme (red, blue, yellow, and green), we conduct three additional experiments. First, we randomly swap the colors used in the standard color scheme. Second, we choose different shades intensities of the same color system for each label class. Third, we only draw the bounding box lines with colors. The results in Table reveal three rules followed by VANCL, which are consistent with the exist- ing findings in cognitive science (Gegenfurtner, 2003; Elliot and Maier, 2014): 1) different shades intensities of the same color system do not sig- nificantly affect the results; 2) swapping different colors in the current color scheme does not signifi- cantly affect the results; and 3) VANCL is effective when using colors with strong contrasts while being sensitive to grayscale.
5.8 Non-pretrained testing
To verify whether VANCL'S improvement is simply due to pretrained visual encoders containing real- world color priors, deliberately tested visual encoders that were pretrained and initialized their parameters when introduced into the model. In this way, the visual encoder would need to learn to extract visual features from seratch. Considering

---

Model
Enc.
#I
Pre.
FUNSD
SEABILL
formation from multiple modalities (Powalski al.,
STAND.
78.66
88.03
2021: Wang et al., 2022b). LayoutLM (Xu et al.,
2020) introduces 2D relative position information based on BERT (Devlin et al., 2019), which enables the model to perceive the 2D positions of text seg- ments throughout the document. LayoutLMv2 (Xu et al., 2021), StructTeext (Li et al., 2021b), Struct- Textv2 (Yu et al., 2023) and LayouLLMv3 (Huang et al., 2022) further integrate visual channel input into unified multimodal transformer framework to fusing textual, visual, and spatial features. Thus, these methods learn more robust multimodal rep- resentations and have made promising progress in visually rich document understanding tasks. Re- cently, Wang et al. (2022a) recently propose a bi- directional attention complementation mechanism (BiACM) to enable cross- moos-mmo interaction in- dependent of the inner language model. Lee et al. (2022) exploit serializing orders of the text segment output by OCR engines.
STAND. RESNET 101
< 80.20
88.68
STAND. RESNET
18
x
79.51
88,50
CNN
80.03
88.42
VANCL
RESNET
101
V
81.33
89.65
VANCL
RESNET
18
77.62
88.64
VANCE
CNN
80.60
88.97
Table 8: Effect of using a non-pretrained outer visual encoder LAYOUTLM (wfing). Pre. means pretrained. Enc. and #I. denote the outer visual encoder and the number of inside layers.
that deeper networks are usually more difficult to train in a short time, we choose a smaller ResNet- 18 and an extremely simple two-layer CNN (Lecun et al., 1998) as the outer visual encoders for our experiments. The results in Table show that even simple visual encoder networks, such as CNN-2 and ResNet-18, can still surpass the standard basc- line models, indicating that the original backbone models learn better visual features through visually- asymmetric consistency learning (VANcI.).
6.2 Consistency learning
In recent years, there have been significant advance- ments in consistency learning, which aims to re- duce variances across different model predictions. The core idea of consistency learning is to struct cross-view constraints that enhance consis- teney across different model predictions. Previ- ous research on consistency learning has focused on adversarial or cuni-supervised learning, which provides supervised signals that help models learn from unlabeled data (Miyato et al., 2019; Xie et al., 2020; Lowell et al., 2021 There is also inter- est in incorporating consistency mechanisms into supervised learning (Chen al., 2021a; Lowell et al., 2021b). Batra and Parikh (2017) propose cooperative learning, allowing multiple networks to learn the same semantic concept in different environments using different data sources, which is resistant to semantic drift. Zhang et al. (2018) propose a deep mutual learning strategy inspired by cooperative learning, which enables multiple networks to learn from each other.
6 Related Work
6.1 Visually-rich document understanding
To achieve the goal of understanding visually-rich documents, natural attempt is to enrich statc-of- the-art pretrained language representations by mak- ing full use of the information from multiple modal- ities including text, layout and visual features. Researchers have proposed various paradigms, such as text-based, grid-based, graph-band based, and transformer-based methods. Text-based methods, e.g., XLM-ROBERT (Conneau et al., 2020), In- foXLM (Chi et al., 2021), only consider the textual modality, rely the representation ability of large- scaled pretrained language models. Grid-based methods, e.g., Chargrid (Katti al., 2018), Bert- Grid (Denk and Reisswig, 2019), and ViBERTgrid (Lin et al., 2021), represent documents using a 2D feature map. allowing for the use of image seg- mentation and/or object detection models in com- puter vision (CV). GNN-based methods (Liu et al., 2019a; Tang et al., 2021) treat text segments in the document as nodes and model the associations among text segments using graph neural networks. Transformer-based methods leverage the latest multimodal pretrained models to learn better se- mantic representations for VFDs by capturing in-
Previous works enable cross moosl learning ei- ther using the classical knowledge distillation meth- ods (Hinton et al., 2015) ative adversarial networks (Ren and Xiong, 2021). Hinton et al. (2015)'s method starts with a large. powerful pretrained teacher network and performs one-way knowledge transfer to a simple student, while Ren and Xiong (2021)'s 's method attempts to
gener-

---

encourage the partial-multimdall flow strong as the full-multimodal flow. In contrast, we explore the potential of cross-modal learning and informa- tion transfer, and enhance the model's ability to information extraction by using visual clues. This allows the model to learn a priori knowledge of entity types, which is acquired before encounter- ing the specific task data. In terms of model structure, our model architecture is more similar to mutual learning (Zhang et al., 2018) or cooperative learning (Batra and Parikh, 2017).
Ethical Statements
The potential ethical implications of this work are discussed here. (1) Data: FUNSD and SROIE datasets are publicly available, accessible, and well- developed benchmarks for document understand- ing. (2) Privacy-relatedd The SEABILL dataset is collected from real-world business scenario. While the data provider has deleted sensitive infor- mation, such as personally identifiable information, the SEABILL dataset still contains some location names and company names. Given the above rea- son, we could not release this dataset. (3) Repro- ducibility: Our model is built on open-source mod- els and codes, and we will release the code and model checkpoints later reproduce the reported results. (4) Ethical issues: The proposed approach itself will not produce information that harms any- one. Therefore, we do not anticipate any ethical issues.
7 Conclusion
We present VANCL, a novel consistency learning approach to enhance layout-aware pretrained mul- timodal models for visually-rich form-like docu- ment information extraction. Experimental results show that VANCL successfully learns prior knowl- edge during the training phase, surpassing exist- ing state-of-thh-aan models on benchmark datasets of varying sizes and large-scale real-world form document dataset. VANCL exhibits superior perfor- mance against strong baselines with no increase in model size. We also provide recommendations for selecting color schemes to optimize performance.
References
Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Mammatha. 2021. Docformer: End-to-end transformer for document understanding. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 1003.
Limitations
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang. Xiaodong Liu, Yu Wang, Jianfeng Gao, Song- hao Piao, Ming Zhou, and Hsiao- Hsiao-Wuen Hon. 2020. UniLMv2: #seudo-masker language models for uni- fied language model pre-training. In Proceedings of the 37th International Conference Machine Learn- ing, volume 119 of Proceedings of Machine Learning Research, pages 642-652, PMLR.
The limitations of our work in this paper are the following points:
1. For the sake simplicity, we only experiment with the LayS Lay0utLMv2- backbones, we
BASE, and have not tested other backbones, e.g., Struc- Text and StrvcTexxTv.
Tanmay Batra and Devi Parikh. 2017. erative learning with visual attributes. 05ss/1005.5512.
Coop- CoRR,
2. We have not studied the effectiveness of asym- metric consistency learning in the training stage.
Jiaao Chen, Dinghan Shen, Weizhu Chen, and Diyi Yang. 2021a. HiddenCut: Simple data augmenta- tion for natural language understanding with better generalizability. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 4380-4390, Online, Association for Computational Linguistics.
Acknowledgements
This work was supported by National Natu- ral Science Foundation of China (Young Pro- gram: 62306173, General Program: 62176153), JSPS KAKENHI Program (JP23H03454), Shang- hai Sailing Program (21YF1413900), Shanghai Pujiang Program
Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jing- dong Wang. 2021b. Semi- semi-supervised semantic seg- mentation with cross pseudo supervision. In 2021 IEEF/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 2613-2622
Shanghai Mu- nicipal Science and Technology Major Project (2011HH2D7X00022), the Alibaba-AIR Program (22088682), and the Tencent AI Lab Fund (RBFR2023012).
Zewen Chi, Dong. Furu Wei, Nan Yang. Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao,

---

Heyan Huang, and Ming Zhou. 2021. InfoXLM: An information-theon theoretic framework for cross- Lonnsll language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- mun Lunguage Technologies, pages 3576-3586 On-
XYLayoutLM: Towards layout-aware multimodal networks for visually-ricd rich document understanding. In IEFF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 4573-4582. IEEE.
line. Association for Computational I. Linguistics.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov, 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. CoRR, abs/ htd//50330253...
Teakgyu Hong, DongHyun Kim, Mingi Ji, Wonseok Hwang, Dachyun Nam, and Sungrae Park. 2021. BROS: A pre-trained language model for understand- ing texts in document.
Ciaran Cooney, Rachel Heyburn, Liam Madigan, Mairead O'Cuinn, Chloe Thompson, and Joana Cavadas. 2023. Unimodal and multimodal repre- sentation training for relation extraction. In Artificial Intelligence and Cognitive Science, pages 450-461, Cham. Springer Nature Switzerland.
Yupan Huang. Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. LayoutLMv3: Pre-training for doc- ument ai with unified text and image masking, In Proceedings of the 30th ACM International Confer- ence on Multimedia, MM '22, page 4083-4091, New York, NY, USA. Association for Computing Machin- ery.
Timo I. Denk and Christian Reisswig. 2019. BERTgrid: Contextualized embedding for 2d document represen- tation and understanding. In Workshop on Ducument Intelligence at NeurlPS 2019.
Zheng Huang, Kai Chen, Jianhua He, Xiang Bai. Di- mosthenis Karatzas, Shijian Lu, and C. V. Jawahar. 2019. ICDAR2019 competition scanned receipt ocr and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1516-1520.
Jacob Devlin, Ming-Woo Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Guillaume Jaume, Hazim Kemal Ekenel, and Jean- Philippe Thiran. 2019, FUNSD: dataset for form understanding in noisy scanned documents. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 2. pages 1-6.
Andrew J. Elliot and Markus A. Maier. 2014. Color psy- chology: effects of perceiving color on psychological functioning in humans. Annual review of psychology, 65:95-120.
Anoop R Katti, Christian Reisswig, Cordula Guder, Se- bastian Brarda, Steffen Bickel, Johannes Höhne, and Jean Baptiste Faddoul. 2018 Chargrid: Towards un- derstanding 2D documents. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4459-4469, Brussels, Belgium. Association for Computational Linguistics.
Lukasz Garncarek, Rafal Powalski, Tomasz Stanistawek, Bartosz Topolski, Piotr Halama. Michat Turski, and Filip Gralinski. 2021. LAM- BERT: Layeut-aware language modeling for information extraction. In Document Analysis and Recognition - ICDAR 2021, pages 532-547, Cham. Springer International Publishing.
S. Kullback and R. A. Leibler. 1951. On information and sufficiency. The Annals of Mathematical Statis- tics, 22(1))79-86.
Karl R. Gegenfurtner. 2003. Cortical mechanisms of colour vision. Nature Reviews Neuroscience, ((7):563-572
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. 1998. Gradient- learning applied to document recog- nition. Proceedings of the IEEE, 86( 1 ():2782-2324.
Jiuxiang Gu, Jason Kuen, Vlad Morariu, Handong Zhao, Rajiv Jain, Nikolaos Barmpalios, Ani Nenkova, and Tong Sun. 2021. Unidoc: Unified pretraining framework for document understanding. Advances in Neural Information Processing Systems, 34:39-50.
Chen-Yu Yu Lee, Chun-Liang Li, Timothy Dozat, Vincent Perot, Guolong Su, Nan Hua. Joshua Ainslie, Ren- shen Wang. Yasuhisa Fujii, and Tomas Pfister. 2022. FormNet: Structural encoding beyond sequential modeling in form document information extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume
Zhangxuan Gu, Changhua Meng. Ke Wang, Jun Lan, Weiqiang Wang, Ming Gu, and Liqing Zhang. 2022.

---

Long Papers). pages 3735-3754, Dublin, Ireland. Association for Computational Linguistics.
data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 4992-5001, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu. 2021a. Selfdoc: Self-supervised doc- ument representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
Takeru Miyato, Shin-Ichi Macda, Masanori Koyama, and Shin Ishii. 2019. Virtual adversarial training: A regularization method for supervised and semi- supervised learning. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 41(8):1979 1993.
Yulin Li, Yuxi Qian. Yuechen Yu, Xiameng Qin, Chengquan Zhang, Yan Liu, Kun Yao, Junyu Han, Jingtuo Liu, and Errui Ding. 202 StrucTexx:: Structured text understanding with multi-modal trans- formers. In Proceedings of the 29th ACM Interna- lional Conference on Mullimedia, MM '21, page 1912-1920, New York, NY. USA. Association for Computing Machinery.
Rafat Powalski, Lukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michat Pietruszka, and Gabriela Palka. 2021. Going full-tilt boogie on document un- derstanding with text- image- transformer. In Document Analysis and Recognition-ICDAR 2021: 16th International Conference, Lausanne, Switzer- land, September 5-10, 2021, Proceedings, Purt II 16, pages 732-747. Springer.
Xiaobo Liang, Lijun Wu, Juntao Li, Yue Wang. Qi Meng, Tao Qin, Wei Chen, Min Zhang, and Tic- Yan Liu. 2021. R-drop: Regularized dropout for neural networks. In Advances in Neural Information Processing Systems, volume 34, pages 10890-10900 Curran Associates, Inc.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015. Faster R-CNN: towards real-time ob- ject detection with region proposal networks. In Ad- vances in Neural Information Processing Systems 28: Annual Conference on Neural Information Process- ing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 91-99.
J. Lin. 1991. Divergence measures based on the shan- non entropy. IEEE Transactions Information The-- ory,
Weihong Lin, Qifang Gao, Lei Sun, Zhuoyao Zhong, Kai Hu, Qin Ren, and Qiang Huo. 2021. ViBERT- grid: A jointly trained multi-modal 2d document representation for key information extraction from documents. In Document Analysis and Recognition ICDAR 2021, pages 548-563, Cham. Springer Inter- national Publishing.
Yuqi Ren and Deyi Xiong, 2021. CogAlign: Learning to align textual neural representations to cognitive lan- guage processing signals. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume I: Long Papers). pages 3758 3769, Online. Association for Computational Linguistics.
Xiaojing Liu, Feiyu Gao, Qiong Zhang, and Huasha Zhao. 2019a. Graph convolution for multimodal in- formation extraction from visually rich documents. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Industry Papers). pages 32-39, Minneapo- lis, Minnesota. Association for Computational Lin- guistics.
Guozhi Tang, Lele Xie, lianwen Jin, Jiapeng Wang. Jingdong Chen, Zhen Xu, Qianying Wang, Yaqiang Wu, and Hui Li. 2021. MatchVIE: Exploiting match relevancy between entities for visual information ex- traction. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 1039-1045, International Joint Conferences on Artificial Intelligence Organization. Main Track.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A robustly optimized bert pretraining approach.
Jiapeng Wang, Lianwen Jin, and Kai Ding. 2022a. LiLT: A simple yet effective language-indepente layout transformer for structured document understanding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 7747-7757, Dublin, Ireland. Association for Computational Linguistics.
David Lowell, Brian Howard, Zachary C. Lipton, and Byron Wallace. 2021a. Unsupervised data augmenta- tion with naive augmentation and without unlabeled data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4992-5001, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Zilong Wang. Jiuxiang Gu, Chris Tensmeyer, Nikolaos Barmpalios, Ani Nenkova, Tong Sun, Jingbo Shang, and Vlad I Morariu. 2022b. MGDoc: Pre-training with multi graaulg hierarchy for document image understanding. arXiv preprint arXiv:2211. 144455
David Lowell, Brian Howard, Zachary C. Lipton, and Byron Wallace. 2021b. Unsupervised data augmenta- tion with naive augmentation and without unlabeled
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoe Le. 2020. Unsupervised data augmenta- tion for consistency training. In Advances in Newral

---

Information Processing Systems, volume 33, pages 6256-622 6268. Curran Associates, Inc.
A
Appendix
A.1
Dataset Details
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu
Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. 2021. Lay2 Multi-modal pre-training for visually-rich document understanding. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1ith International Natural Language Processing (Volume I: Long Papers). pages 2579-2591, Online. Association for Computational I Linguistics.
FUNSD This dataset contains 199 well- scanned forms. Each semantic entity consists of unique identifier id, a label (1 ([Question, Answer, Header, or Other)), a bounding box, list of links to other entities, and a list of words.
Joint Conference
SROIE The dataset contains 626 receipts for train- ing and 347 receipts for testing. Each receipt con- sists of its scanned image and OCR transcription, organized as list of text segments with bounding box position information pairs. Each receipt is la- beled with four types of entities, namely (Company, Date, Address, Total).
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020. LayoutL.M: M: Pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD Interna- tional Conference
Knowledge Discovery & Data Mining. KDD '20. page 1192-1200, New York, NY. USA. Association for Computing Machinery.
SEABILL The dataset a complex collection of documents derived from maritime business scenar- ios and consists of 3,562 training documents and 953 test documents. The data consists of PDF im- ages and rule-based transcriptions of the documents with three labels (Question, Answer, Other).
Yuechen Yu, Yulin Li, Chengquan Zhang, Xiao- qiang Zhang. Zengyuan Guo, Xiameng Qin, Kun Yao, Junyu Han, Errui Ding, and Jingdong Wang- 2023. StrucTexTT2: Masked visual-textual pre- diction for document image pre-training. CoRR, abs/2303.00289.
A.2 Effect of sharing weights
Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. 2018. Deep mutual learning. In Pro- ceedings of the IEEE Conference Computer Vision and Pattrnn Recognition (CVPR), pages 4320-4328.
In this section, we design experiments to inves- tigate the effect sharing weight. As shown Table 9, we conduct experiments on three datasets by controlling the variables of the inputs and model weight sharing.
Visual
Prompt
FUNSD
SROIE
SEABILL
77.47
95.13
88.63
-shared
80.03
95.84
89.13
x
80.38
96.02
89.13
shared
81.33
96.50
89.65
Table Effect of sharing weights of the network pa- rameters between the inner and outer visual encoders in LAYOUTLM (w/ img) on model Fl score. Sharing the weights of the model brings significantly better results than not sharing the weights.
It can be seen that sharing the parameters of the visual encoder is better than not sharing the weights in most cases (see Table 9), and the way of sharing parameters also greatly reduces the overall number of parameters of the model (see Table 2). In addition, we also compare the results whether using the colorful prompt paints under the settings of weight sharing or not. Even under the condition that model weights are not shared, we observe that colored visual clues still significantly improve the performance of the model.

---

X
19000 Naundaan Sana
14 11
91.45
11 M
18
6 14
18 2:
10
Dand
69:82
16.00
BRANCHAL.CHI
KELANNG,AALASIA
16 1 :l
1.
88
#RT
#ORT A0RS
HET
PLASTIC CLOSURE
CLOSURE
(a) Vanilla LAYOUTI. (ww/w///////// img) (left) vs. V (right)
MON
PROM:
QINEI METII PRIDECT CELERN
Pastectn COoH
Eilkee Nonespe- und bereet gungetacheic GmbHi
XTCOEEGUE GLEE
Vanilla LV3 (left) vs. VANCI_LAVOUTLMVV3 (right)
Figure 5: Case study of the predicted results on the SEABILL dataset using the standard fine-tuning pipeline LAYOUTLM(W img) (a) and LAYOUTLMY3 against the VANCL pipeline.
A.3 Case study
also gives a wrong label "Question", whereas VANCI[LAYOUTLMV3]] gives the right label.
Figure 5(a) and Figure 5(b) visualize the cor- responding prediction results using the vanilla LAYOUTLM(w/ img) and LAYOUTLMV3 mod- els against VANCI[[LYOOULLM img)] and VANCI|LAYOUTLMV3] models on the SEABILL dataset, respectively. In Figure 5(a), LAY- OUTLM(w/ img) predicts wrong label "Other" for "PLASTIC CLOSURE", * T. (Loading way)", tb (Detailed address)", "23tE (23 GP)", while VANCI|LAYOUTLLME img)] can make correct predictions that match the human- annotated ground truths. There are ambiguous cases that VANCI[LAYOUTLM gives wrong predictions. For example, both LAYOUTLM(w/ img) and VANCI_LAYOUULMM(. img)] predict "By sea (23 GP)" as "Other", but the true label is "Answer".
A.4 Visualization
The full version of the t-SNE visualization is shown in Figure 6. The upper left and upper right graphs show a comparison of the visual encodings of the two flows before and after VANCL training, respec- tively. After training, the results show that each classification cluster is more separated and the cor- responding information of the two flows is more aligned. The lower left and lower right graphs show a comparison of the multimodal output of the two flows before and after VANCL training, respec- tively. In summary, VANCI. effectively transfers label information to the standard flow through the vision vision-enhanen enhanced flow.
img)] also
Figure 5(b) shows that LAYOUTLMV3 gives more accurate predictions than LAYOUTLM(w) img). However, LAYOUTLMV3 also predicts the entities with the wrong label "Other" for " # A (Goods description)", "2176.000M: "17PAL- LETS", "I "1394486"" though these entities are an- notated as "Question" or "Answer" in the train- ing data. For "KGS", "CBM", LAY

---

vicual ancodings betare VANCL training
visual encad after VANCL training
1.0
Vilon-enhanced
0.5
0.8
0.8
0.4
0.4
0.0
0.0
0.4
0.8
0.2
0.6
1.0
encodings VANCL training
multimodal encodings training
1.0
Standard
1.0
Standard
Weln-anaanced
Visio--er-anced
0.2
0.2
0.0
0.0
0.0
0.4
0.8
0.2
0.6
1.0
Figure 6: The full t-SNE visualization result of hidden states from the outputs of the standard and vision-enhanced flows. The uppers are visual encodings before and after VANCL training. The lowers are multimodal fused encodings before and after VANCL training. The bounding boxes indicate clusters shown Section 5.5.


Reviews:
[{"review_id": "L80X72eRxC", "reviewer": "Reviewer_SYbT", "summary": "", "questions": "Question A: The method acronym is a bit awkward, why not simply name it VACL?\n\nQuestion B: The color painting step is not sufficiently explained and requires some guesswork on the part of the reader. Referred to by different terms throughout the paper: color priors, synthetic painted images, colorful prompt paints, reverse aligning -- some of which are never mentioned anywhere else in the paper.\n\nQuestion C: The overall architecture diagram is beautiful but is somewhat confusing. Especially separating the encoder in box 2 from LMMs in box 3, which in my understanding are all part of the same LMM. The detach relations are also hard to follow. It appears as the robust outer encoder is not training? Yet, why color assigning and reverse aligning steps are not detached?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer’s valuable feedback!\n\n***Response-Q1: \"On the conceptual level.\"***\n\nAlthough the method is simple, we have a strong motivation to pursue this approach. Firstly, the idea is inspired by the color bar stimuli used in human visual attention experiments in cognitive science. We wanted to investigate whether similar color cues would impact the attention of multimodal AI-based machine document understanding models, just as the cognitive experiment that prompts participants to focus on a specific area. Of course, the final experimental conclusion revealed that the results deviated somewhat from our initial expectations. The experiment demonstrated that color did not have a particularly significant impact (for instance, merely altering the grayscale could also produce some effect), but rather the incorporation of entity-level label prior information would enable the machine to learn more robustly.\n\n\n***Response-Q2:explanation for performance improvement without color patches*** \n\nThis approach to consistency learning we propose is well able to transfer entity-level priors to the standard model. As illustrated in the color assignment process in Figure 2, the allocation of color patches is grounded in truth, signifying that the input image within the vision-enhanced flow encompasses the mapping relationship between the entity and the label that the standard model is required to learn. Fundamentally, as elaborated in Appendix A.4, we acquire this mapping by converging the multimodal output of the standard model with the vision-enhanced flow through the application of consistency learning. Consequently, even in the absence of color patch augmentation in the input images, the standard model can still effectively learn the representative features of entities and relative relationship between entities.\n\n\n***Response-Q3/A: acronym*** \n\nAlthough VACL is an impressive name and shorter than VANCL, VANCL possesses a straightforward and clear meaning. Therefore, we suggest that VANCL is a more suitable title for this work.\n\n***Response-Q4/B: Terminology issue*** \n\nRegarding the terminology issue, we appreciate your suggestion and will further optimize certain descriptions to enhance the clarity and comprehensibility of the article.\n\nIn order to adhere to the 8-page limit, some content might not have been elaborated upon sufficiently. We kindly request the chance of revision to provide a more detailed explanation of the color correspondence and the relationship between terminologies.\n\n***Response-Q5: the overall architecture diagram is relatively abstract, need some guesswork*** \n\nWe agree that the overall architecture diagram is somewhat abstract, which may require a certain degree of interpretation. In response to the feedback received during the previous round of ACL review, we have simplified the illustration as suggested. As a result, we have endeavored to depict the core concept, despite the fact that the implementation entails more intricate details when dealing with LayoutLM, LayoutLMv2, and LayoutLMv3, each of which possesses distinct architectures.\n\nThe robust outer encoder is indeed involved in the training process.\n\nIn the case of LayoutLM, the encoder and the main LMM-transformers are separate components. However, for LayoutLMv2/v3, the LMMs encompass the encoder, which is accurate.\n\nRegarding the detach relations, it is essential to understand that we detach the additional vision-enhanced flow. Consequently, there are no color assignment or reverse alignment steps following the detachment.\n \n\n***Response-Q6: Typos*** \n\nThank you. We will correct them!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper introduces a novel asymmetric consistency training scheme that enhances visual representations learned by multimodal encoders and is empirically shown to outperform a comprehensive set of baseline models.\n\nThe authors back the quantitative results with thorough and insightful follow up experiments looking at competing training schemes, visualizing learned embeddings, settings with low data volume (in which the approach especially shines) and without pre-training.\n\nThe novel learning scheme can be adapted to an existing LMM backbone.\n\nThe paper is generally well written and uses visualizations effectively.", "reasons_to_reject": "On the conceptual level: The authors claim to bridge the gap between visual feature extraction and entity detection. As evidence that standard visual encoder is limited, authors cite (Cooney et al., 2023) but it merely asserts text has highest predictive power compared to layout geometry but doesn't specifically indicate models lacks capacity for leveraging rich visual features. They additionally argue LayoutLM receives already preprocessed text tokens and are not trained to detect fine grained features such as ones needed for entity detection. Yet, their own addition merely augments images with color patches corresponding to entities, which one could conceivably argue isn't particularly fine-grained either.\n\nIt is not clearly articulated why performance improvement would carry over when input images are not augmented with color patches at test time.", "questions_to_authors": "Question A: The method acronym is a bit awkward, why not simply name it VACL?\n\nQuestion B: The color painting step is not sufficiently explained and requires some guesswork on the part of the reader. Referred to by different terms throughout the paper: color priors, synthetic painted images, colorful prompt paints, reverse aligning -- some of which are never mentioned anywhere else in the paper.\n\nQuestion C: The overall architecture diagram is beautiful but is somewhat confusing. Especially separating the encoder in box 2 from LMMs in box 3, which in my understanding are all part of the same LMM. The detach relations are also hard to follow. It appears as the robust outer encoder is not training? Yet, why color assigning and reverse aligning steps are not detached?", "paper_topic_and_main_contributions": "The paper tackles the problem of multimodal information extraction, specifically entity recognition from visually rich form-like documents.\n\nThe authors argue that existing multimodal extraction systems like LayoutLM (referred to as large scale pretrained multimodal models, LMMs) are limited in leveraging fine grained visual features, and propose a novel objective called visually asymmetric consistency learning (VANCL) that can be used with any multimodal encoder backbone. \n\nThey introduce a novel model training scheme that proceeds along two distinct flows: \n- standard, utilizing coarse visual encoder of LMM on standard textual, spatial, and image inputs; and \n- visually-enhanced, comprised of an external visual encoder that accepts \"augmented\" images where entity's bounding boxes are colored in accordance with entity type. \n\nThe resulting encodings are then passed independently through the rest of LMM (if I understand correctly), producing two independent predictions for the span's entity labels, and the final tagging is obtained via Viterbi decoding over predicted probability distributions. \nStandard supervised loss against the gold labels is combined with the consistency loss between the two representations. \n\nGiven the asymmetry in the inputs to the two learning flows, the authors hope to refine LMM encoder to capture finer visual features, which is evidenced by the method consistently outperforming standard LayoutLM of different versions and other baselines on FUNSD, SROIE and SEABILL datasets. \n\nThe visually-enhanced training flow can be detached during inference.\n\nAdditionally, the authors conducted several follow up studies, including \n- empirical comparisons between different divergence measures and related training schemes: mutual learning and R-drop,\n- visualization of learned representations that demonstrates stronger coupling between visual and textual embeddings,\n- evaluation of the method in the low-resource and un-pretrained settings, all of which demonstrate consistent improvement over baseline.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typo:\n084 et al., 2021b), we propose a novel vision-enhance\n\nClarity:\n097 The inputs are asym-\n098 metric in the visual modality, with the original doc-\n099 ument images input to the standard flow, while im-\n100 ages to the vision-enhanced flow are the synthetic\n101 painted images. \n\nIt is 150 \nworth noting that the input images for the vision- 151 \nenhanced flow are colorful prompt paints,  152\n\n170 where PY\n171 tributions output by the standard learning flow and\n172 the extra vision-enhanced flow, which are the la-\n173 tent outputs after softmax normalization (i.e., soft\n174 label)."}}, {"review_id": "g5x1yQ7k9a", "reviewer": "Reviewer_uW1H", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments!\n\n***Response-Q1:\"Why the proposed method utilize color information more effectively .\"*** \n\nColors are primarily incorporated into the fine-tuning process as a prior knowledge of entity-level labels, as described in Section 3.2. In fact, color information simply serves as a mapping of the label for the vision channel.   \n\nIn summary, the primary objective of this paper is to enhance the multimodal output of the standard flow model by reducing the divegence between them. We utilize color information as a means to improve the model by striving to achieve consistency between the output of the standard model and the output of the visual enhancement through consistency learning.\n\n\n\n\n***Response-Q2: Ablation Experiments*** \n \nOur model primarily contains on two modules: color painting and consistency learning. The ablation experiment for color painting involves ablating the color, which includes scenarios such as no color, different colors, and varying shades of colors. Please refer to Section 5.5 for more details on this ablation experiment.\n\nAs for a comprehensive evaluation of each module, we have conducted the additional ablation experiment by examining the impact of using consistency loss. The experimental results reveal that removing the consistency loss leads to a decline in the model's performance to varying extents. This finding demonstrates the significance of consistency loss in the model. Simultaneously, it indicates that without the driving force of consistency, the addition of color information may actually increase the learning noise.\n\n\n| Model                           | FUNSD（F1-score）             | SROIE （F1-score）      | SEABILL （F1-score）    |\n| ------------------------------- | ---------------- | ---------------- | ---------------- |\n| LayoutLM                        | 80.20            | 95.80            | 88.68            |\n| VANCL [LayoutLM]                | 81.33            | 96.50            | 89.65            |\n| VANCL [LayoutLM] -consis_loss   | **79.36(-1.97)** | **95.68(-0.82)** | **88.19(-1.46)** |\n| -                               | -                | -                | -                |\n| LayoutLMv2                      | 82.76            | 96.25            | 91.31            |\n| VANCL [LayoutLMv2]              | 83.12            | 97.50            | 91.89            |\n| VANCL [LayoutLMv2] -consis_loss | **81.52(-1.60)** | **97.08(-0.42)** | **89.28(-2.61)** |\n| -                               | -                | -                | -                |\n| LayoutLMv3                      | 90.29            | 96.77            | 90.64             |\n| VANCL [LayoutLMv3]              | 92.35            | 97.20            | 91.35            |\n| VANCL [LayoutLMv3] -consis_loss | **90.93(-1.42)** | **96.98(-0.22)** | **90.87(-0.48)** |\n\n\nFurthermore, we kindly suggest reviewing Figure 4 and Appendix A.4 for a more comprehensive understanding of our research. These sections primarily demonstrate that, by employing consistency learning, the multimodal output of the standard flow tends to lean towards the vision-enhanced flow. This, in turn, facilitates the learning of the mapping relationship between the entity and its corresponding ground truth label.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation behind this article is highly reasonable, as colors evidently have a positive impact on human recognition of table content in reality.\n2. The model design is sound.\n3. We observe a significant increase in performance.", "reasons_to_reject": "1. The authors should provide more analysis on why the method proposed in this article can more effectively utilize color information.\n2. The ablation experiments are insufficient; it is strongly recommended that the authors conduct a comprehensive evaluation of each module.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a model for enhancing visual encoding, which demonstrates remarkable performance in table recognition.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "q5pMbZFkkl", "reviewer": "Reviewer_XvBS", "summary": "", "questions": "Does all LMMs have  weaker representational capabilities in visual encoders? \nI cannot understand Figure 4. Can you help explain why after training, the representation is vision enhanced?\nCan the authors give some explanations on of JS and KL in consistency loss?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback!\n\n***Response-Q1: \"Previous LMMs have weaker representational capabilities ... no evidence to show this.*** \n\nSorry for the confusion. \nWe want to clarify that the intended meaning of this statement is that the visual encoders in earlier LMMs are somewhat less effective than the most recent OCR engines. This is due to the absence of task-specific visual features in those encoders, such as text segment detection and bounding box regression. \n\nIn reality, we do not explicitly state that the visual encoders of previous LMMs are weak; rather, we emphasize that their visual representations in VFD-specific domains are biased. General visual representation alone is insufficient for VFD information extraction. In this work, we enforce the model to extract task-specific visual features.\n\n***Response-Q2: SOTA and other backbones***\n\nIn fact, we believe that we have achieved state-of-the-art (SOTA) results. The fact has been confirmed by all the previous round of ACL reviewers. However, we recently found FormNetV2 (ACL2023), a concurrent work, but comparing with our method, it only achieves better scores on the SRIOE dataset.\n\nTo our knowledge, FormNetV1 does not effectively utilize visual features, while LILT primarily addresses cross-language issues. They are not suitable to be a backbone model. However, it might be worthwhile to explore extensions to FormNetV2 in the future – a promising suggestion.\n\nWe would like to caution reviewers to concentrate more on the problem our paper addresses rather than solely focusing on the resulting SOTA.\n\n***Response-Q3:why after training, the representation is vision enhanced?*** \nAs previously elucidated, the visual representation within the specific domain of VFD is relatively weak, and the general visual representation does not sufficiently fulfill the requirements for information extraction in the context of VFD.\n\nIn reference to Figure 4 and Appendix A.4, the intention is not to directly augment the visual representation. Rather, through the implementation of consistency learning, the standard visual representation and the vision-enhanced flow representation become more congruent, ultimately rendering the multimodal representation more efficacious. Given that the visual representation of the vision-enhanced flow is robust, it can be inferred that if the standard flow representation converges towards this vision-enhanced representation, the standard flow representation can also be considered enhanced.\n\n***Response-Q4:KL and JS?***\nSee line 193-194 and Equation.4, KL divergence is used to measure the degree of difference between two probability distributions, the larger the KL divergence, the greater the difference between the two; when the KL divergence is smaller, it means that the difference between the two is smaller. The difference between JS divergence and KL divergence is that KL divergence is asymmetric, while JS divergence is symmetric.\n\n***Response-Q5: Presentation Improvement: \"... change the color of Figure2. ...hard to read.\"***\n\nThank you very much for your feedback. \nWe will change the color. If there are any areas that are difficult to read or require further clarification, please do not hesitate to point them out, and we will try to improve the presentation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The results are impressive without extra cost in inference. The paper potentially provides a general way to improve complex document understanding. \n2. The findings are consistent with psychology theory, and the color schemes are useful to future research in complex document understanding. \n3. Overall, the presentation is clear and easy to understand.", "reasons_to_reject": "1. The authors point out that visual encoders in previous LMMs have weaker representational capabilities in L72-74. It's not convincing as there is no evidence to show this. \n2. The results cannot achieve state-of-the-art results with LayoutLM. I would suggest authors producing more results with LILT and FORMNET backbones.", "questions_to_authors": "Does all LMMs have  weaker representational capabilities in visual encoders? \nI cannot understand Figure 4. Can you help explain why after training, the representation is vision enhanced?\nCan the authors give some explanations on of JS and KL in consistency loss?", "paper_topic_and_main_contributions": "This paper proposes VANCL to improve visual encoders of LMM. Specifically, it utilizes asymmetric consistency learning to enhance the visual representations in the visual encoder. With no extra cost during inference time, VANCL could achieve better results then without the visual enhanced flow.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The authors could change the color of Figure2. It is a bit hard to read."}, "extra_text": {}}, {"review_id": "4YKK93sYSy", "reviewer": "Reviewer_XaQk", "summary": "", "questions": "(A) I would like to have a discussion on the effectiveness of consistency loss. \n(B) It would be nice to have some discussions on the (not necessarily statistic) significance of the performance gain.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your suggestion for an ablation experiment!\n\n***Response-Q1: Ablation study over the consistency loss***\n\nAs the reviewers' suggested, we have now included an ablation study to investigate the impact of utilizing consistency loss. We believe now it is sufficient to support the initial motivation.\n\nThe experimental results are presented below:\n\n\n| Model                           | FUNSD（F1-score）        | SROIE （F1-score）       | SEABILL  （F1-score）    |\n| ------------------------------- | ------------ | ------------ | ------------ |\n| LayoutLM                        | 80.20        | 95.80        | 88.68        |\n| VANCL [LayoutLM]                | 81.33        | 96.50        | 89.65        |\n| VANCL [LayoutLM] -consis_loss   | **79.36(-1.97)** | **95.68(-0.82)** | **88.19(-1.46)** |\n| -                               | -            | -            | -            |\n| LayoutLMv2                      | 82.76        | 96.25        | 91.31        |\n| VANCL [LayoutLMv2]              | 83.12        | 97.50        | 91.89        |\n| VANCL [LayoutLMv2] -consis_loss | **81.52(-1.60)** | **97.08(-0.42)** | **89.28(-2.61)** |\n| -                               | -                | -                | -                |\n| LayoutLMv3                      | 90.29            | 96.77            | 90.64             |\n| VANCL [LayoutLMv3]              | 92.35            | 97.20            | 91.35            |\n| VANCL [LayoutLMv3] -consis_loss | **90.93(-1.42)** | **96.98(-0.22)** | **90.87(-0.48)** |\n\nThe performance of the model has experienced varying degrees of decline after removing the consistency loss. On one hand, this highlights the significance of consistency loss. On the other hand, it demonstrates that without the consistency loss, the additional color information actually introduces learning noise. This implies that merely incorporating colors is not sufficient; the process of consistency learning is essential. This finding further supports the conclusion drawn in Section 5.5 from another perspective: the color itself is not our primary concern.\n\n***Response-Q2: Some discussions on the significance of the performance gain***\n\nWhile we do not explicitly present statistical significance, we give case study and visualization to better support the performance gain, which offers a reliable indication of the validity of our findings.\n\nCase Study: Appendix A.3 provides several examples for comparing results, and we can offer additional ones if necessary.\n\nVisualization: Refer to Figure 4 and Appendix A.4. Through consistency learning, the visual representation of the standard flow becomes more similar to that of the vision-enhanced flow, ultimately making the multi-modal representation more effective. Since we know that the vision-enhanced flow representation is robust, we can also consider the standard flow representation to be enhanced when it aligns with the vision-enhanced representation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and the method is well-motivated. Although the performance gain by the method looks marginal, I think the idea is interesting. Also, the paper provides a good set of comparisons and ablation studies (though some aspects are missing).", "reasons_to_reject": "As far as I understood, the ablation study over the consistency loss (whether it’s used or not) is not provided. This point is the main drawback of the paper.\n\nThe performance gain is marginal. Together with the above point, this makes the evaluation of the method difficult.", "questions_to_authors": "(A) I would like to have a discussion on the effectiveness of consistency loss. \n(B) It would be nice to have some discussions on the (not necessarily statistic) significance of the performance gain.", "paper_topic_and_main_contributions": "This paper presents a method for semantic entity recognition, in which some entities in a document (e.g., headers, questions, and answers) are identified. The main idea of the paper is to facilitate training a model for this task with additional visual input that encodes the semantic entity types using colors. The paper also proposes a training scheme with this additional visual input.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors address the challenge of extracting information from documents containing visual and textual elements. They focus specifically on the task of identifying entities in form-like documents that have a rich visual layout. The authors argue that existing multimodal information extraction systems, such as LayoutLM, have limitations in effectively utilizing fine-grained visual features. To overcome this limitation, they propose a new objective named visually asymmetric consistency learning (VANCL), which can be used in conjunction with any multimodal encoder backbone. The experimental results have shown the effectiveness of the proposed method. Although some reviewers have raised some concerns about some statements in the experiment, all of them generally agree with the effectiveness of the proposed method.
Decision: Accept-Main