LayoutDIT: Layout-Aware End-to-End Document Image Translation with Multi-Step Conductive Decoder
Authors: Zhiyang Zhang
Yaping Zhang
Yupu Liang
Lu Xiang
Yang Zhao
Yu Zhou
Chengqing Zong
Venue: EMNLP 2023 Findings (2023)
Keywords: document image
machine translation
layout
multi-step
conductive
end-to-end
TL;DR: The proposed framework effectively incorporates layout knowledge into DIT in an end-to-end way.

Abstract: Document image translation (DIT) aims to translate text embedded in images from one language to another. It is a challenging task that needs to understand visual layout with text semantics simultaneously. However, existing methods struggle to capture the crucial visual layout in real-world complex document images. In this work, we make the first attempt to incorporate layout knowledge into DIT in an end-to-end way. Specifically, we propose a novel Layout-aware end-to-end Document Image Translation (LayoutDIT) with multi-step conductive decoder. A layout-aware encoder is first introduced to model visual layout relations with raw OCR results. Then a novel multi-step conductive decoder is unified with hidden states conduction across three step-decoders to achieve the document translation step by step. Benefiting from the layout-aware end-to-end joint training, our LayoutDIT outperforms state-of-the-art methods with better parameter efficiency. Besides, we create a new multi-domain document image translation dataset to validate the model’s generalization. Extensive experiments show that LayoutDIT has a good generalization in diverse and complex layout scenes.

LayoutDIT: Layout-Aware End-to-End Document Image Translation with Multi-Step Conductive Decoder
Zhiyang Zhang!? Yaping Zhang!?, Yupu Liang', Lu Xiang!
Yang Zhao! Yu Zhoul Chengqing Zongl.
State Key Laboratory Multimodal Artificial Intelligence Systems (MAIS),
Institute of Automation, Chinese Academy of Sciences, Beijing, China
2 School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China Fanyu Al Laboratory, Zhongke Fanyu Technology Co., Ltd, Beijing, China
zhangzhiyang222208a..o cn, Saaping.zhang lu. xiang, yang. zhao, yzhou, cazong Jénlpr ia ac.
Abstract
Miwing Labnee
Baldha
st5te .910 X eite
Bloks
Better Sucineta
From Duester
Document image translation (DIT) aims to translate text embedded in images from one language to another. that needs to understand visual layout with text semantics simultaneously. However, ex- isting methods struggle to capture the crucial visual layout in real-world complex document images. In this work, we make the first attempt incorporate layout knowledge into DIT in end-to-end way. Specifically, we propose a novel Layout-an aware end-to-end Document Image Translation (LayoutDIT) with multi- step conductive decoder. A layout-aware coder is first introduced to model visual lay- out relations with raw OCR results, Then novel multi-step conductive decoder is unified with hidden states conduction across three step- decoders achieve the document translation step by step. Benefiting from the layout-aware end-to-end joint training, our Lay out- performs state-of-the-arr methods with better parameter efficiency. Besides, we create new multi-domain document image translation dataset to validate the model's 's generalization. Extensive experiments show that LayoutDIT has good generalization diverse and com- plex layout scenes.
Sulinn Paacn Telett
Koree Suidding
Bocks
a challenging task
Eromuuesee
From Quester
Beller Duslness
Camvel
Bullding
Bocks
To Bettr Businees
From Queeler
Decument Image
Rau OCR Results
OCR with Visual Legeut
Translation Recalts
The role of layout in DIT. Words the document image are arranged with certain visual layouts critical for translation. Either missing or error layout would negatively affect the subsequent translation.
Figure I:
However, existing works struggle to capture the visual layout with logical reading order docu- ment images (Sable al., 2023; Long et al., 2022; Hinami et al., 2021; Afli and Way, 2016; Chen et al., 2015; Du et al., 2011). As shown in Fig- ure 2, given sequence of word pieces extracted by Optical Character Recognition (OCR), existing cascade methods (Hinami et al., 2021) get the ratio- nal translation through four separate processes:(I) layout parsing, (2) logical order detection, (3) sen- tence segmentation, and (4) translation. They handled as isolated individuals and optimized inde- pendently, which leads severe error propagation. Another problem that it ignores the semantic correlation shared different modules.
1 Introduction
Document image translation (DIT) aims to trans- late text embedded in scanned documents from one language to another. Automated DIT is crucial research area for business and academic values. Different from plain text translation, texts doc- ument images are arranged with certain layouts. It requires the model to understand the visual lay- outs and text semantics simultancously. Figure showcases visual layout plays vital role DIT. Missing or error layout information could result in subsequent translation failure.
To address the above problems, in this work, we propose a unified end-to-end trainable Layout- aware Document Image Translation (LayoutDIT) network to map a document image processed by OCR to its document translation. LayouDDIT could simultaneously achieve layout understanding, read- ing order detection, sentence segmentation, and translation, and share computation and semantic information among these complementary tasks. Specifically, LayoutDIT comprises a layout-aware encoding phase and a multi-step conductive decod- ing phase. The layout-aware encoding phase aims
"Corresponding author.

---

(o) Existing Method
conceptual conciseness and better parameter efficiency.
Rectvard F'mm
Lagout
Lopical Onder Detector
Sertence
Trundater
Dicunctt Trnnerrion
We devise novel layout-aware encoder and multi-step conductive decoder enable lay- awareness and sub-module interactions si- multaneously, which could share computation among four complementary tasks, including layout parsing, reading order detection, sen- tence segmentation, and translation.
F.F. LayoutDIT (Preponn))
Layout-Anare
Multi-Srep Condacthe
Documett Transation
Decoder
DIT methods overview. Given document image with words and word bounding boxes extracted by OCR, DIT aims to generate the document-level trans- lation in the target language. Different from the tradi- tional cascade framework, LayoutDIT goes beyond with layout-aware end-to-end modeling, showing conceptual conciseness and model compactness.
Figure 2:
new multi-domain DIT dataset named DI- Trans is created to evaluate LayoutDIT. Ex- tensive experiments' and analysis on public benchmark and our dataset show LayouDDIT's better overall performances and superior capa- bility, especially for low-resource DIT.
to model document image's 's texts with layouts by jointly embedding text tokens and their 2D lay- outs. The multi-step conductive decoding phase is to capture the reading logic and to get rational translation results. The decoding phase identifies chain of three decoding sub prob-peecess:: reading order decoding, sentence boundary decoding, and translation decoding. It decodes intermediate in- ferences for each sub-process to approach the final document translation step by step. Each interme- diate decoding step is fulfilled dedicatedly with a step-decoder. All step-decoders are unified into a multi-step decoder with the conduction of hid- den states across them, thus enabling LayouDDIT's sub-module interaction and end-to-end training.
Layout-aware Document Image Translation
In this section, we describe in detail our model ar- chitecture, as seen in Figure 3. LayoutDIT consists of a layout-aware encoding phase and a multi-step conductive decoding phase. The former employs layout-aware encoder for document image under- standing by producing its layout-aware represen- tations; the latter generates the document transla- tion following a -odnd---ndcooueer" philosophy. Concretely, a multi-step conductive decoder is pro- posed to divide the decoding phase into a chain three sub-processes and conquers them with dedi- cated step-decoders, one by one:
We evaluate LayouDDIT's strong ability on the public ReadingBank (Wang et al., 2021) for general-domain document image translation. In multi-domain Docu- ment Image Translation dataset (DITrans), which covers three different document domains to test LayoutDIT's domain-specific capacity. Experi- mental results show that the proposed LayoutDIT can achieve state-of-the-art translation performance with better parameter efficiency. Further analysis demonstrates LayoutDIT's great power in cross- domain adaptation and few-shot learning, indicat- ing its superiority in these practical DIT scenarios. The contributions of this work summarized as follows:
Decoding Step 1 (Reading order decoding). Given an OCR document image with out-of- order words and word bounding boxes, a re- ordering step-decoder decodes each word's reading order index based on the layout- representations.
addition, create
Decoding Step 2 (Sentence boundary decod- ing). Given an in-order word sequence, seg- mentation step-decoder decodes each word's sentence boundary tag find the begging/end of each sentence.
Decoding Step 3 (Translation decoding). A translation decoder decodes the target lan- guage translation for each sentence.
We propose layout-aware end-to-end doc- ument translation framework named Layout- DIT. To the best of our knowledge, this the first work that incorporates layout information into DIT an end-to-end way, which shows
The final document translation can be obtained by concatenating all sentences' translations.
'Code will be released at http://gihuhh.com/hanggiyyygg 2020/LayoutDIT.

---

Decoding Steps
Target Sent,
Target Sent,
Target Senty
Step Reading Order Decoding
Step Scot. Boundary Decoding
Translation Step-Decoder
Translation Step-Decoder
Translation Step-Decoder
Step Translation Decoding
Sent. Boundary Classes
H
H
Hys
Sentences Context Memory
BOS Begin of Sentence
IOS Inside of Sentence
R Sentennce Segmentation Rule
ldx
Idx;
Idx.
Idx,
Decoder Hidden States
BOS
IOS
BOS
IOS
nd
Hdd
I nut
H sdd
Hosed
Hadd
H, sed
Multi-Step
Conductive
Segmentation Step-Decode
Reordering Step-Decoder
Decoding Phase
X1
X
X
Layout-Aware
Encoding Phase
Layout-Aware Encoder
i=
i-5 5
i-6 6
i-L
ID Position
Oo 2 Y Yo w. bly
Oce Yis Xi- Yo W. his
Ol. Fa Xi. his
Dig Yo
Xyy Vi- h)
2D Layout
Document Image
"Strategy"
"
"years"
Text Tokens
Figure 3: Proposed DIT framework LayoutDIT. During the layout- encoding phase, it jointly encodes document image's visual layouts and text semantics. During the multi-step conductive decoding phase, it generates the document translation step by step with chain of decoding uro-procssses: reading order decoding, sentence boundary decoding, and translation decoding. Each decoding step is achieved with cenction-speciffe step-decoder. With the conduction hidden states, three step-decoders are unified into one multi-step conductive decoder. enabling LayouDDIT's sub-modulc interaction and end-to-end training.
2.1 Document Image and Recognized Words
2.2 Layout-Aware Encoder
Similar to Xu et al. (2021), the i-th token ty em- bedding I; E R# consists of token embedding zeent, ID position embedding -.! and 2D layout embedding a
We normalize image's [width, height] to [1. 1, pixels and use off-the-sheeff OCR engine? for word recognition. Each word w; is identified by text 8; and bounding box b: (Imin, Umin; Tmars }maz)i. (Jimin: Ymin); ) and (Imaz, Umax)) are coordinates of the top-left and bottom-right corners of i-th word's bounding box. The recognized words are arranged "left-to- right, top-to-bottom" and are tokenized to form the input token sequence T = {4/i E [1, LJ}. where L is the total number with max length limit of 512.
loyout,
2; = LayerNorm (a/ + a + x:
foyout
(1)
Text Embedding
Position Embedding. The look-up table and absolute position are used for each token's 's text and ID position embedding:
x 2eezt (mbTTex((.);
pos EmblD(i) (2)
2D Layout Embedding. out representation x
We introduce 2D lay- layout E Rf that encodes i-th

---

token's visual layout:
LayouDDIT's segmentation step-decoder (SSD) em- ploys a lightweight Transformer (Zhao et al., 2023; Vaswani al., 2017) encoder for context encoding and a linear projection for token-level tagging:
oi layout = Lincar(((Emb., (Tmin: Rman, W))
Emby (Jmin: Mouens h):)])
where [::]] is the concatenation operator, Linear(-) is a linear projection layer to map embedding di- mension to H. w and h are word bounding box's width and height. For a word tokenized into mul- tiple sub-word tokens, we assign its bounding box to each token approximation due to OCR's inability for sub-word recognition.
h4 aad
S AM
(7)
pasd (oftmax(Linear(14
(8)
B Argmax(P."M)
(9)
where Pead is i-th token's 's classification probabil- ity over pre-defined sentence boundary classes (BOS, IOS), which represents the beginning and inside of a sentence, respectively.
The layout-aware encoder (LAE) is based Transformer (Zhao et al., 2023; Vaswani et al., 2017) encoder architecture and produces layout- aware representations x;- = I.AF((a)) for each to- ken.
Based on the decoded sentence boundary tags
(B.) the representation sequence {hr=d) [ /i=1
segmented into M sub-sequences (hgerd )
2.3 Multi-Step Conductive Decoder
hssnt R(( J==1 (B.)
(10)
Three step-decoders are unified through the con- duction of hidden states to form the multi-step con- ductive decoder and fulfill the decoding chain in threc-step end-to-end manner.
where R(-) the sentence segmentation rule that searches for BOS, and BOS within (B,) !-
and extracts sub-sequence hg from {hrad) fi=1 between [BOS BOS
2.3.1 Reordering Step-Decoder
2.3.3 Translation Step-Decoder
With layout-aware representations x* as context memory and previously generated reading order indexes, the reordering step-decoder (RSD) autore- gressively produces hidden states hrdd and reading order index Ide; of current input token 4;
Taking hgent the context memory of the k-th source sentence, the translation step-decoder (TSD) generates each sentence's 's translation autoregres- sively. Specifically, at the j-th decoding step of k-th source sentence, we have:
hand RSD(x", bee))
(4)
Toy = TSD( Mzj)
(11)
pirdd Softmax (Linearr
(5)
Plad = YkJ CeaamScarch(Pkg ptad Vk,<j)
softmax (LinearrxXL (ad))
(12)
Ida; = Argmax(P,"
(6)
(13)
where prod is the classification probability for tz's reading order index over L (input length) index classes, meaning that the Idx;-th token of input sequence should be "read" i-th decoding step. The reordering step-decoder based on Trans- former (Zhao et al., 2023; Vaswani et al., 2017) decoder architecture. Similar to the layout-aware encoder. it encodes tokens' bounding boxes at the embedding layer to make use of the 2D layout fea- ture.
TSD is based on the Transformer (Zhao et al., 2023;
Vaswani et al., 2017) decoder architecture. Trans- lations of all source sentences are generated in par- allel to promote training and inference efficiency.
The final document translation can be obtained by concatenating all sentences' translations.
2.4 Optimization
LayouDDITs is optimized with supervision from all three decoding steps as follows:
2.3.2 Segmentation Step-Decoder The hidden states redd outputted by RSD can be viewed as the semantic representations of in- order token sequence. Therefore, they are utilized to segment sentences from the decoded token se- quence in a sequence-labeling manner. Specifically,
Lysd
CE(Idr, PL
(14)
L
Leed
FocalLoss(B, P,ssdd P
(15)

---

G
ALGORITHM
Poee Strede
(A)
(
d
Taste Victory!
DO YOU I
1002
2002
IAD
New, True
R
hh)
Figure 4: Dataset samples. (a) (b) (c) (d) (e) (f): From DITrans. (a) (b): Political report. (c) (d): Newspaper. (e) (f): Advertisement. (g) (h): From ReadingBank.
Datasat
BeadingRank
DiTraw
general-domain document images with pseudo la- bels. Therefore, we first manually annotate a multi- domain DIT dataset named DITrans to test Layout- DIT's domain-specific capability. DITrans contains sophisticatedly selected images of three domains: political report, newspaper, and advertisemeni. We hire 35 professional annotators to label the reading order and sentence boundaries and 30 translators for document translation. Translators are shown document image with reading order and sentence boundaries and are required to produce correct and fluent translations for them in Chinese. addition, We hire 8 professional annotators to sample and check the annotated instances for quality control. We annotate 1,675 instances in total with 34,456 parallel sentence pairs. Figure 4 presents document image examples from our DITrans and the public ReadingBank. Table gives cursory review of the two datasets.
Doszzn
Geseral
Poltical Repect
Newspapes
Advestiention
410K
902
245
Page)
7W0/12
Trais/Tout
1,467
19,21022,00
Table Datasets Statistics. Word/Sent. Avg. means average # word/sentence of each document image.
1T2 Ltsd CE(Y,j, Pkad ΣΣ (16)
M Yi
Here Idx;, Bi, YE refer the ground-truth la- bels of the reading order index, sentence bound- ary class, and target-language token, respectively. CE(-) refers to CrossEntropy. FocalLoss(() (Lin et al., 2017) is introduced to alleviate the class- imbalance issue. Sum of these loss functions forms LayouDDIT's total training loss:
4 Experiments
Lisl Lystt + Lasd + Lisd
(17)
4.1 Datasets and Evaluation Protocal
Experiments are conducted on ReadingBank and our DITrans for the en-zh DIT task. The original split of ReadingBank Train: Test 400K: 50K. As shown Table I, we sample 10K from its 50K test examples as the testset. For DITrans, each
3 Multi-Domain DIT Dataset
To the best of our knowledge, there is only one publicly available dataset named ReadingBank (Wang et al., 2021) that consists of web-crawled

---

domain is split with the ratio Train: Test 7:1. We evaluate model performance with average page-level BLEU (Papineni et al., 2002) that refers to the micro-average precision of n-gram over- laps within a page between model prediction and ground-truv truth reference.
Method
End-to-Ead
Params (M)
BLEU
DocHandlee-- (Sable al., 2023)
142
30.47
DecHandler
172
37.75
(Sable al., 2023)
LayoutLM-Des M-Dec
232
45.54
LiLT
250
45.79
4.2 Settings
LayonDiT Cascade
293
48.29
We use Transformer's encoder/decoder layer to build Layou(DIT's components. Concretely. LayoutDIT's LAE/RSD/SSD/TSD consists of 6/6/1/6 layers, respectively. Each layer has 768- dimensional hidden sizes, 12 attention heads, and 3,072 feed-forward hidden units.
LayouDIT
206
48.20
Table 2: Results on ReadingBank. LayouDiT signif- icantly outperforms exsiting cascades and end-to and-to-end baselines. It also achieves competitive results with a lighter model architecture compared with its cascade variant LayoutDIT-Casent
We continue training LayoutDIT on DITrans' experiments after pretraining it on the large-scale ReadingBank. Adam optimizer (Kingma and Ba, 2015) is applied for training with 31 = 0.9, 32 0.98. The learning rate is 7 x 10 with a linear schedule strategy. Both the dropout rate and label smoothing value are set to 0.1. The training of LayoutDIT lasts for epochs with batch size of 30 for all training runs.
is employed as the encoder and is equipped with text decoder for translation.
LiLT-Dec. It replaces LayoutLM with the dual-stream LiLT (Wang et al., 2022) encoder that shows stronger document image under- standing ability.
For fair comparisons, the numbers of en- coder/decoder layers for LayoutL and LiLT- Dec are 12 and share the same hidden size, # attention head and feed-forward unit as that of LayoutDIT. All the training hyperparameters (batch size, training epochs, learning rate, etc) are kept the same LayoutDIT. We also continue training all baselines on DITrans' experiments after pre- training them on ReadingBank.
4.3 Baselines
We consider existing cascade methods as baselines:
DocHandler (Sable et al., 2023). DocHandler-1: It enhances DocHan- dler's Transllaaot cascade by inserting rule-based reading order detector ("left- to-right, top-to-bottom") & an unsupervised sentence segmenter (Kiss and Strunk, 2006) between them. DocHandler-2: It replaces DocHandler-I's sentence segmenter with supervised Transformer-based sequence labeler.
4.4 Main Results
On ReadingBank. The results are shown Ta- ble 2. Since ReadingBank has no layout annota- tions for training layout parser, we exclude MG- Trans in this experiment. LayourD[T-Cascade is LayoutDIT's cascade variant that explicitly out- puts all intermediate results (tokens tags) during decoding sub-processes. It also replaces Layout- DIT's translation decoder with an encoder-decoder to encode tokens, thus increasing model parame- ters. Our proposed LayouDIT significantly out- performs existing methods by improving BLEU from 37.75 to 48.20. Compared with LayouDDIT- Cascade, LayouDDIT achieves competitive results with a lightweight model size (87M fewer param- eters), demonstrating its parameter efficiency be- cause training and maintaining a cascade is costly, On DITrans. The following two experimental settings are conducted on DITrans to verify Lay- outDIT's effectiveness.
MGTrans (Hinami et al., 2021). It employs layout parser for layout boxes detection and equips rule-based reading order detector, sentence segmenter, and translator. We em- ploy two SOTA layout parsing frameworks: 1) Convolution-based Cascade- RCNN (Cai and Vasconcelos, 2018), denoted as MGTrans- Conv. 2) Transformer-basee DETR (Zhu et al., 2020), denoted as DerTrann--ERR
We also build two strong end-to-end baselines to compare with our proposed LayoutDIT.
LayoutLM-Dec. LayoutLM (Xu et al., 2020). multi-modal document image understanding model that pre-trained on scale IIT- CDIP (LEWIS et al., 2006) document images,

---

Leomingg
Poltical Repori
Paliical
Advetticmant (TD.2)
2013)
D
3 X00Taa DETR (Hiani 2021)
212
26.30
27.61
25.76
27.34
1626
13.00
Manooao (ltromi is 2011)
233
28.99
35.44
2885
2051
17.78
1417
Dep
232
45.86
48.66
36,00
45.71
27.56
21.40
280
46.11
40.11
16.22
40.1
3171
18
LepertDi- Caasak
200
36.78
34.56
35.36
39,85
21,99
16.02
Lapesbor
86.7
16.43
4532
66.73
25/11
20.46
Table Results on DITrans. SD/TD means source/target domain. For DITrans' three domains, LayoutDIT shows the best results compared with existing cascade methods. Despite lightweight model size, it also achieves better or competitive results compared with end-to-end baselines, in both multi-domain learning and cross-domain transfer.
Model
BLEU
able translation performance (39.40) as model #2b shows, which may be because the 2D layout can guide LayouDDIT to generate a reasonably good reading order. However, the performance drop from model #1's 48.20 to model #2b's 39.40 in- dicates the requirement of 1D position for the trans- lation step-decoder. Combining the three features produces the best results.
LayoutDIT
48.20
2a who 2D) layout Embedding
43.87
2b
w/o 1D position Embedding
39.40
reading order decoding loss
3.15
decoding processes decomposition
13.99
hidden states context memory
45.85
Table 4: Ablation study of our model on ReadingBank.
On the Multi-Step Conductive Decoding and Step-Decoder Supervision. Model #3a abandons the supervision from the reordering step-decoder during training. Model #3b disables the multi- step conductive decoding paradigm and utilizes Vallina encoder-decoder directly transform the out-of-order source word sequence to document translation. Compared with model #1, the drastic performance drops of model #3a and #3b prove the effectiveness LayoutDIT's multi-step conductive decoding and multi-task learning strategy.
1) Multi-Domain Learning. In this setting, models are trained on DITrans' all three domains and tested on each domain. As shown in Table 3, proposed LayoutDIT achieves the best results, outperforming previous cascade methods by sub- stantial margin. Moreover, It also shows better or comparable results than end-to-end baselines with a lighter architccture, demonstrating its excellent capability in domain sppecif DIT conditions.
2) Zero-Shot Cross-Domale Transfer. In this setting, models are trained only on political report and are tested on each domain. As shown in Table 3. LayoutDIT achieves state-of-the-ard results for all three domains, demonstrating that can better transfer knowledge from the seen domain to the unseen domain in zero-shot DIT scenarios.
On the Translation Context Memory. Layout- DIT's translation context memory from reorder- ing step-decoder (RSD) hidden states instead of segmentation step-decoder (SSD). Our conjecture for this is that RSD's s hidden states are for token in- dex prediction while SSD's s are for sentence bound- ary classification, So the former is closer to source text understanding and should act as the translation context memory. This assumption is empirically confirmed by comparing model #1 and #4.
4.5 Ablation Study
To investigate the effectiveness of different compo- nents, we further compare LayoutDIT with several variants in Table 4.
4.6 Low-Resource DIT
On the Input Embedding. Removing Layout- DIT's 2D layout embedding or ID position embed- ding results in model #2a/#2b. Comparing #1, #2a, #2b shows that: 2D layout is useful for Layout- DIT understand document layouts for translation improvement from model #2a's 43.87 to model #1's 48.20. 2) Incorporating 2D layouts and text semantics w/o. ID position can give an accept-
Benefiting from the joint optimization of compo- nents, end-to-end models have always shown su- periority (Bansal et al., 2018; Tu et al., 2019) than cascades in low-resource conditions. which is par- ticularly challenging for DIT due to the expensive annotation of document images. Therefore, we compare LayoutDIT's s few-shot Iearning ability on DITrans political report with the cascade method

---

N-shot Results
Trans misrecognizes several small layout boxes be large layout (red box). This falsely parsed layout disrupts the overall box-level reading or- der. Moreover, as shown in (b), because MG- Trans' "left-to-right, top-to-bottom" reading order within a layout box, some words are placed in the wrong position, causing further errors prop- agated to sentence segmenter. The accumulated er- rors ultimately cause mis-segmented, semantically- confusing source sentences and a failed translation, as shown in (c). On the contrary, by incorporating layout information, our LayoutDIT correctly cap- tures the reading logic of words. The subsequent sentence boundary decoding and translation decod- ing also exhibit better results, validating the effec- tiveness of layout incorporation in LayoutDIT's layout-aware encoder and joint optimization of its step-decoders for minimal error accumulation.
50
10.20
35
30
28.05
LayoutDIT LayoutDIT Cascade 2434
MGTrans-Conv
5D
100
150
200
250
300
350 400 450 500
fine-tuning samples
Figure 5: Comparison of few-shot learning.
Method
ReadingBank En-De
ReadingBank En-Fr
DocHandler-2
17.15
18.23
LiLT-Dec
30.60
36.00
LayoutDT
33.25
40.09
Table 5: Results on En-De and En-Fr DIT tasks. Evalu- ation metric BLEU.
Related Work
Sentence Image Translation. Sentence Image Translation (SIT) aims to translate an image with single embedded sentence from one language to another. It is typically achieved by joining the OCR and translator. Recently, end-to-end models (Ma et al., 2023a; Ma et al., 2023b; Ma et al., 2023c; Ma al., 2022; Chen et al., 2022; Chen al., 2021) have been proposed address cascades' er- ror propagation and parameter redundancy. One approach is taking the recognition model for SIT like TRBA (Baek et al., 2019). To alleviate the end-to-end data scarcity, multi-task learning (Ma et al., 2022; Chen et al., 2021; Su et al., 2021) is leveraged to incorporate external OCR/MT datasets for training end-to-end models. MHCMM (Chen et al., 2022) further enhances feature representa- tion through cross-modal mimic learning with ad- ditional MT data.
MGTrans-Core
As shown Figure 5, LaysutDeT-Cascade con- stantly outperforms MGTrans by a large mar- gin. Notably, LayoutDDT-Cascade achieves almost equivalent performance (28.05) using 25 samples than MGTrans does (28.07) using 500 samples. LayoutDIT brings further improvement on top of LayoutDIT-Cascad,, showing great superiority in low-resource DIT conditions.
4.7 Evaluation More Languages
We further evaluate LayoutDIT on English-Genne German (En-De) and English-French (En-Fr) document im- age translation. We first use document images downloaded from ReadingBank (Wang et al., 2021) to construct the synthetic translation labels with Google Translate API, resulting in the En-De and En-Fr versions of ReadingBank. All models are trained for 80K steps with batch size of 10. Table 5 shows the results of our proposed LayoutDIT on these two language pairs. Again we observe that our model substantially outperforms the cascaded DocHandler-2 by 16. 1./21.86 BLEU and the end- to-end LiLT-Dec by 2.65/4.09 BLEU on En-De and En-Fr directions, respectively.
Although end-to-end SIT models have shown ac- ceptable performance compared with their cascade counterparts, all of them presuppose that the tence bounding boxes of document image are pro- vided by annotation and only focus on the cropped sentence images translation, leaving the more prac- tical document image translation under-explored. Document Image Translation. Existing DIT methods (Sable et al., 2023; Hinami et al., 2021: Shekar al., 2021: Afli and Way, 2016) are scarce and are almost cascade methods, suffering layout- unaware and error propagation problems. The first attempt (Jain et al., 2021) !) at constructing an image-
4.8
Case Analysis
To further reveal LayoutDIT's advantages, we provide a translation example of LayoutDIT and MGTrans Cony in Figure 6. As shown in (a), MG-

---

HIGHLIGHTS INCLUDE:
HIGHLIGHTS INCLUDE:
develop smart systems and tratetion construction p>:l businesses pledged take 60+ growth during GB werk pnoee gevernment funding for low 1320m carbon
develop systems and transform construction |I 00- businesses pladged tike clean growth during Green GiB wock | |330mm gosernment funding for low curbon hesting. matched
Maximi sing the advantegee ges for Industri from the glot-al shift
Maximiing snn the advanta Res Industry from the global a shift clear growth
cleari growth
heating, matcheal industry
(h)
7GA EUG 2130 2330 3P28
OF t2 GB 10 1810 13
9531 28%
SERR S
X
1760
2035 H (252
Mt X
122802
(5) Abv) #1 62 St tast 83
BERd
BOS
Translation References
2002
RUR 75
(g)
(d)
Figure 6: A political report translation example of MGTrans-Com and LayoutDIT. (a): Parsed layout boxes and reading order produced MGTrans. Red color denotes MGTranss errors. (b): Segmented sentences from the red box in (a). Red color denotes MGTrans' errors of word order and sentence boundaries "Il". Blue color means ground-truth labels. (c): MGTrans' translation results. (d): Reading order decoded by LayoutDIT.
clarity, we post-process LayoutDIT's word-level reading order to box-level. (c): Sentence boundaries decoded by LayoutDIT. It only misses one sentence boundary denoted in blue. (f): LayoutDIT's translation results. (g): Translation references.
to translatiion end-to-end DIT model is based Vanilla visual-encoderrdecodee framework and only focuses on one-column regular-layout docu- ment images. It requires massive data for training and fails to handle real-world complex-layon doc- ument images.
Limitations
Since our model involves an additional step OCR, its model size could be further compressed to achieve OCR-free DIT with the incorporation of OCR models, which will be our future explo- ration. Besides, our model leaves the document image's vision information be exploited, which an important clue for better document image un- derstanding. We will conduct experiments with Lv223 for image feature incorporation our future works.
Different from them, LayoutDIT goes beyond and pursues the end-to-end translation of arbitrary- layout document images, effectively and efficiently.
6 Conclusion
Acknowledgements
We are grateful to all annotators for constructing DI- Trans. This work is supported by the National Nat- ural Science Foundation of China (NO. 62 62106265).
DIT has been shown unsatisfactory performance due to the struggling layout incorporation and cas- cade sub-module isolation of existing methods. In this work, we propose LayoutDIT to alleviate these two issues. utilizes a layout-aware encoder for layout-text joint understanding and a multi-step conductive decoder for step-by-ster translation de- coding. Benefiting from the layout-aware end-to- end modeling, LayouDDIT significantly surpasses previous approaches and promotes DIT to higher performance level. the future, we will explore in- corporating OCR and images' vision features into our framework to realize a more efficient and pow- erful DIT.
References
Haithem Afli and Andy Way. 2016. Integrating optical character recognition and machine translation of his- torical documents. In Proceedings of the Workshop on Language Technology Resources and Tools for Digital Humanities (LT4DH), pages 109-116.
Jeonghun Back, Geewook Kim, Junyeop Lee, Sungrae Park, Dongyoon Han, Sangdoo Yun, Scong Joon Oh, and Hwalsuk Lee. 2019. What is wrong with scene text recognition model comparisons? dataset and model analysis. In Proceedings of the IEEF/CVE

---

international conference on computer vision. pages 4723.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In Proceedings of the IEFF international conference on computer vision, pages 2980-2988.
Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, and Sharon Goldwater. 2018. Low-resource speech-to-text translation. In Interspeech, pages 1298-1302.
Shangbang Long, Siyang Qin, Dmitry Panteleev, Alessandro Bissacco, Yasuhisa Fujii, and Michalis Raptis. 2022. Towards end-to-ent unified scene text detection and layout analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 1049-1044 1059.
Zhaowei Cai and Nuno Vasconcelos. 2018. Cascade r-cnn: Delving into high quality object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6164-6162.
Cong Ma, Xu Han, Linghui Wu, Yaping Zhang. Yang Zhao, Yu Zhou, and Chengqing Zong. 2023a. Modal contrastive learning based end-to-end text image ma- chine translation. IEEE/ACM Transactions Audio, Speech, and Language Processing, pages 1-13.
Jinying Chen, Huaigu Cao, and Premkumar Natarajan. 2015. Integrating natural language processing with image document analysis: what we learned from two real-world applications. International Journal on Document Analysis and Recognition (IJDAR), 18:235-247.
Cong Ma. Yaping Zhang, Mei Tu, Xu Han, Linghui Wu, Yang Zhao, and Yu Zhou, 2022. Improving end-to-end text image translation from the auxiliary text translation task. In 2022 26th International Con- ference on Pattern Recognition (ICPR), pages 1664- 1670.
Zhuo Chen, Fei Yin, Qing Yang, and Cheng-Lin Liu. 2022. Cross-linge lingual text image recognition via multi- hierarchy cross-mi modsl mimic. IEEE Transactions on Multimedia.
Zhuo Chen, Fei Yin. Yao Zhang, Qing Yang, and Chena-Lin Liu. 2021. Cross-lingual text image recog- nition via multi-task sequence to sequence learning. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 3122-3129.
Cong Ma. Yaping Zhang. Mei Tu, Yang Zhao, Yu Zhou, and Chengqing Zong. 2023b. E2timt: Efficient and effective modal adapter for text image machine translation. In Document Analysis and Recognition ICDAR 2023, pages 70-88, Cham. Springer Nature Switzerland.
Jun Du, Qiang Huo, Lei Sun, and Jian Sun. 2011. Snap and translate using windows phone. 2011 1 Interna- tional Conference on Document Analysis and Recog- nition, pages 809-813.
Cong Ma, Yaping Zhang, Mei Tu, Yang Zhao, Yu Zhou, and Chengqing Zong. 2023c. Multi-teacher knowl- edge distillation for end-to-end text image machine translation. In Document Analysis and Recognition ICDAR 2023, pages 484-501, Cham. Springer Nature Switzerland.
Ryota Hinami, Shonosuke Ishiwatari, Kazuhiko Yasuda, and Yusuke Matsui. 2021. Towards fully automated manga translation. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, pages 12998-13008.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing 7hu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 311-318.
Jain Orhan Firat, Qi Ge, and Sihang Liang. 2021. Image translation network. In Image Translation Model.
Nilesh P. Sable. Priya Makarand Shelke, Ninad De- ogaonkar, Nachiket Joshi, Rudra Kabadi, and Tushar Joshi. 2023. Doc-handler: Document scanner, manip- ulator, and translator based on image and natural lan- guage processing. 2023 International Conference on Emerging Smart Computing and Informatics (ESCT), pages 1-6.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings.
Tibor Kiss and Jan Strunk. 2006. Unsupervised multi- lingual sentence boundary detection. Computational linguistics, 32(4):485-55)
Chandra Shekar, Maria Anisha Cross, and Vignesh Vasudevan. 2021. Optical character recognition and neural machine translation using deep learning tech- niques. Innovations in Computer Science and Engineering: Proceedings of 8th ICICSE, pages 277- 283. Springer.
D LEWIS. G AGAM, S ARGAMON, O FRIEDER, D GROSSMAN, and J HEARD. 2006. Building a test collection for complex document informa- toon processing. SIGIR 2006 (proceedings of the Twenty-Ninth annual international ACM SIGIR Conference research and development in informa- tion retrieval, August 6-11, 2006, Seattle, WA, USA),
Tonghua Su, Shuchen Liu, and Shengjie 7hou, 2021. Rtnet: An end-to-end method for handwritten text image translation. In Document Analysis and Recognition-CDDAR 2021: 16th International
pages 665-666.

---

Conference, Lausanne, Switzerland, September 5- 10. 2021, Proceedings, Part 16, pages 193. Springer.
Tao Tu, Yuan-Jui Chen, Cheng chich Ych, and Hung yi Lee. 2019, End-to-end text-to-speech for low- resource languages by cross-lingual transfer learning. In Interspecch.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Jiapeng Wang, Lianwen Jin, and Kai Ding. 2022. Lilt simple yet effective Imguage-independenn layout transformer for structured document understanding. Proceedings of the 60th Annual Meeting of Association for Computational Linguistics VVolume 1: Long Papers). pages 7747-7757.
Zilong Wang, Yiheng Xu, Lei Cui, Jingbo Shang, and Furu Wei. 2021. Layoutreader: Pre-training text and layout for reading order detection. In Proceed- ings the 2021 Conference Empirical Methods Natural Language Procexsing, pages 4735-4744
Yang Xu, Yiheng Xu, Tengchao Lx, Cui, Furu Wei. Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang. Wanxiang Che. et al. 2021. Layoutlmv2: Multi-modal pre-training for visually-rich document understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2579-2591
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020. Layoutlm: Pre-training of text and layout for document image understanding, In Proceedings of the 26th ACM SIGKDD Interna- tional Conference on Knowledge Discovery Data Mining, pages 1192-1200.
Yang Zhao, Jiajun Zhang, and Chengqing Zong. 2023. Transformer: A general framework from machine translation to others. Machine Intelligence Research, 20:514-55 538.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. 2020. Deformable detr: De- formable transformers for end-to-end object detec- tion. ArXiv, ab4/2010.04159.


Reviews:
[{"review_id": "k5qNK6Gq1b", "reviewer": "Reviewer_sqAx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the acknowledgment of our novelty and impressive results, where we propose the first end-to-end DIT framework that incorporates visual information.\n#### 1. DITrans Dataset Analysis.\n  * We appreciate your attention to the size and analysis of our DITrans dataset. Actually, our main contribution is to propose the first end-to-end DIT framework towards document images with complex layouts.\n  * DITrans and ReadingBank are used to evaluate specific-domain and general-domain performance, respectively. Compared with ReadingBank, DITrans was created to validate that the model can handle diverse and complex document images in the real world. Specifically, DITrans contains sufficient document images selected, scanned and annotated by humans with high-quality labels, while ReadingBank is created with the large-scale pseudo-labels. We will present more layout examples in the revision’s appendix.\n\n#### 2. The impact of each component.\n * In the proposed framework, reading order decoding, sentence boundary decoding and translation decoding are indispensable in multi-step decoding. These three sub-modules divide and conquer the DIT task step by step in a serial, interrelated mode instead of a parallel, irrelevant way.\n * As you point out, our paper could benefit from more ablation studies analyzing the impact of each component. Actually, we have already analyzed the impact of each component in Table 4. Specifically, comparing model #1 with #3a in Table 4 indicates the necessity of an optimized reordering step-decoder. Without it, the out-of-order hidden state sequence conducted to the subsequent decoders causes a severe performance drop from 48.20 to 3.15. Moreover, comparing model #1 with #3b in Table 4 verifies the necessity of reordering step-decoder and segmentation step-decoder. Without them, the translation step-decoder struggles to translate an out-of-order long document instead of in-order, short sentences, causing a performance drop from 48.20 to 13.99. Following your suggestions, based on current experiments, we will conduct more ablations to analyze the impact of each component in the revised version.\n\n#### 3. On the language generalization.\n * Thanks for the valuable recommendation about LayoutXLM. Following your suggestion, we will evaluate the proposed framework on multilingual models, including LayoutXLM in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents an end-to-end document image translation (DIT) framework called LayoutDIT, which aims to improve the accuracy of translating text within document images from one language to another. The authors make several key contributions that address limitations in existing methods:\n1. They introduce the first end-to-end DIT framework that incorporates visual layout information, avoiding error propagation seen in traditional cascade approaches.\n\n2. The authors design a multi-step conductive decoder with three step-decoders to handle reading order, sentence boundaries, and translations in a unified end-to-end manner.\n\n3. The proposed LayoutDIT achieves impressive results on the ReadingBank dataset and outperforms baselines on their newly created multi-domain DIT dataset, DITrans.", "reasons_to_reject": "1. The proposed DITrans dataset is relatively small (1675) compared to ReadingBank (400k). More analysis on the diversity and complexity of DITrans could be useful. When analyzing a dataset, the intermediate English annotations between the source documents and the translated documents are important for the quality of this dataset. But this analysis is not included.\n2. The paper could benefit from more ablation studies analyzing the impact of each component (Reading order decoding, Sentence boundary decoding, and Translation decoding). For example, using existing machine translation benchmarks to provide translation performance in terms of correct and incorrect reading orders would be an insightful benchmark for comparing and analyzing translation decoding steps. Based on current experiments, it is still unclear which component has the most impact on the final translation performance.\n3. Leveraging multilingual models like LayoutXLM could potentially improve generalization across languages. The current baseline uses an English-only LayoutLM model.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents a novel end-to-end document image translation (DIT) framework called LayoutDIT, which aims to improve the accuracy of translating text within document images from one language to another. The authors make several key contributions that address limitations in existing methods:\n\n1. They introduce the first end-to-end DIT framework that incorporates visual layout information, avoiding error propagation seen in traditional cascade approaches.\n\n2. To jointly model text semantics and 2D layout features, they propose a layout-aware encoder, which enhances the overall translation process.\n\n3. The authors design a multi-step conductive decoder with three step-decoders to handle reading order, sentence boundaries, and translations in a unified end-to-end manner.\n\n4. The proposed LayoutDIT achieves impressive results on the ReadingBank dataset and outperforms baselines on their newly created multi-domain DIT dataset, DITrans. It also demonstrates advantages in low-resource DIT scenarios.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yejI6Pp2L3", "reviewer": "Reviewer_ZMNX", "summary": "", "questions": "1. Will you release the DITrans dataset for community reproduction?\n\n2. Technical questions:\n\n(1) I would like to know the decoder configurations in LayoutLM-Dec and LiLT-Dec. This can help the readers compare LayoutDIT to the baselines. It would be better to list the detailed decoder configurations in the experiment section.\n\n(2) Have you tried LayoutLMv2-Dec as the baseline given that LayoutLM-Dec is compared? Because LayoutLMv2 [2] should be a better encoder backbone compared to LayoutLM [1].\n\n\n[1] LayoutLM: Pre-training of Text and Layout for Document Image Understanding, KDD 2020\n\n[2] LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding, ACL 2021", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your valuable reviews and for pointing out typos. We will carefully check it in the revision.\n#### 1. Comparison with span-based OCR and the ERNIE-Layout.\n * Thanks for your valuable recommendations about span-based OCR and the ERNIE-Layout literature. We have carefully inspected the ERNIE-Layout paper, especially the details about the preprocessing in its sections 3.1 and A.1. Actually, we have experimented with MGTrans, which shares nearly the same workflow with ERNIE-Layout. Both of them fall into the conventional cascade: OCR for word recognition -> layout parser for layout boxes detection -> rule-based layout block reordering, except that MGTrans is equipped with an additional sentence segmenter since it is for the translation task. In this regard, the case in Figure 6 straightforwardly displays that MGTrans’ cascade convention suffers error accumulation at every step. Experimental results from Table 3 and Figure 5 also give a statistical verification of MGTrans’ inferior performance compared with our end-to-end LayoutDIT.\n* As for the span-based OCR, we further conduct a non-exhausted examination of some mainstream OCR engines including the commercial [MSOCR](https://azure.microsoft.com/en-ca/products/form-recognizer), [BaiduOCR](https://ai.baidu.com/tech/ocr/general), [TencentOCR](https://cloud.tencent.com/product/ocr), and the open-source [EasyOCR](https://github.com/JaidedAI/EasyOCR) and [TesseractOCR](https://github.com/tesseract-ocr/tesseract). Most of them only return a batch of recognized words or cropped text lines arranged in “left-to-right, top-to-bottom” order. This observation may explain our motivation to explicitly bring RSD and SSD into our framework as a solution for the unavailable external tools and in pursuit of sub-module joint optimization.\n * We also want to give a justification from an empirical optimization perspective. Compared with the fixed, unlearnable external tools, our learnable RSD and SSD modules could benefit from the sub-module interaction to reduce error accumulation, especially for the adaptation to low-resource document image layouts (Figure 5).\n\n#### 2. On the DITrans dataset.\n * Yes, we will release our DITrans to the research community through the HuggingFace platform, for reproduction of this paper. We will also keep enriching it with more domains and layout formats to facilitate long-term DIT research.\n\n#### 3. Answers to technical questions.\n##### 3.1. On the decoder configurations.\n * Detailed configurations of LayoutDIT have been described in section 4.2. For fair comparisons, both decoders of LayoutLm-Dec and LiLT-Dec are 12-layer Transformer decoders and share the same # hidden size, # attention head and # feed-forward units as that of LayoutDIT. We will list all parameters about model configurations and training runs in the final 9-page revision.\n##### 3.2. On the comparison between our model and LayoutLmv2-Dec.\n * We have experimented with the canonical LayoutLM [1] and the state-of-the-art layout model LiLT [2] which shows significant improvement compared with LayoutLMv2 [3]. Following your suggestions, we will add experiments about LayoutLMv2 as well. Thanks for your valuable advice.\n\n> [1] LayoutLM: Pre-training of Text and Layout for Document Image Understanding, KDD 2020\n\n> [2] Lilt: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding, ACL 2022\n\n> [3] LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding, ACL 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well-written and easy to understand.\n\n2. Considering layout information for document image translation is reasonable. The three-step conductive decoder can implement modularized word reordering, sentence segmentation and ultimate translation. It is a clear and rational model architecture.\n\n3. The experiments prove that LayoutDIT can obtain SOTA and generalized performance.", "reasons_to_reject": "As far as I know, some popular OCR tools can also achieve reordering (step 1) and segmentation (step 2) functions, and their reordering and segmentation accuracies are satisfactory. For example, ERNIE-Layout [1] uses the preprocessed span-based OCR (reordered and segmented) to perform large-scale pretraining. Directly using span-based OCR instead of sequence-based MSOCR may further alleviate the error propagation of reordering and segmentation. LayoutDIT also relies on OCR preprocessing, it seems that RSD and SSD can be replaced by off-the-shelf span-based OCR tools.\n\n\n[1] ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding", "questions_to_authors": "1. Will you release the DITrans dataset for community reproduction?\n\n2. Technical questions:\n\n(1) I would like to know the decoder configurations in LayoutLM-Dec and LiLT-Dec. This can help the readers compare LayoutDIT to the baselines. It would be better to list the detailed decoder configurations in the experiment section.\n\n(2) Have you tried LayoutLMv2-Dec as the baseline given that LayoutLM-Dec is compared? Because LayoutLMv2 [2] should be a better encoder backbone compared to LayoutLM [1].\n\n\n[1] LayoutLM: Pre-training of Text and Layout for Document Image Understanding, KDD 2020\n\n[2] LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding, ACL 2021", "paper_topic_and_main_contributions": "This paper studies document image translation. To address the error propagation problem, the authors propose LayoutDIT with a three-step conductive layout-aware decoder including reordering, segmentation and translation sub-decoders. These multi-step conductive decoders can be optimized together and outperform previous baselines of isolated components. This paper also brings a new dataset DITrans, for document image translation. Experiments show that the overall performance of LayoutDIT is promising.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In line 44, \"Given\" -> \"given\""}, "extra_text": {}}, {"review_id": "TFShNbf0az", "reviewer": "Reviewer_iJGw", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the valuable review and the time dedicated to reviewing our paper.\n\n#### 1. On the novelty of our method.\n * Our novelty mainly lies in the original DIT framework, which indeed for the first time achieves end-to-end DIT for complex-layout document images.\n * For its decoding phase, we innovatively design a multi-step conductive decoder to unify reading order detection, sentence segmentation and ultimate translation. It contributes a lot to end-to-end DIT, comparing our LayoutDIT with LayoutLM-Dec in Table 2 & Table 3.\n * For its encoding phase, we make the first attempt to insert the layout knowledge into DIT. The layout-aware encoder is not limited to the 2D-style layout box embedding and could be easily extended to other advanced layout-aware strategies, e.g., from document layout analysis tasks as you recommend. There are alternative implementations of the layout-aware encoder, and in this paper, we follow the simple and effective layout embedding strategy in LayoutLM [1] as an instantiation, as described in section 2.2. Following your suggestions, we will conduct experiments with more layout strategies from document layout analysis tasks.\n\n#### 2. About the supplemental reference\n * Thanks for your recommendation about the literature [2]. We have carefully inspected its task and model which focuses on unified word detection and word clustering (layout analysis) for scene text images, while our framework pursues end-to-end translations for document images with long document text and complex layouts. In this regard, they are two orthogonal contributions to complementary tasks. We will add the missing reference in the revised version.\n\n> [1] LayoutLM: Pre-training of Text and Layout for Document Image Understanding, KDD 2020\n\n> [2] Towards End-to-End Unified Scene Text Detection and Layout Analysis, CVPR 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It's an end-to-end document image translation model, probably first of its kind.", "reasons_to_reject": "Lack of novelty:\n(1) The novelty of the so-called layout-aware encoder lies in the combination of 1D, 2D, pos embedding and word embeddding. However, this is just a standard practice in most document layout analysis tasks so the novelty is limited.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This ppaer propose a layout-aware end-to-end document translation framework named LayoutDIT, with a novel layout-aware encoder and\nmulti-step conductive decoder to enable layout awareness and sub-module interactions simultaneously, which could share computation among four complementary tasks, including layout parsing, reading order detection, sentence segmentation, and translation. This paper also curated a new multi-domain DIT dataset named DITrans.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- Long, Shangbang, et al. \"Towards end-to-end unified scene text detection and layout analysis.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an end-to-end solution to document image translation.  The encoder jointly takes text and the layout information as input.  The decoder has three step to deal with ordering, sentence boundary, and translation respectively.  Experiments show significant improvement over baselines.   The paper also present a new dataset for this research.  Reviewers raised concerns and the authors responsed in details.  I think the main contribution of the paper is clear: the end-to-end architecture / the three step decoder and the dataset.  The results are positive.
Decision: Accept-Findings