CCIM: Cross-modal Cross-lingual Interactive Image Translation
Authors: Cong MA
Yaping Zhang
Mei Tu
Yang Zhao
Yu Zhou
Chengqing Zong
Venue: EMNLP 2023 Findings (2023)
Keywords: cross-modal cross-lingual interactive decoding
text image machine translation
text image recogntion
TL;DR: A novel Cross-modal Cross-lingual Interactive text image translation model is proposed to incorporate source language information by synchronously generating source and target language results through an interactive attention mechanism.

Abstract: Text image machine translation (TIMT) which translates source language text images into target language texts has attracted intensive attention in recent years. Although the end-to-end TIMT model directly generates target translation from encoded text image features with an efficient architecture, it lacks the recognized source language information resulting in a decrease in translation performance. In this paper, we propose a novel Cross-modal Cross-lingual Interactive Model (CCIM) to incorporate source language information by synchronously generating source language and target language results through an interactive attention mechanism between two language decoders. Extensive experimental results have shown the interactive decoder significantly outperforms end-to-end TIMT models and has faster decoding speed with smaller model size than cascade models.

CCIM: Cross-modal Cross-lingual Interactive Image Translation
Cong Ma Yaping Zhang : Mei Tu', Yang Zhao' Yu Zhou? Chengqing Zong' 1.2 'School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences, Beijing, China Fanyu Al Laboratory, Zhongke Fanyu Technology Co., Ltd, Beijing, China "Samsung Research China Beijing (SRC-B) (cong.ma, yaping. zhang, yang. yang.zhao, yzhou, cqzong)@nlpr ia mei tu@samsung... com
ia.ac.cc..... ac. cn
Abstract
source language embedded in text images, the cas- cade model combines TIR models and MT models for the TIMT task. However, two individual mod- els in the cascade frame have double parameters and the decoding speed is slow, Meanwhile, er- rors in the TIR model are further propagated in the MT model leading to performance decrease. The end-to-end method directly translates the source language text image into target language through unified encoder-decoder architecture, which is more parameter-efficien than cascade models with faster decoding speed (Ma et al., 2022; Su et al., 2021; Mansimov et al., 2020; Chen et al., 2020; Ma et al., 2023a,b,c).
Text image machine translation (TIMT) which translates source language text images into tar- get language texts has attracted intensive at- tention in recent years. Although the end- to-end TIMT model directly generates target translation from encoded text image features with an efficient architecture, lacks the rec- ogmized source language information resulting decrease in translation performance. In this paper, propose novel Cross-modal Cross-lingual Interactive Model (CCIM) to in- corporate source language information by syn- chronously generating source language and tar- get language results through an interactive at- tention mechanism between two language de- coders. Extensive experimental results have shown the interactive decoder significantly out- performs end-to-end TIMT models and has faster decoding speed with smaller model size than cascade models."
However, the performance of end-to-end mod- els is limited because the translation process lacks explicit source language guidance from recogni- tion texts. An intuitive solution is to incorporate the recognition history into the translation decoder to offer more efficient guidance. Recently, multi- source interaction has been studied incorporate effective information into target model (Lu et al., 2022; Xu et al., 2021; He et al., 2021; Liu et al., 2020; Zhou et al., 2019a,b; Wang et al., 2019; Zoph and Knight, 2016). Although multi-source inter- action is vital enhance the encoding capacity of TIMT model through attending recognition infor- mation explicitly, has not been explored yet.
1 Introduction
Text image machine translation (TIMT) aims translating text in images from the source language into the target language, which has been widely used in various applications such as photo transla- tion, scene text translation, digital document trans- lation, and so on. Existing research on TIMT is mainly divided into two categories of methods: cascade method and end-to-end method. Cascade method (Hinami al., 2021; Shekar et al., 2021; Afli and Way, 2016; Chen et al., 2015; Du al., 201 1) takes a text image recognition (TIR) model for source language text recognition (Baek et al.: Shi et al., 2017, 2016; Zhang et al., 2021, 2019) and then translates them into target language texts with a machine translation (MT) model (Vaswani et al., 2017; Gehring et al., 20,,,,b; Johnson et al., 2017; Bahdanau et al., 2015; Sutskever et al., 2014; Zhao et al., 2019, 2020). To explicitly recognize the
To address the above issues, we propose a novel Cross-modal Cross-lingual Interactive Model (CCIM) for TIMT, which effectively incorporates source language recognition information into the TIMT decoder through interactive attention. The interactive decoder has two decoding modules, one for source language and the other one for target language generation. A cross-lingual interactive at- tention mechanism introduced to bridge the two language decoders. When generating translation re- sults, the target language decoder not only receives the hidden states from the encoder and previous decoded translation history but also attends to the
"Conresponding author. shanss//gtthaa..oo///rCCoogaaaaa

---

Recognition
Transittion
Result
X
Yt
Result
Brocder-eradder Cross Attention
Encoder-Decocir Decodlr Cross Attantion
Feed Forward Layer
Feed Forward Layer
Mull-HHad Sall Attention
Interaciive Attention
Interaciion Attention
Mull--Hed Self Attennttotionn
Mono lingual
Cress- lingual Cross Attention
Interactive Decoder
I X--
Y1-ma Transiation History
Cross lingual Interactive Attention
Recagrition History
(b) Weighted Cross-lingual Interactive Attention
EncodenDecoder
Encoder-Dercder
Recogrition History
Translation Hisstory
Cross-Mitention
Ag. o 1
Y2 -.. Ye-1
Interactive
Interactive Attention
Attention
Layer
Image Encoder
Multi Head Self-Attenbion
Muhi Head Self-Attention
Multi Head Self Attention
Convolutional Neural Network Modules
Xg ca X Recognition Histary
Y ue Yt-11 Translation Histrry
(a) Architecture of Cross-modal Cross-lingual Interactive Model for Text Image Translation
(c) Hierarchical Cross-lingual Interactive Attention
Figure I: (a) illustrates the proposed cross-modal cross-lingual interactive model (CCIM). (b) illustrates the weighted cross-lingual interactive attention module. (c) illustrates the hierarchical cross-lingual interactive attention module.
decoded recognition history. Our contributions are summarized follows:
MHA(Q, K, V) = Concat(heud,,..,, head,, Wo
We propose novel cross-modal cross-lingual interactive model (CCIM) for the TIMT task, which effectively enhances the translation de- coder by incorporating recognition features.
where head, Attention(QWq,.KKV KWk, VWF)
where W Wi and W represent query, key, and value projection matrices for head respectively, Wo denotes the output projection matrix.
Weighted and hierarchical interactive decod- ing strategies have been studied validate the effectiveness of interactive generation.
Self-attention (SA) The interactive decoder first calculates self-attention hidden states for both source language X and target language Y given the same query, key, and value:
Experimental results on three evaluation datasets have revealed the CCIM improves the translation quality of end-to-end TIMT models and outperforms cascade models with fewer parameters and faster decoding speed.
H MHA(X, H MHA(Y,Y,))
Then, cross-lingual interactive-attention (IA) hidden states are calculated through two language decoders. Two types of IA mechanisms utilized to recognize and translate synchronously:
2 Methodology
2.1
Cross-modal Cross-lingual Interactive Model
Weighted Interactive Attention (WIA) As
The proposed CCIM model consists of an image encoder and an interactive decoder. As shown in Figure! (a), the image encoder first extracts image features given the source language text image, then two decoders are utilized for text image recognition and translation synchronously.
shown in Figure 1 (b), the self-attention and inter- active attention are calculated separately and then weighted summation:
H H + A MHA(X, Y,Y)
H H + A WHAAYYX,XX
For image encoding, convolutional neural net- work is utilized extract image representation through multi-layer convolution and pooling op- erations (He al., 2016). While for multi-head attention (MHA), the model collects information from different positions to update the hidden state the current position (Vaswani et al., 2017):
where the query of WIA is from the corresponding language decoding history. Key and value are from the other language history.
Hierarchical Interactive Attention (HIA) To fuse self- and interactive-attentio together, a hi- erarchical calculation mechanism is introduced to

---

Synthetic TIMT
Subtitle TIMT Street-view TIMT MT Dataset
TIR Dataset
#Train
#Valid #Test
#Test
#Test
#Train
#Train
Zh En
1,000,000
2,000 2,502
1,040
1,198
5,984,287
1,000,000
En
Zh
1,000,000
2,000 2,502
1,040
5,984,287 20,895,771
1,000,000
En + De
1,000,000 2,000 2,000
1,000,000
Table 1: Statistics of text image machine translation (TIMT), machine translation (MT), and text line image recognition (TIR) datasets.
obtain interactive information through serial com- puting as shown Figure 1 (c):
language texts. The model is updated by optimiz- ing both TIR and TIMT loss functions:
H ( HY H M M HX )
(4)
L LTIR + LIIMT
[P]
CrR Σ
log vey))
where the query is from the inner-lingual self- attention results, while the key and value are from the other language self-attention features.
(6)
[dl
LTIMT Σ>
logP(vil "eliil
Encoder-Decode Cross-Attention (CA) Cross- lingual interactive-attention hidden states are fed into the encoder-decoder cross-attention mecha- nism to further incorporate encoder features into the decoder as in (Vaswani et al., 2017).
where æzj and y-j denote the recognition and translation history. M and N represent the token length of the source language and target language. Note that the interactive decoder has three atten- tion modules (self-attention, interactive attention from the other task, and mcoder-decoder attention), the decoder generates tokens given the condition of both text image, recognition history, and trans- lation history. Thus the interactive decoder has the potential to generate more accurate translation results.
CA M H1, Hr) H5* MHA(H) HI)
(5)
where H, represents the hidden states from the image encoder. H HY can be WIA or HIA for source and target languages, IT HI denote the output of the encoder-decoder cross-attention mod- ule for source and target language, respectively.
2.3 Training and Inference
The hidden states from the encoder-decoder cross-attention mechanism then further encoded by the feedforward layer obtain the interactive decoder layer outputs. Notes that the residual con- nections (He et al., 2016) and layer normaliza- tion (Ba et al., 2016) in standard transformer de- coder are also utilized after self-attention, interac- tive attention, encoder-decoder cross attention and feedforward modules in interactive decoder (Zhao al., 2023), which are not drawn in Figure for simplification.
For each decoding step during training, the teacher- forcing decoding strategy is utilized to train the parameters in the decoder in a parallel computing way and each position in the decoder can attend to all positions in the decoder up to and includ- ing that position through the attention mask. Dur- ing inference, the decoder generates both souree and target language tokens by tokens in an auto- regressive way. For each step, the two uub-bracches of the interactive decoder can attend encoder fea- tures, recognition history features, and translation history features, and predict both source and target language at the current step.
2.2 Loss Functions for Optimization
Since the interactive decoder has two decoders for the source language and target language respec- tively, TIR and TIMT tasks are optimized syn- chronously by multi-task learning. The training dataset contains triple paired samples as D (r,xi, ri)!"! where I the i-th source lan- guage text image, X is the i-th source language texts and yi the corresponding translated target
Experiments and Results
3.1 Datasets
The experiments have been conducted on a public TIMT corpus released by (Ma et al., 2022). The training set contains one million triple-paired sam- ples of source language images, source language

---

Synthetic
Subtitle
Street
Architecture
En= Zh
En
En
En-+Zh
Zh=+En
Zh-+En
CLTIR (Chen et al., 2020)
18.02
15.55
10.74
16.47
9.04
0.43
+TIR
19.44
16.31
13.52
17.96
11.25
1.74
RTNet (Su et al., 2021)
18.91
15.82
12.54
17.63
10.63
1.07
+TIR
19.63
16.78
14.01
18.82
11.50
1.93
MTETIMT(Ma al., 2022)
19.25
16.27
13.16
17.73
10.79
1.69
+MT
21.96
18.84
15.62
19.17
12.11
5.84
CCIM
22.21
19.13
15.72
19.48
12.12
5.88
Table 2: Performance of end-to-end models. All end-to-end models trained with the same TIMT training dataset. External TIR and MT corpus are also kept the same among different architecture settings.
texts, and target language translation pairs for each translation direction. The source language text im- ages in the training dataset are synthesized by using bilingual text sentences. To validate the general- ization of models, real-word (subtitle and street-view) test are uti- lized to evaluate the translation performance. The statistics of the dataset are shown Table 1.
Architecture
BLEU
Param../
Speedt
Cascade
20.46
195M
3.07
Our work: CCIM
22.21
147M
5.04
synthetic test set and two
Table 3: Comparison of cascade and end-to-end CCIM models. The unit for parameters is million (x10%) while the unit for speed is sentence per second.
3.4 Comparison with Different End-to-End TIMT Models
3.2 Experimental Settings
Image encoder in CCIM utilizes the same configu- ration in (Ma et al., 2022). The source language and target language decoder are 6-layer transformer decoder with 512-dimensional hidden sizes as in (Vaswani et al., 2017; 7; Zhao et al., 2023). The max- imum sentence length for English, German, and Chinese set to 80, 80, and 40 respectively. The preprocessed image height set to 32 and the chan- nel is 3. To align the length of the image feature and text feature, preprocessed image width is re- sized to 320. 320, and 160. The batch size set to 64, and the training step is 300,000. All models are initialized with Xavier initiation method (Glorot and Bengio, 2010) and optimized with Adam opti- mizer (Kingma and Ba, 2015) single NVIDIA V100 GPU. E (Papineni et al., 2002) is utilized for evaluation metric.
Table shows the main results on three evaluation domains. As shown Table I, CCIM outperforms the existing best multi-task based MTETIT by 0.21 BLEU scores on average. Mcanwhile, CCIM im- proves the translation performance on real-world domains by 0.12 BLEU scores on average, indi- cating the good generalization of our proposed method. Furthermore, CCIM can generate source language and target language synchronously. which can meet the requirement of both recognition and translation tasks in practical applications.
3.5 Model Size and Decoding Speed
The Cascade model deploys TIR and MT mod- els, leading to parameter redundancy and decoding delay. With an end-to-end architecture, CCIM out- performs the cascade model with fewer parameters and faster decoding speed shown in Table 3. Specifically, CCIM decreases around 24.62% pa- rameters and achieves 1.64x acceleration compared with the cascade model. Meanwhile, CCIM sig- nificantly outperforms the cascade model by 1.75 BLEU scores, which effectively alleviates the error propagation problem in the cascade model.
3.3 Baseline Models
CLTIR model is vanilla multi-task learning based TIMT model with auxiliary TIR task training (Chen et al., 2020).
RTNet bridges the TIR encoder and MT de- coder through feature transformer, which is also trained with TIR task (Su al., 2021).
MTETIMT is machine translation enhanced TIMT model, which is trained with both aux- iliary TIR and MT tasks (Ma et al., 2022).
3.6 Comparison of Different Interactive Attention Types
To validate the effectiveness of interactive atten- tion, an ablation study of replacing key and value

---

Interactive Attention Type
BLEU
Wait-k
WIA
HIA
Weighted Interactive Attention (Rand)
8.07
Wait-0
22.94
24.18
Hierarchical Interactive Attention (Rand)
11.23
Wait-1
22.99
24.32
Weighted Interactive Attention (WIA)
22.94
Wait-2
23.07
24.75
Hicrarchical Interactive Attention (HIA)
24.18
Wait-3
23.64
24.91
Wait-4
23.75
25.49
Table 4: Comparison of Various Interactive Attention
Wait-5
23.36
25.18
Types on Englishhto-Chinese validation set.
Table 5: The performance of wait-k strategy for CCIM on English-to-Chinsee Validation Set.
Recagnition Ground Trith
R. M5 #
strategy makes the recognition task decode first, enabling the translation task access source language information for improved translation qual- ity. While the wait-k strategy enhances translation quality, it does introduce some latency increase. The CCIM model achieves the best translation per- formance when k: = 4 in our experiments.
(Pinyin)
suyae palcha liahhng)
End end TIT
We need to check the process
Multii
We need to check the process
CCI
need to doable-check the need to double check the process
Translation Ground Truth
Figure Case study of end-to-end TIMT models.
in interactive attention with random samples noise vector has been implemented. Experimental re- sults Table 4 show that random noise replaced interactive attention generates a poor translation, especially for weighted interactive attention. We attribute that weighted interactive attention incor- porates noise signals through weighted summation, which severely disturbs the information flow. Fur- thermore, hierarchical interactive attention outper- forms weighted interactive attention, which reveals that flexible calculation of hierarchical architecture is better than vanilla summation operation.
Conclusion
This paper proposes novel interactive decoder based end-to-end TIMT model, which explicitly in- corporates recognized hidden states into the trans- lation process. Through the interactive attention mechanism, recognition and translation results are generated synchronously and mutually enhanced. By making full use of the source language recogni- tion information, CCIM outperforms existing end- to-end and multi-task based TIMT models on both synthetic and real-world evaluation sets. Further- more, with the end-to-end architecture, CCIM has fewer parameters and faster decoding speed than cascade models. Ablation study of different inter- active attention types shows hierarchical interac- tive attention has stronger interactive ability across recognition and translation tasks. In the future, we will explore more interactive methods for end-to- end text image machine translation.
3.7 Case Study of CCIM Model
Fig. 2 shows an example of TIMT generated by end-to-end and CCIM models. Although the end- to-end model translates the general meaning of the sentence. it ignores the meaning of "double-check" in the source language text image. Since there is no interaction during decoding, the multi-task based model also ignores this meaning. CCIM suc- cessfully translated this word through interactive attention with the source language decoder, indi- cating CCIM can effectively alleviate the problem of lacking source language information in vanilla end-to-end TIMT models.
Acknowledgements
This work has been supported by the National Nat- ural Science Foundation of China (NSFC) grants 62106265.
Limitations
3.8
Wait-k Strategy for CCIM
The wait-k strategy is commonly employed in speech translation, aiming at generating better translation given more recognition history. To vali- date the wait-k strategy in the TIMT task, we also conducted corresponding experiments as shown in Table From the experimental result, the wait-k
Our method is designed for text line images, which need preprocessing of text detection in im- ages. In the future, we will consider optimizing the text detection and translation in images jointly increase the scalability of our work.

---

References
networks. In Proceedings of the Thirteenth Inter- national Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia I Laguna Resort, Sar- dinia, Italy, May 13-15, 2010, pages 249-256.
Haithem Afli and Andy Way. 2016. Integrating opti- cal character recognition and machine translation of historical documents. In Proceedings of the Work- shop on Language Technology Resources and Tools for Digital Humanities, LT4DH@ COLING, pages 109-116.
Hao He, Qian Wang, Zhipeng Yu, Yang Zhao, Jiajun Zhang, and Chengqing Zong. 2021. Synchronous interactive decoding for multilingual neural machine translation. In Thirty-Fifth AAAI Conference Arti- ficial Intelligence (AAA!), pages 12981-12988.
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hin- ton. 2016. Layer normalization. arXiv preprint
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recogni- tion. In 2016 IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778.
Jeonghun Baek, Geewook Kim, Junyeop Lee, Sungrae Park, Dongyoon Han, Sangdoo Yun, Scong Joon Oh, and Hwalsuk Lee. What is wrong with scene text recognition model comparisons? dataset and model analysis. In 2019 IEFF/CVP International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27- - November 2, 2019, pages 4722-4722
Ryota Hinami, Shonosuke Ishiwatari, Kazuhiko Yasuda, and Yusuke Matsui. 2021. Towards fully automated manga translation. In The Thirty- AAAI Confer- ence on Artificial Intelligence (AAAT).
Melvin Johnson, Mike Schuster, Quoe V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Tho- rat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's 's multilingual neural machine translation sys- tem: Enabling zero-shot translation. Trans. Assoc. Comput. Linguistics, 5:339-351
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning align and translate. In 3rd International Conference on Learning Representations, ICLR 2015. San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings.
Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations (ICLR).
Jinying Chen, Huaigu Cao, and Premkumar Natarajan. 2015. Integrating natural language processing with image document analysis: what we learned from two real-world applications. Int. J. Document Anal. Recognit., 18(3)2235-247.
Yuchen Liu, Jiajun Zhang, Hao Xiong, Long Zhou, Zhongjun He, Hua Wu, Haifeng Wang, and Chengqing Zong. 2020. Synchronous speech recog- nition and speech-to-texx translation with interactive decoding. In The Thirty- Fourth AAAI Conference on Artificial Intelligence (AAAI), pages 8424..
Zhuo Chen, Fei Yin, Yao Zhang, Qing Yang, and Cheng-Lin Liu. 2020. Cross-lingual text image recognition via multi-task sequence to sequence learning. In 25th International Conference on Pat- tern Recognition (ICPR), pages 3122-3129.
Ziyao LU, Xiang Li. Yang Liu, Chulun Zhou, Jianwei Cui, Bin Wang. Min Zhang, and Jinsong Su. 2022. Exploring multi-stage information interactions for multi-source neural machine translation. IEEE ACM Trans. Audio Speech Lang. Process 30:562-570.
Jun Du, Qiang Huo, Lei Sun, and Jian Sun. 2011. Snap and translate using windows phone. In 2011 Interna- tional Conference on Document Analysis and Recog- nition (ICDAR), pages 809-813. IEEE Computer So- ciety.
Cong Ma, Xu Han, Linghui Wu, Yaping Zhang, Yang Zhao, Yu Zhou, and Chengqing Zong. 2023a. Modal contrastive learning based end-to-end text image ma- chine translation. IEEE/ACM Transactions Audio, Speech, and Language Processing, pages 1-13.
Jonas Gehring, Michael Auli, David Grangier, and Yann N. Dauphin. 2017a. A convolutional encoder Proceed- ings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30- August 4, Volume 1: Long Papers. pages 123-135.
model for neural machine translation.
Cong Ma. Yaping Zhang, Mei Tu, Xu Han. Linghui Wu, Yang Zhao, and Yu Zhou, 2022. Improving end-to end text image translation from the auxiliary text translation task. In 2022 26th International Con- ference on Pattern Recognition (ICPR), pages 1664- 1670.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017b. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learn- ing Research, pages 124--222.. PMLR.
Cong Ma, Yaping Zhang, Mci Tu, Yang Zhao, Yu Zhou, and Chengqing Zong. 2023b. E2timt: Efficient and effective modal adapter for text image machine translation. In Document Analysis and Recognition ICDAR 2023, pages 70-88, Cham. Springer Nature Switzerland.
Xavier Glorot and Yoshua Bengio. 2010. Understand- ing the difficulty of training deep feedforward neural

---

Cong Ma, Yaping Zhang, Mei Tu, Yang Zhao, Yu Zhou, knowl-
Weijia Xu, Yuwei Yin, Shuming Ma. Dongdong Zhang, and Haoyang Huang. 2021. Improving multilingual neural machine translation with auxiliary source lan- guages. In Findings of the Association for Compu- tational Linguistics: EMNLP 2021, Virtual Event / Punla Cana, Dominican Republic, 16-20 November, 2021, pages 3029-3041. Association for Computa- tional Linguistics.
and Chengqing Zong. 2023c. edge distillation for end-to-end text image machine translation. In Document Analysis and Recognition ICDAR 2023, pages 484-501, Cham. Springer Nature Switzerland.
Elman Mansimov, Mitchell Stern, Mia Chen, Orhan Fi- rat, Jakob Uszkoreit, and Puneet Jain. 2020. Towards end-to-end in-image neural machine translation. In Proceedings of the First International Workshop on Natural Language Processing Beyond Text, pages 70-74, Online. Association for Computational Lin- guistics.
Yaping Zhang, Shuai Nie, Shan Liang, and Wenju Liu. 2021. Robust text image recognition via adversar- ial sequence-to-sequees domain adaptation. IEEE Trans. Image Process., 30:3922-3933.
Yaping Zhang, Shuai Nie, Wenju Liu, Xing Xu, Dongx- 1nng Zhang, and Heng Tao Shen. 2019. Sequence- to-sequence domain adaptation network for robust text image recognition. In JEEE Conference on Com- puter Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 2740- 2749. Computer Vision Foundation / IEEE.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pages 311-318.
K. Chandra Shekar, Marilyn Cross, and Vignesh Vasude- van. 2021. Optical character recognition and neural machine translation using deep learning techniques. Innovations Computer Science and Engineering.
Yang Zhao, Jiajun Zhang, Yu Zhou, and Chengqing Zong. 2020. Knowledge graphs enhanced neural machine translation. Proceedings of the Twenty- Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 4039 4045. ijcai.org.
Baoguang Shi, Xiang Bai, and Cong Yao. 2017. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Trans. Pattern Anal. Mach. Intell., 39(11):2298-2300 2304.
Yang Zhao, Jiajun Zhang. and Chengqing Zong. 2023. Transformer: general framework from machine translation to others. Mach. Intell. Res., 1044):514- 538.
Baoguang Shi, Xinggang Wang, Pengyuan Lyu, Cong Yao, and Xiang Bai. 2016. Robust scene text recog- nition with automatic rectification. In 2016 IEFF Conference on Computer Vision and Pattern Recog- nition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 4 168-4176.
Yang Zhao, Jiajun Zhang, Chengqing Zong. Zhongjun He, and Hua Wu. 2019. Addressing the under- translation problem from the entropy perspective. In Artificial Intel-
The Thirty-Third AAAI Conference ligence, AAAI 2019, The Thirty-Firsi Innovative Ap- plications of Artificial Intelligncee Conference, IAAI 2019, Honolulu, Hawaii USA, January 27 February 1, 2019, pages 451-458, AAAT Press.
Tonghua Su, Shuchen Liu, and Shengjie Zhou. 2021. Rtnet: An end-to-end method for handwritten text image translation. 16th International Conference on Document Analysis and Recognition (ICDAR). pages 99 113.
Long Zhou, Jiajun Zhang, and Chengqing Zong. 2019a. Synchronous bidirectional neural machine translation. Trans. Assoc. Comput. Linguistics, pages 91-
Tlya Sutskever, Oriol Vinyals, and Quoe V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Sys- tems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Mon- treal. Quebec, Canada, pages 1112.
Long Zhou, Jiajun Zhang, Chengqing Zong, and Heng Yu. 2019b. Sequence generation: From both sides to the middle. In Proceedings of the Twenty-Eightt International Joint Conference on Artificial Intelli- gence (IJCAI), pages 5471-5477.
Barret Zoph and Kevin Knight. 2016. Multi-souree neural translation. In NAACI. HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 30-34. The Association for Computational Linguistics.
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Ilaia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, pages
Yining Wang Jiajun Zhang, Long Zhou, Yuchen Liu, and Chengqing Zong. 2019. Synchronously gener- ating two languages with interactive decoding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3348-3353.


Reviews:
[{"review_id": "mv6E8t7Qwm", "reviewer": "Reviewer_DKy3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable time in reviewing our paper with the acknowledgment of our novelty and experiments. Thank you for your concern on more details of our methods. For easier reproducibility, we will release our code to the public. Here, we address the most crucial details:\n\nQ1. On the details of Eq (6) in Section 2.2 (Loss Functions for Optimization).\n\nA1. Thanks for your consideration of the details of our work. Equation (6) shows the overall loss function by calculating both recognition and translation losses. Since the interactive decoder has 3 attention modules (self-attention, interactive attention from the other task, and encoder-decoder attention), the decoder generates tokens given the condition of both text image, recognition history, and translation history. Our code will be released to the public as introduced in the abstract to ensure good reproducibility. We will also add more technical details in our revised version to help readers understand the details of our work.\n\nQ2. On the details of each decoding step.\n\nA2. For each decoding step during training, the teacher-forcing decoding strategy is utilized to train the parameters in the decoder in a parallel computing way and each position in the decoder can attend to all positions in the decoder up to and including that position through the attention mask. During inference, the decoder generates both source and target language tokens by tokens in an auto-regressive way. For each step, the two sub-branches of the interactive decoder can attend encoder features, recognition history features, and translation history features, and predict both source and target language at the current step. The difference between weighted cross-lingual interactive attention and hierarchical cross-lingual interactive attention is the calculation method of the interactive attention. The hierarchical cross-lingual interactive attention is utilized as the final version of CCIM in Tables 1 and 2. Our code will be released to the public as introduced in the abstract to ensure good reproducibility. We will also add more technical details in our revised version to help readers understand the details of our work.\n\nQ3. On the consideration of the length difference between source and target sentences.\n\nA3. The source and target sentences indeed have different lengths. During the progress of decoding, an end-of-sentence <EOS> token is utilized as the symbol of the end of the sentence. For batch decoding, all the sentences no matter the source language and target language are supposed to generate tokens up to maximum length except for all samples that have met the <EOS> token. When the source sentence has generated <EOS> and the target language has not, the decoding procedure is continued and the interactive attention mask will mask the positions after <EOS> token in the source language so that the features at the new decoding position will attend all the decoded history and ignores the unuseful features after <EOS> position, vice versa. We will also add more technical details in our revised version to help readers understand the details of our work.\n\nQ4. On the consideration of word reordering between the source and target sentences.\n\nA4. Source and target sentences have different word orders, and this is one of the reasons that the interactive decoding strategy is useful. With the different orders of source and target sentences, both the source and target language decoder can attend to more information except for text image encoder features, thus the generation procedure can generate more accurate tokens at each step. We will add a case study about reordering analysis to help readers understand the details of our work.\n\nWe greatly appreciate your attention to the details and experimental details. We assure you that our revised version will incorporate more details to enhance readability and our code will be released to the public. Our proposed method introduces novel interactive attention for the TIMT task and significantly improves the experimental results. Reviewer #2 and Reviewer #3 both have a high acknowledgment of our work, therefore we sincerely hope you could kindly consider raising the scores for \"soundness\" and \"excitement\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.  The paper proposes a novel approach to improve text image machine translation.  \n\n2. Experimental results demonstrate the effectiveness of the proposed approach.", "reasons_to_reject": "1. The most essential parts of the method are not described in detail.  Eq（6）in Section 2.2 (Loss Functions for Optimization) and Figure 1  indicate that the interactive decoder generates a source token and a target token from the hidden states at each decoding step. \n\nHowever, for the machine translation task, the interactive decoding strategy faces the following challenges: 1) The length of the source sentence is different from that of the target sentence. 2) Word reordering between the source and target sentences.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studies the text image machine translation, which translates the source sentence in the image into the target language text. The authors propose a novel interactive decoder that generates the source and target language texts in interactive decoding strategies.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UdcSUZCdWQ", "reviewer": "Reviewer_jT2T", "summary": "", "questions": "1. What's the version of Cross-lingual Cross-Modal Attention, weighted or hierarchical. Have you tried the other way?\n2. Are the results of other work from original paper or the author re-implement them?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable time in reviewing our paper with the acknowledgment of our motivation and comparison experiments. For your suggestions and questions, we address the most crucial details as follows:\n\nQ1. On the more introduction of related work.\n\nA1. Thanks for your nice suggestion for the related work part. Following your suggestions, we assure you that we will add more details and analysis of related work in our revised version.\n\nQ2. On the details of more analysis such as attention representation and visualization.\n\nA2. Thanks for your suggestions on the clarification of interactive attention. We will add an analysis section on interactive attention to deeply analyze the effectiveness of interactive attention representation. Meanwhile, we will also follow your suggestion and visualize the interactive attention map to explain the reason behind the effectiveness of our proposed interactive method for the TIMT task.\n\nQ3. What's the version of Cross-lingual Cross-Modal Attention, weighted or hierarchical? Have you tried the other way?\n\nA3. The version of cross-lingual cross-modal Attention in Table 1 and Table 2 is the hierarchical interactive attention. We also conduct analysis experiments on the English-to-Chinese validation set as shown in Table 3 and the results show that hierarchical interactive attention outperforms weighted interactive attention. As a result, we evaluate the hierarchical interactive attention-based CCIM on the test set to compare with existing related work.\n\nQ4. Are the results of other work from the original paper or did the author re-implement them?\n\nA4. The experimental results we report for related work are sourced from the published results in [Ma et al., 2022]. We also re-produced the models in Table 1 and there are minor differences between the re-produced results and the original results in the paper. Thus we reported the original results in the paper to make it consistent with the results in published related work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation is clear. The methodology is clearly illustrated with well-drawing figures.\n2. Well comparison like with cascade and other work.\n3. Attempts with Weighted Cross-lingual Interactive Attention and Hierarchical Cross-lingual Interactive Attention.\n", "reasons_to_reject": "1. More related work should be introduced more.\n2. More analysis should be conducted, such as what does those attention represent? Can they be visualized?", "questions_to_authors": "1. What's the version of Cross-lingual Cross-Modal Attention, weighted or hierarchical. Have you tried the other way?\n2. Are the results of other work from original paper or the author re-implement them?", "paper_topic_and_main_contributions": "This paper aims at Text image machine translation (TIMT) and proposes an architecture to model visual features and cross-lingual features. Specifically, a cross-lingual cross-modal attention is used to capture image context, and a cross-lingual interactive attention is used to capture cross-lingual text context. The loss consists of both original language recognization and cross-lingual translation. Experiments show the effectiveness.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ztJ9XsbHSy", "reviewer": "Reviewer_gHG5", "summary": "", "questions": "Have you tried the wait-k decoding policy in interactive decoding as proposed in previous works?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable time in reviewing our paper with the acknowledgment of our experimental results and paper writing. For your suggestions and questions, we address the most crucial details as follows:\n\nQ1. On the novelty of the proposed cross-lingual cross-modal interactive model.\n\nA1. In fact, Compared to the interactive decoding method in speech translation and multilingual machine translation, the proposed CCIM not only blended it into the TIMT task but also innovatively introduced the hierarchical interactive attention mechanism. First, in our paper, experimental results on various translation directions and domains prove the interactive decoding strategy is effective in the TIMT task. Second, different from the previous weighted sum interactive attention mechanism in speech translation or multilingual machine translation, the proposed novel hierarchical interactive attention mechanism unifies the source and target language decoding progress and jointly calculates the hidden states in decoding, which has been proven more effective than previous works in Table 1. Therefore, we sincerely hope that the reviewers can consider raising the score for \"Excitement\". Moreover,  Table 3 shows that the hierarchical interactive attention mechanism outperforms other interactive attention methods with 1.24 BLEU highlighting the hierarchical interactive attention mechanism is effective and more suitable for the TIMT task. Thus, we sincerely hope reviewers to kindly consider increasing the score for \"Soundness\" as well.\n\nQ2. On the consideration of the wait-k decoding policy in interactive decoding. \n\nA2. Thanks for your suggestion. Indeed, the wait-k strategy is commonly employed in speech translation, and we also conducted corresponding experiments in the TIMT task as well. From the experimental result, the wait-k strategy makes the recognition task decode first, enabling the translation task to access more source language information for improved translation quality. While the wait-k strategy enhances translation quality, it does introduce some latency increase. We will provide a detailed analysis experiments of the trade-off between translation quality and decoding latency in the revised version. Thanks again for your valuable suggestion to improve the analysis part of CCIM.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method achieves promising results on the TIMT benchmark.\n- The writing is clear and easy to follow.", "reasons_to_reject": "This paper directly applies interactive attention to the TIMT task, similar to its previous applications in speech translation (Liu et al., 2020) and multilingual NMT (He et al., 2021). As a result, the novelty of the proposed method is somewhat limited.", "questions_to_authors": "Have you tried the wait-k decoding policy in interactive decoding as proposed in previous works?", "paper_topic_and_main_contributions": "This paper introduces interactive decoding into Text Image Machine Translation (TIMT), enabling synchronous generation of source and target sentences. Results on the recently proposed benchmark demonstrate that the proposed method, CCIM, outperforms both the cascaded baseline and previous TIMT models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an interactive image translation method that generates source and target language tokens jointly as a multi-task problem using cross-lingual and cross-modal attention mechanisms.
The reviewers agree with the clarity of the paper and the effectiveness of the proposed method.
There was a discussion on the novelty of the approach, but it seems to be resolved by the authors’ rebuttal.

Here is a list of the pros and cons of this paper.
* Pros
- Clear and well-written (jT2T, gHG5)
- Effectiveness against existing methods (DKy3, jT2T, gHG5)
* Cons
- Some essential technical details should be improved (DKy3)
- More analyses should be conducted (e.g., visualization of attentions) (jT2T)
Decision: Accept-Findings