In-Image Neural Machine Translation with Segmented Pixel Sequence-to-Sequence Model
Authors: Yanzhi Tian
Xiang Li
Zeming Liu
Yuhang Guo
Bin Wang
Venue: EMNLP 2023 Findings (2023)
Keywords: In-Image Machine Translation
Neural Machine Translation
TL;DR: We propose an end-to-end model for In-Image Machine Translation by encoding images to segmented pixel sequences.

Abstract: In-Image Machine Translation (IIMT) aims to convert images containing texts from one language to another. Traditional approaches for this task are cascade methods, which utilize optical character recognition (OCR) followed by neural machine translation (NMT) and text rendering. However, the cascade methods suffer from compounding errors of OCR and NMT, leading to a decrease in translation quality. In this paper, we propose an end-to-end model instead of the OCR, NMT and text rendering pipeline. Our neural architecture adopts encoder-decoder paradigm with segmented pixel sequences as inputs and outputs. Through end-to-end training, our model yields improvements across various dimensions, (i) it achieves higher translation quality by avoiding error propagation, (ii) it demonstrates robustness for out domain data, and (iii) it displays insensitivity to incomplete words. To validate the effectiveness of our method and support for future research, we construct our dataset containing 4M pairs of De-En images and train our end-to-end model. The experimental results show that our approach outperforms both cascade method and current end-to-end model.

In-Image Neural Machine Translation with
Segmented Pixel Sequence-to-Sequenee Model
Yanzhi Tian'
Xiang Li Zeming Liu"
Yuhang Guo Bin Wang
SScool of Computer Science and Technology, Beijing Institute of Technology, Beijing, China Xiiaomi AI Lab, Beijing, China
School of Computer Science and Engineering, Beihang University, Beijing, 100191, China tianyanzhi@bit. edu. en 1ixiang21@xiann com zmliu@buaa edu. cn wangbin1 10xiaomi. moniaomi.com com
guoyuhang@bit. edu.
Abstract
Incorrect
Outzut
ACCESS TO STAGT
ACCESS TO CITY
Correct Qurput
(Randering)
In-Image Machine Translation (IIMT) aims convert images containing texts from one language to another. Traditional approaches for this task are cascade methods, which uti- lize optical character recognition (OCR) fol- lowed by neural machine translation (NMT) and text rendering. However, the cascade meth- ods suffer from compounding errors of OCR and NMT, leading to decrease in translation quality. In this paper, we propose an end-to- end model instead of the OCR, NMT and text rendering pipeline. Our neural architecture adopts an encoder-decoder paradigm with seg- mented pixel sequences inputs and outputs. Through end-to-end training, our model yields improvements across various dimensions, achieves higher translation quality by avoiding error propagation, (ii) it demonstrates robust- ness for out domain data, and (iii) it displays insensitivity to incomplete words. To validate the effectiveness of our method and support for future research, we construct our dataset con- taining 4M pairs of De-En images and train our end-to-end model. The experimental results show that our approach outperforms both cas- cade method and current end-to-end model.'
Incorrect Translation
ACCESS TO STAOT
(NMT)
w/o Expict Intermediate Curputs
Incorrect
Recogrition (OCR)
Cascade Method
End End Model
ZUGANG ZUR STADT
Figure 1: The illustration of two main architectures of
IIMT. We observe that even minor enror in the OCR output can negatively affect the subsequent translation quality for the cascade IIMT method.
understand street signs, scanned documents with- out the input of text manually (Mansimov al., 2020). Compared with previous researches on im- age modality MT (Salesky et al., 2021; Ma et al., 2022; Lan et al., 2023), the most significant distinc- tion lies in the outputs of IIMT which are images containing translated texts, making users under- stand the meaning of images more easily.
IIMT is a novel machine translation task, with inherent research value and wide range of appli- cation scenarios. An intuitive method of IIM is decomposing the task into several subtasks: firstly, recognizing the text in the image, then translating the text, and rendering the translated text into the image, which is commonly referred to as the cas- cade method. Specifically, the cascade method of 1LMT usually contains three parts: (1) Recogniz- ing: recognize the text in the origin image with optical character recognition (OCR) model. (2) Translating: translate the text recognized OCR model to the target language with neural machine translation (NMT) model. (3) Rendering: eliminate text content in the original image and render the translated text.
1 Introduction
Machine Translation (MT) is an important tech- nique for achieving communication between differ- ent languages (Zhang and Zong. 2020). Recently researches on MT mainly focus on text and speech modality (Fang et al., 2022; Fang and Feng, 2023), and there is less work based on image modality MT. In In-Ime Machine Translation (IIMT) is an image- to-image MT task which aims to transform images containing texts from one language to another. The technique of IIMT enables visual text translation across different languages, which helps users to
"Corresponding Author. 'The code is available at VanzhiTian/E2E-
However, a disadvantage of most cascude meth- ods is the issue of error propagation. errors occur
com/

---

in the output of the previous stage model, it can directly impact the performance of the following models. As shown in Figure I, "STADT" in the im- age is incorrectly recognized as "STAOT" by the OCR model, and the NMT model incorrectly trans- lates it. Consequently, the incorrect translated text is rendered into the image, resulting in the incorrect output image. Besides, some of the texts in the im- ages may be incomplete because of the blocked by obstacles, resulting in the OCR model recognizing incomplete texts, thereby leading the decreasing quality NMT translations and the output images.
which can also serve as valuable resources for future research of IIM.
Task Formulation
IMMT is machine translation task where both input and output images containing texts,
HIMT: x CC RRR zU "UJRR"X''X X
(1)
where x is the input image and y is the output image, conforming
One of the methods to reduce the influence of error propagation and incomplete texts is design- ing end-to-end models instead of cascade methods. End-to-end models take into account the entire task directly and learn the transformation from input images to output images without explicit interme- diate outputs, eliminating the need for intermediate steps or stages. Building end-to-end IIMT models is full of challenge because it is hard to generate images with specific texts (Ma et al., 2023), and few IMTT dataset is publicly available. Mansimov al. (2020) propose end-to-end models for IIMT firstly. But the translation quality of their end-to- end models much lower than that of cascade methods.
Y arg max P(YIX);
(2)
where is the text in the image x and Y the translated text in the image y, and Y is the text in the decoded image.
3 Data Construction
To our knowledge, there is no publicly available dataset for IIMT. Since collecting real image pairs with aligned parallel texts for IIMT is costly, it is more practical to generate image pairs by rendering the texts from parallel corpus. Therefore, build datasets containing images with black font and white background of one-line text.
Specifically, we render texts in the parallel cor- pora into images with white background to build the image pairs, replacing the text pair (X, Y) parallel corpus to the image pair (x, v). For an image C H# we set height H = 32, and width W is proportional to the length of the text X. An image pair that contains parallel German- English (De-En) text in the IlMT dataset is shown in Figure 2.
In this paper, we aim to develop an end-to-end IIMT model which achieves better translation qual- ity than the cascade method. We regard images as pixel sequences, transforming the image-to-image generating task a pixel translation task and applying requence-to-sequeee architccture to build our end-to-end model. We also build IMTT datasets for training and validating our IIM models.
The main contributions of this paper as fol- lows:
Morges wind er varvrclen, mit der Mormecaufh zu trattienee.
Text Pair
Manrschali
Image
To the best of our knowledge, we regard the image pixel sequence, transforming IIMT into pixel seateence-t-sequeeee translation task first.
En Text: Tomoroow he will anterpt to tra n with the team.
Pair
Tomoeroo be will attempt to triin with the somn.
Figure 2: An example of image pair in IIMT dataset.
We propose an end-to-end model based on segmented pixel sequences for IIMT with an improvement translation quality compared with current end-to-end models and cascade methods.
Our method can be used to construct IIMT datasets for existing parallel corpora, making it casier to expand data. Besides, the font of texts in images can be changed by changing the font
We render texts into images using the Python library Pillow, and the font for De and En texts is Times New Ro- man.
To facilitate the study of IIMT, we construct IMM datasets to train and validate models,

---

file, and the size or color of texts in images can be changed by adjusting parameters in the program, which means our method has strong scalability.
Tomorrow he will attempt to train with the team.
We perform inspections on the constructed datasets to ensure the quality, specifically verifying the integrity of text within the images and ensuring the absence of garbled characters.
XXXXXX XXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXX XXXXXXXXXXXXX XXXXXXXXXXXXXXXXX
4 Method
57XX
PixelSequence-To--maa.
4.1 Model Architecture
Transformer (Vaswani et al., 2017) is widely used sequence-to-ssqueeee model in machine transla- tion. Recent researches (Dosovitskiy et al., 2021; Liu et al., 2021) demonstrates the efficacy of the Transformer model in the field of computer vision. These findings demonstrate the capability of the Transformer model for the IIMT task. We use vanilla Transformer the backbone to build end-to-end IIMT model as shown in Figure 3.
Transformer
XXXDXX
XOX
XX
XOXXRXOOXXXOOOOOOOOXOXXXXXXXXXXO XXXXXXXXXXXXXX
We regard images as segmented pixel sequences for 11MT. In our end-to-end IIM model, the input image is converted into pixel sequence, and then segmentation on the pixel sequence is applied to build token sequence, which is converted to beddings by the embedding layer. The embedding of the token sequence is the input of the Trans- former encoder. Another token sequence is gen- erated by auto-regressive decoding of the Trans- former decoder, and then the sequence is recovered to the output image.
Segmentation
Morgeu wird versuchen, mit der Mannschaft trainieren.
Figure 3: The architecture of our end-to-end IIMT model, including ITS, Transformer, and STI. The im- age converted into pixel sequence according to ITS, and segmented to the token sequence which is the input of the Transformer. The output of the Transformer is another token sequence that can be converted into the image with target text by STI.
4.2 Image-To-PixelSeennn
The
(ITS) refers to pro- cedure of converting images into pixel sequences which is image tokenization method:
Firs(æ): x Z BXXXX
Fiok E ERTX
Tx1
(3)
where is the input image with shape H x W C. and Ptok is the pixel sequence of x with length T.
The first step of ITS is converting RGB im- age E xx into grayscale map² ig
#x and each value in the grayscale map a float number ranging between 0 and 1. each float number is regarded as a token, the type of token is too much. To solve this problem, we can quan- tize them into several discrete buckets and each bucket is represented by character. A value in the grayscale map is converted into character
Given input image x and output image y, after applying ITS, we can regard the IIMT task as sequence-to-sequene translation task:
[Nokk]
P(yeok| Ttok, 0)
II
pluiok Kroke yroks 0), (4)
<i
where Ttok = Firs(x), Yook = Firs(y) and is the model parameters.
Converted by transforrms. Graaosalll of torchvision pack- age.

---

with the following equation:
We conduct segmentation on pixel sequences to build token sequences which can decrease the sequence length. One of the most widely used seg- mentation methods is BPE (byte pair encoding) (Sennrich et al., 2016). At each iteration of the BPE algorithm, the most frequent bi-gram in the sequences is merged together to build new token sequences. Generally, more iteration times cor- respond larger granularity and shorter token sequence.
c c= char 0.1
(5)
The function char() refers to converting the in- leger to character. For example, the integer is converted to "1", and the integer 2 is converted "2" and the integer 10 is converted "X". After applying this method, the grayscale map converted into a character map, and arrange the character map into sequence of length T, where
We explore the use BPE to pixel sequences and build token sequences which is called pixel sequence segmentation. Specifically, the most fre- quent appearance characters in the sequence will be merged at each iteration, which can increase the granularity and reduce the sequence length. Af- ter applying segmentation to pixel sequences with several times of iterations, the length of token se- quences is close text-to-text machine translation, which will have a similar computational complex- ity.
T 11 x W.
4.3 PixelSequence-To- To-eeeennee-T-I--eeee
The PixelSequence-Tov To-Image (STI) refers to a pro- cedure of converting pixel sequences into images which is image detokenization method:
EsTr (Ftok)
Ttok
R
R #xWxC
(6)
The output of the Transformer decoder is also a sequence decoded by auto regressegeeevve which can be recovered to the image using STI.
Experiments
Assume the length of the output sequence is L, and remove several characters from the end of the sequence so that its length L' is divisible by H. Reshape the output sequence character map with shape H x W where W 4. and recover the character map to the grayscale map. A character e the character map is converted into a grayscale value v with the following equation:
5.1 Datasets
De-En HMTT Dataset We build an IIMT dataset (4M image pairs) with WMT14 De-En parallel cor- pus* to train our end-to-end IIMT model. The par- allel corpus is filtered by the clean-corpus-n, of mosesdecoder with ratio 1.5, min I and max 250. Two in no-oomiin test sets are built by newstest- 2013 (3,000 image pairs) and newstest-2014 (3,003 image pairs).
== in) x 0.1
The function (nf() refers to converting the char- acter to integer. For example, "1" is converted to 1, "2" is converted 2 ..... and "X" is converted 10. Following this process, the character map is converted a grayscale map. For each value in the grayscale map v, replace it with the RGB value (u x 255, v x 255, x 255). And finally get the output image y E pilxxxxC;
addition, we build two out-of-domain test sets to evaluate the domain adaption ability of our end-to-end model. We use Ist-COMMON', which includes ASR transcriptions and translations of TED speech to build test sets in the spoken lan- guage domain (2,581 image pairs). We also the Himl 2017 test set? which includes the health information texts and translations from NHS 24 and Cochrane to build test sets the biomedical domain (1,512 image pairs).
4.4 PixelSequence Segmentation
For an image x E IR XC the length of the pixel sequence is H x W which is much larger than the length of the token sequence of text-to-text MT. The cost of the self-attention in Transformer for sequence of length N has O(N2) complexity, and the auto-regressive decoding also requires N steps to generate the total output sequence. Which means the computational effectiveness is directly impacted by the sequence length.
5.2 Metries
Since the outputs of HIMT models are images, it is hard evaluate the translation quality with images directly. In order to employ evaluation metrics for
"ht www.sstttttt..o//mtt/44t
s tht com con/mosess-sn smt /osssdecoode "https://ict fbkk eu/ huu/must-ca release- v2-0/ hhttps://ww..f. himl. aum.l.eu/eeee ee/ test-sets

---

In- In-Domain
Out-Domain
Systems
newstest-2013
newstest-2014
Ist-COMMON
Himl
BLEU
COMET
BLEU
COMET
BLEU
COMET
BLEU
COMET
Cascade
27.1
78.3
27.3
75.8
30.1
79.7
34.3
80.7
Our E2E
28.1
81.9
28.2
80.4
30.7
83.1
36.0
84.3
Table 1: Experimental results on in-domain (news) and out-domain (spoken language and biomedical) IIMT datasets.
Systems
BLEU
size is 5.
Conv Baseline (Mansimov et al., 2020)
0.5
Our E2E Our end-to-end 1MM model which is introduced in Section 4. We apply the Trans- former Big model implemented by Fairseq toolkit (Ott al., 2019) to build our end-to-end model. We use the training set learn the order of pixel quence segmentation and apply the segmentation the total dataset. The iteration times for both pixel sequences converted from German and English im- ages is 50,000. The token sequence is decoded by beam search, and the beam size is 5.
AttnConv (Mansimov et al., 2020)
7.7
Our E2E
28.1
Table Experimental results of end to-end models on n HMM dataset.
machine translation, we use OCR model to recog- nize text the image, and evaluate the translation quality with the output of OCR model and refer- ence.
5.4 Main Results
5.3 Experimental Settings
The experimental results are shown in Table 1, in- cluding the BLEU score and COMET in-domain and out-domain datasets. On both in-domain and out-domain datasets, the performance of our end-to- end model is better than the cascade method. This phenomenon is mainly caused by error propagation of the cascade method, especially the error of the OCR model.
We utilize the OCR model in PaddleOCR which is a commonly-used OCR toolkit containing mod- els of different languages to recognize texts in the generated images. We compute BLEU score (Pap- ineni al., 2002) by SacreBLEU" (Post, 2018) and COMET (Rei al., 2020) to evaluate translation quality.
We conduct experiments on the following sys- tems:
Besides, the translation quality of our end-to-end model has an improvement compared with the cur- rent end-to-end models (Conv baseline, AttnConv) (Mansimov et al., 2020), which is shown in Table 2. The reason why our end-to-end model performs better is mainly that our model uses ITS and STI. converting images into pixel sequences and apply- ing pixel sequence segmentation to enhance the utilization of textual information within images. But the current end-to-end models can not fully use the textual information in the image.
Cony Baseline An end-to-end model for 1MT (Mansimov et al., 2020) which is based on UNet architecture.
AttnConv An end-to-end model for 1IM (Mansimov et al., 2020) which contains Covo- lutional Encoder, a Convolutional Decoder, and Self-Attention Encoder.
Cascade The IIMT cascade method, includ- ing an OCR model, an NMT model, and text ren- dering, is the same the cascade method in Figure I. We use the OCR model in the PaddleOCR toolkit recognize German texts in images. The NMT model Transformer Big, trained with the same parallel corpora which are used to construct the IIMT dataset. Both German and English texts are applied 32K BPE, Beam search is applied to the decoding stage of the NMT model, and the beam
6 Analysis
In this section, we focus on the following research questions (RQs): (1) Does the segmentation of different iterations influence the translation qual- ity? (2) Compared to the cascade method, does our model has better translation quality for images with incomplete texts? (3) Does the position of texts in images influence the performance of our model? (4) Is our method effective on datasets with mul- tiple font sizes or font types? (5) How does enror
potttss:/gittie http://ghhuuuu.oom//adeeeaalllle http:////ggttoooooooppsss//ssaae /ttttps:/githhb.... wttp:ttuw...om//ubaeee/////EEEE

---

propagation influence the translation quality of the cascade method?
orgen wird
versuchien milt der Mannischafe traini
M orgen wied ed verssch en, milt der Mann Mann-shaafi schafi
zui
to nini erea.
6.1 RQ1: Translation Quality of Different Segmentation
10W
M orgen wind (vers en, mitt der Ma schof) zu
tim even.
We conduct different segmentation to pixel se- quences, building token sequences with different lengths and granularity. Figure 4 shows the relation- ship between the iteration times of pixel sequence segmentation and the BLEU score. The results demonstrate that with the increasing iteration times of segmentation, the length of the token sequence is decreased, and the granularity of the token increased.
Moc wird
versouctin, mnit der Malmischff Inalini
Morgen wird er versuchen, mit der Mannsclaft zu trainieren.
Figure Visualization of segmentation with different iteration times. The iteration times for each segmen- tation are indicated in the upper-left corner. With the increasing iteration times, the granularity the ken is increased and the length of the token sequence is decreased.
20
obstacles. We simulate this type of data to build test sets with images of incomplete texts.
620
BLEU (tewstesh-2011
BLEJ (newstest 2014)
500
Length
We randomly mask 20 20 for each image in the origin test sets build new test sets with incom- plete images, and Figure shows an example incomplete image. Experimental result in Table 3 shows that our end-to-end model has a better trans- lation quality compared with the cascade method this type of test set. The result is benefiting to approach of converting images pixel quences. Since there no need to explicitly recog- nize texts within images, our model is less affected by the problem of incomplete texts.
0LE Score
22-
Soquence Longth
20
300
200
14
100
Sagmentation Benations (x100m)
Figure 4: The relationship between BLEU score and iteration times of segmentation. Larger segmentation iterations refers to larger granularity.
We also find that pixel sequences representing images containing specific characters or subwords will be merged together as segments after several iterations, and different iteration times correspond- ing to different segmentation, shown in Figure 5, Therefore, the token contains more textual informa- tion with clearer semantic boundaries after more iteration times of segmentation, leading better translation quality.
Das Problem werde diesem Abend behobęn
Das Problem werde diesem Abend behob:
Figure 6:
An example of an incomplete image. The figure above represents the original image, while the figure below is the image after masking the areas within dashed box of the original image.
Besides, with the decreasing length of the token sequence, the decoding time is firstly decreased, However, too many iteration times cause a larger to- ken vocabulary, with more parameters in the model, which may cause longer decoding time. The numerical experimental results are shown in Ap- pendix
Incomplete texts in images affect the OCR out- puts in the cascade method and directly impact the subsequent results. We conduct experiments to calculate the WER (word error rate) of the OCR model for each type of images. For the origin images, the WER of newstest-2013 and newstest- 2014 are 1.6% and 1.4%. For the incomplete im- ages, the WER of newstest-2013 and newstest- 2014 are 3.7% and 3.6%.
6.2 RQ2: Incomplete Texts in Images
Some of the texts the images from the real world may be incomplete because of the occlusion of
"Computed by nltk. edit adit_distance()

---

newstest-2015
newstest-2014
Ist-COMON
Himl
Types
Systems
BLEU
COMET
BLEU
COMET
BLEU
COMET
BLEU
COMET
Cascade
27.1
78.3
27.3
75.8
30.1
79.7
34.3
80.7
Origin
Our E2E
28.1
81.9
28.2
80.4
30.7
83.1
36.0
84.3
Cascade
24.2
71.2
24.5
69.1
26.4
71.7
30.5
72.4
RQ2
Our E2E
25.0
72.3
24.7
71.6
26.9
72.1
31.1
73.6
Cascade 27.1
78.3
27.3
75.8
30.1
79.7
34.3
80.7
RQ3
Our E2E
27.8
81.5
27.8
80.0
30.5
82.7
35.5
84.1
Table 3: Experimental results on different types of test sets, including the origin test sets, the test sets with incomplete texts images (RQ2) and the test sets with different positions of texts in images (RQ3).
6.3 RQ3: Influence of Texts Position in Images
TNRI5 is constructed with Times New Roman, and the font size is 15. The TNR25 is constructed with Times New Roman, and the font size is 25. The Arial20 is constructed with Arial, and the font size is 20.
Since the OCR model usually contains a text detec- tion module, the cascade method not sensitive to the position of texts images. However, our end-to-end model does not conduct OCR and text detection explicitly.
Original Dataset: Times New
Problem werde diesem Abend behoben.
To validate whether our model is influenced by the texts position in images, render texts with shifting position compared with the origin test sets, and Figure 7 shows an example of an image containing text with shifting position.
TNR15: Times New Roman; Font Size 15
Problns dicsem Abesd bchobrn.
TNR25: Times New Roman; Font Size 25
Das Problem werde diesem Abend behoben.
Das Problem werde diesem Abend behoben.
Arial20: Arial; Font Size 20
Das Problem werde an diesem Abend behoben.
Das Problem werde an diesem Abend behoben.
Figure 8: Datasets with multiple font sizes and font types.
Figure example of an image with shifting position of text. The figure above represents the original image, while the text in the figure below is vertically and hori- zontally shifting by pixels.
We train several models with these datasets, and compute the BLEU score with the method in Section 5.2. The BLEU scores are shown in Table The BLEU scores of TNR25 and Arial20 are similar to the results in Table I, indicating that our method is still effective on datasets with multiple font sizes and font types.
As the experimental results shown in Table 3, although there is certain decrease in translation quality, it is still obtaining better translation quality compared with the cascade method. This result is attributed our ITS method. For the pixel se- quence of an image with different position of text, is equivalent to adding a prefix with character "X" representing several white pixels, and the length the prefix dependent on the shifting distance text. This means the rest of the pixel sequence remained unchanged and different position of text in the image influences ITS slightly, leading to the minimal decrease of translation quality.
Datasets
newstest-2013
newstest-2014
TNR15
22.3
22.6
TNR25
27.5
27.7
Arial20
27.9
28.1
Table BLEU scores of models trained on different datasets.
However, the BLEU score of TNR15 is much lower than that in Table and the reason is that the WER the target-side OCR model is higher while recognizing English texts with smaller font
6.4
RQ4: Mutiple Font Sizes and Font Types
We also construct IIMT datasets with multiple font sizes and font types, as shown in Figure 8. The

---

newstest-2013
newstest-2014
Systems
BLEU
COMET
BLEU
COMET
Golden NMT
29.8
83.0
29.7
81.5
Cascade
27.1 (-2.7) 78.3 (-4.7)
27.3 (-2.4)
75.8 (-5.7)
Ist-COMMON
Himl
BLEU
COMET
BLEU
COMET
Golden NMT
33.0
84.0
37.8
84.7
Cascade
30.1 (-2.9)
79.7 (-4.3)
34.3 (-3.5)
80.7 (-4.0)
Table 5: Comparison of translation quality of golden NMT (w/o OCR error) and cascade method.
size. The WER of newstest-2013 and newstest- 2014 with font size 20 (same setting as Table 1) are 1.4% and I. 1.6%. The WER of newstest-2013 and ncwstcst-2014 with font size are 4.2% and 4.0%.
method the text such BPE will give wrong subwords, which will damage the performance of NMT lot.
An example of error propagation in the cascade method is shown in Table 6. The OCR model recog- nizes the word "außerdem" as "auBerdem", and the BPE result changes from "auBerdem" to "au @ Ber@ @ dem", resulting in the wrong translation "Berdem".
6.5 RQ5: Error Propagation of Cascade Method
In our experiments, observe that the OCR model may recognize some characters incorrectly. For example, the character 'ü' may be recognized as and 'B' as 'B' because of their similar shapes. The incorrectly recognized results are transferred to following stages, leading to eтror propagation of the cascade method.
The result of our input image is shown in Table 7, and we can ob- serve that our end-to-end model mitigates the prob- lem of error propagation in the cascade method.
model with the same
Related Work
To analyze the influence of error propagation, we first use the OCR model to recognize images con- taining German texts in the datasets, and then cal- culate WER of recognized results. For images built by newstest-2013 and newstest-2014, the WER is 1.6% and 1.4% separately.
7.1 In-Image Machine Translation
There are mainly two difficulties to build an end-to- end model for IIMT: Firstly, there is no dataset for 11TT publicly available. Secondly, although there are several researches on image synthesis such as UNet (Ronneberger et al., 2015), GAN (Goodfel- low et al., 2014), VAE (Kingma and Welling, 2022), Diffusion Models (Ho al., 2020), and the Sta- ble Diffusson Model (Rombach et al., 2022) has made great progress generating high- high-resolution images, all them have limitation on generating text within images. Liu et al. (2022) and Ma et al. (2023) focus on image synthesis with characters, and achieve an improvement compared with the previous models. But there is still certain gap implement end-to-end IIMT.
We also compute the BLEU score and COMET of the NMT model using the origin text as input (namely golden NMT), which is without errors caused by the OCR model. In order to mitigate the influence of OCR errors on the evaluation of the cascade method, we render the output of the golden NMT into images with the same method and recognize the texts using the same OCR. As shown in Table 5, both the BLEU score and COMET of the cascade method are lower than the golden NMT because of the errors the outputs of the OCR model.
The reason why errors in OCR outputs influence the performance of the NMT model obviously is because the OCR model recognizes the text with character level which means there may be an error occurring with one letter in word while other letters in the same word are correct. It will result in the word with the wrong letter and the tokenization
Current end-to-end models (Mansimov et al., 2020) include Conv baseline and an AttnConv. The Conv baseline is based on UNet architecture, which can only generate the image with one two words, and quickly devolves into noise. The AttnConv contains a Convolutional Encoder, Con- volutional Decoder, and a Self- Self-Attention Encoder.

---

Cascade Method (w/ OCR error)
Golden (w/o OCR error)
Input Image
Die Täter schossen außerdem auf eine Bushaltestelle.
Die Täter schossen auBerdem auf eine Bushaltestelle.
Die Täter schossen außerdem auf eine Bushaltestelle.
OCR
Die Täter scho@ @ ssen @ @ Ber@ @ dem auf eine Bushaltestelle. The perpetrators shot Berdem
Die Täter scho @ ssen auBerdem auf eine Bushaltestelle.
BPE
The perpetrators also
NMT
on a bus stop
shot at a bus stop..
Output Image
The perpetrators shot Berdem bus stop..
Table 6: Example of error propagation of cascade method, the golden result is generated by NMT model using the original text (w/o OCR error).
Input Image
Die Täter schossen auBerdem auf eine Bushaltestelle.
Output Image of
The perpetrators shot Berdem bus
Cascade Method
Output Image of
The perpetrators also shot a bus stop.
Our End-to-End Model
Table 7: Comparison of the output of the cascade method and our end-to-end model.
7.2 Text Image Translation
be split into the smallest unit and the bi-gram will be merged according to the BPE code list in turn. Recent work explores the use of the BPE technique to image modality (Razzhigaev et al., 2022). They apply BPE on pixel sequences of iGPT (Chen et al., 2020), increasing the performance and efficiency.
Text Image Translation (TTT) aims to translate im- ages containing source language texts into the tar- get language, which an image-to-text machine translation task. Compared with IIM,, TIT is mul- timodal machine translation (MMT) task (Elliott et al., 2016; Calixto et al., 2017; Ive et al., 2019; Zhang et al., 2020). TIT primarily contains two categories of methods, namely cascade methods including OCR and NMT, and end-to-end models. Cascade methods of TIT also suffer from error propagation, one of the methods is to associate images with relevant texts, providing useful sup- plementary information for translation (Lan et al., 2023). Another method is to design end-to-end models (Ma et al., 2022) which mainly contain vi- sion encoder such as ResNet (He et al., 2016) and text decoder.
8 Conclusion
In this paper, we propose an end-to-end model for IIMT based on segmented pixel sequences to re- duce the influence of error propagation. Exper- imental results on 1IM datasets show that our model has better translation quality compared with both the current end-to-end model and the cas- cade method. Furthermore, to demonstrate our end-to-end model has better performance on different types of IIMT datasets. In future work, we will try to build more realistic IIMT datascts, and design new model architecture to achieve better translation quality.
conduct analysis
7.3 Byte Pair Encoding (BPE)
BPE (Sennrich et al., 2016) mainly contains two stages, learning BPE and applying BPE. During learning BPE, the tokens will be split into the small- est unit, and the most frequent bi-gram is merged new token. This process usually involves mul- tiple iterations and finally results in a BPE code list that contains n pairs of bi- bi-gram after n itera- tions. During applying BPE, the tokens will also
Limitations
While in in-image machine translation is new MT task, and our end-to-end model outperforms the previous methods, this work has certain limitations. Firstly, our work is preliminary study on in-image machine translation (c.g. images with black font and white background, converting RGB images to

---

grayscale maps). And the Image-Too-PixeSSquenne phase of our end-to-end model requires a learning segmentation method based on pixel sequences, thereby demanding substantial CPU computational resources. Besides, the application of the segmen- tation method also requires a certain computational cost. Consequently, there is no improvement in decoding time compared to the cascade methods. Furthermore, it is hard to combine our method with vision models. The imput images of recent vision models (e.g. Vision Transformer) are fixed size, but the images in our dataset are different sizes, which need to crop the images. However, the Vision Transformer is still worth trying. In the future, we will try to combine our method with vision models, and take both the original images and the OCR results as inputs.
Desmond Elliott, Stella Frank, Khalil Sima' an, and Lu- cia Specia. 2016. Multi3OK: Multilingual English- German image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pages 70- 74, Berlin, Germany. Association for Computational Linguisties.
Qingkai Fang and Yang Feng, 2023. Understanding and bridging the modality gap for speech translation.
Qingkai Fang, Rong Ye, Lei Li, Yang Feng, and Mingxuan Wang. 2022. STEMM: Self-learning with seet manifold mixup for speech translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Vooluee I: Long Papers), pages 7050-7062, Dublin, Ireland. Association for Computational Linguisties.
Ian J. Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farlcy, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial networks.
Ethics Statement
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
This paper proposes an end-to-end IIMT model. We take ethical considerations seriously and ensure that the methods used in this study are conducted in a responsible and ethical manner. Our IIMT datasets are built by publicly available parallel cor- pora, which are used to support research of IIMT.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. De- noising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840- 6851.
Acknowledgements
Julia Ive, Pranava Madhyastha, and Lucia Specia. 2019. Distilling translations with visual awareness. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguisties, pages 6525- 6538, Florence, Italy. Association for Computational Linguistics.
We thank all the anonymous reviewers for their insightful and valuable comments. This work is supported by the National Key R&D Program of China (No. 2020AAA0106600).
Diederik P Kingma and Max Welling. 2022. Auto- encoding variational bayes.
References
lacer Calixto, Qun Liu, and Nick Campbell. 2017. Doubly-attentive decoder for multi-modal neural ma- chine translation. Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1913-1924, Vancouver, Canada. Association for Computational Linguistics.
Zhibin Lan, Jiawei Yu, Xiang Li, Wen Zhang, Jian Luan,
Bin Wang, Degen Huang, and Jinsong Su. 2023. Ex- ploring better text image translation with multimodal codebook.
Rosanne Liu, Dan Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, RJ Mical, Mohammad Norouzi, and Noah Constant, 2022. Character-aware models improve visual text rendering.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. 2020. Generative pretraining from pixels. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1691-1703. PML.R
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10012-10022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image worth 16x16 words: Transformers for image recognition at scale.
Cong Ma, Yaping Zhang, Mei Tu, Xu Han, Linghui Wu, Yang Zhao, and Yu Zhou. 2022. Improving end- to-end text image translation from the auxiliary text translation task.

---

Jian Ma, Mingjun Zhao, Chen Chen, Ruichen Wang. Di Niu, Haonan Lu, and Xiaodong Lin. 2023. Glyph- draw: Learning to draw chinese characters in image synthesis models coherently.
Dominican Republic. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Pupers), pages 1715-1725, Berlin, Germany. Association for Computational Lin- guistics.
Elman Mansimov, Mitchell Ster,, Mia Chen, Orhan Fi- rat, Jakob Uszkoreit, and Puneet Jain. 2020. Towards end-to-end in-image neural machine translation. In Proceedings of the First International Workshop on Natural Language Processing Beyond Text, pages 70-74, Online. Association for Computational Lin- guistics.
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need, Advances in neural information processing systems, 30.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: fast, extensible toolkit for sequence modeling. In Procedinggs of NAACL-HIL 2019: Demonstrations.
Jiajun Zhang and Chengqing Zong. 2020. Neural ma- chine translation: Challenges, progress and future.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- lational Linguistics, pages 31 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Zhuosheng Zhang, Kehai Chen, Rui Wang, Masao Utiyama, Eiichiro Sumita, Zuchao Li, and Hai Zhao. 2020. Neural machine translation with universal vi- sual representation. In International Conference on Learning Representations.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Belgium, Brussels. Association for Computa- tional Linguistics.
Anton Razzhigaev, Anton Voronov, Andrey Kaznacheev, Andrey Kuznetsov, Denis Dimitrov, and Alexan- der Panchenko. 2022. Pixel-level BPR for auto- regressive image generation. In Proceedings of the First Workshop on Performance and Interpretubility Evaluations of Multimodal, Multipurpose, Massive- Scale Models, pages 26-30, Virtual. International Conference on Computational Linguistics.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685-2702, Online. Association for Computational Linguistics.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Fiser, and Björn Ommer. 2022. High- resolution image synthesis with latent diffusion mod- cls. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation.
Elizabeth Salesky, David Euter, and Matt Post. 2021. Robust opea-voca translation from visual text representations. In Proceedings of the 2021 Confer- ence on Empirical Methods Natural Language Pro- cessing, pages 7235-7252, Online and Punta Cana,

---

BLEU
Decoding Time (s)
Iterations
Avg. Len (De/En)
newstest-2013
newstest-2014
newstest-2013
newstest-2014
1,000
679/429
15.1
13.4
2935
3344
2,000
257/190
23.5
22.4
1094
1074
3,000
137/112
25.3
24.9
566
647
4,000
102/86
25.9
25.2
449
476
5,000
86/74
26.4
25.8
368
456
6,000
76/66
26.8
26.0
334
323
7,000
70/61
26.7
26.0
311
320
10,000
60/52
26.8
26.5
260
266
30,000
42/37
27.9
27.7
244
246
50,000
37/33
28.1
28.2
187
201
100,000
33/30
27.4
27.1
214
211
Table 8: Numerical experimental results of RQ1.
Numerical Experimental Results of RQ1
The numerical experimental results are shown in
Table 8, and the experiments of decoding time conducted on Tesla V100 GPU with batch 10.


Reviews:
[{"review_id": "qXqvw7mtKz", "reviewer": "Reviewer_ApGL", "summary": "", "questions": "1. In the task formulation section, what's the meaning of \\hat{Y}? Although the readers might guess the meaning of \\hat{Y}, the authors should clarify to eliminate misunderstandings.\n2. The data construction in this paper just renders texts in the parallel corpora into images with white background, which is quite easy data distribution of text images. The authors mention their method has strong scalability, I wonder whether authors conduct any experiments on other data distribution of text images like changing the font, font colors, background colors, and background images?\n3. What's the method of converting an RGB image into a grayscale map in your implementation (Line 186-188 in the paper)? There are multiple methods of converting an RGB image into a grayscale map, what's your implementation? PIL.image.convert(“L”) or pixel averaging through channels or any other methods? If the channel conversion is conducted using open-source packages or through existing methods, please provide the conversion formula, reference citation, or footnote with a reference link to the open-source package.\n4. Since the dataset in your work is images with black font and white background of the one-line text. Is the RGB image necessary and do you conduct any ablation study?\n5. Figure 5 shows the Visualization of segmentation with different iteration times. Each segmentation results have several missing pixels at the bottom-right corner, what’s the reason for these missing pixels? Furthermore, What about the robustness of the proposed segmentation method? When there are Gaussian noise, Salt-and-pepper noise, and other pixel noises, what are the influences of the segmentation results? Have you conducted any analysis experiments? Experiments and analysis should be conducted in the paper to show the generalization of the proposed pixel sequence segmentation. \n6. In sub-section 6.4, there is no comparison and analysis of end-to-end models with cascade models. As so, error propagation indeed exists in cascade methods, and can your proposed end-to-end methods address this problem? What's the relationship between the analysis of error propagation of the cascade method with your proposed end-to-end method? Furthermore, the case study in Table 5 lacks the results of your proposed end-to-end model, which cannot prove your end-to-end model can reduce the influence of error propagation.\n7. For image encoding and decoding, the pixel sequence transformation and pixel sequence segmentation might be one of your contributions. However, there is no comparison with vanilla patch-based image encoding and decoding methods, which is a quite vital baseline for your work.\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments, and here are our responses.\n\n**Reason To Reject 1:**\n\n> The description of the pixel sequence-based in-image machine translation model is unclear, such as 1) how to transform the image into pixel sequence is incomplete, 2) the method of transforming RGB to grayscale map is unclear, and 3) the training and inference details are missing, and the optimization object is not mentioned.\n\n1)At Line 202 - Line 203, we describe the character map that is converted into a sequence of length HxW. We may not describe this process clearly. Actually, the character map is organized as a sequence from top to bottom and then from left to right, and the character map with a size of HxW is converted into the 1-D sequence of length 1xT (T=HxW).  Thanks for your suggestion, and we have described this step more clearly in the new version. \n\n2)We do not introduce the detailed method. We use transforms.Grayscale() of torchvision package to convert the RGB image into the grayscale map. Thanks for your suggestion, and we have added it in the new version.\n\n3)We may not describe it clearly at Line 304 - Line 313. Our method of transforming the image into pixel sequence modeling the image as sequence, and the widely used method of seq2seq task can be used, such as the model (Transformer Big), the loss function (Cross Entropy) and decoding method (Beam Search). Thanks for your suggestion, and we have added more detail at Line 304 - Line 313 in our new version.\n\n**Reason To Reject 2:**\n\n> The constructed dataset in this paper is quite simple, which just considers the black font and white background image. As so, the experiments and analysis are based on a toy task in the in-image translation area, which has limited influence in real-world applications. Various text image effects should be considered in ablation studies to show the generalization of the proposed methods.\n\nIn-image machine translation is a new MT task, which requires images as outputs, exploring the modality of MT. And the research of end-to-end in-image machine translation is important, but there is few related work [1].\n\nTo the best of our knowledge, there is no public available dataset for in-image machine translation, and our dataset serves as a good test-bed for initial attempts. Our work is a preliminary study on in-image machine translation, and we achieve better translation quality than cascade method in this task. \n\nOur dataset is close to real-world scenarios that contain a lot of text, such as webpage screenshots and scanned documents. We have constructed datasets with different fonts and font sizes, and our end-to-end models reach similar results. We will add the results in our new version.\n\nWe hope our method and dataset can facilitate the study of in-image machine translation, and there will be more further research.\n\n**Reason To Reject 3:**\n\n> There are certain problems with paper writing. Some explanations for variables and symbols are missing. Thus, this paper should take a major revision before publication.\n\nThanks for your suggestion, we have proofread and revised the manuscript thoroughly according to your constructive comments, including but not limited to:\n\n- We describe the variable more clearly in task formulation (Line 122).\n- We add the description of the variable x_{tok} of Equation 3 (Line 179).\n- We introduce the method of converting the RGB image into grayscale map (Line 187 - Line 188).\n- We fix the typo in Equation 5 (Line 196), which should be c=char(\\lfloor \\frac{v}{0.1} \\rfloor).\n- We introduce the step of converting the character map to sequence in detail (Line 202 - Line 203). \n- We add the detailed explaination of Figure 5.\n\n**Question 1:**\n\n> In the task formulation section, what's the meaning of \\hat{Y}? Although the readers might guess the meaning of \\hat{Y}, the authors should clarify to eliminate misunderstandings.\n\nThe definition of variable \\hat{Y} is the target text in the decoded image, which is similar to the definition of the MT or seq2seq tasks. Thanks for your suggestion, we have added the explanation of the variable in the new version to eliminate misunderstandings. \n\n**Question 2:**\n\n> The data construction in this paper just renders texts in the parallel corpora into images with white background, which is quite easy data distribution of text images. The authors mention their method has strong scalability, I wonder whether authors conduct any experiments on other data distribution of text images like changing the font, font colors, background colors, and background images?\n\nWe also construct datasets with different font types and font sizes, and our end-to-end models reach similar results, and we will add the results in our new version. \n\n**Question 3:**\n\n> What's the method of converting an RGB image into a grayscale map in your implementation (Line 186-188 in the paper)? There are multiple methods of converting an RGB image into a grayscale map, what's your implementation? PIL.image.convert(“L”) or pixel averaging through channels or any other methods? If the channel conversion is conducted using open-source packages or through existing methods, please provide the conversion formula, reference citation, or footnote with a reference link to the open-source package.\n\nWe use transforms.Grayscale() of torchvision package to convert the RGB image into the grayscale map. Thanks for your suggestion, it is better to introduce more detail, and we have added a footnote at Line 188 where we firstly mention grayscale map. \n\n**Question 4:**\n\n> Since the dataset in your work is images with black font and white background of the one-line text. Is the RGB image necessary and do you conduct any ablation study?\n\nWe construct images with black font and white background. However, the images are not binarized (only black and white pixels). The shape of images in our constructed dataset is HxWxC (C=3), and we call it RGB image in our paper, and then the RGB image is converted into the grayscale map with shape HxWxC (C=1).\n\n**Question 5:**\n\n> Figure 5 shows the Visualization of segmentation with different iteration times. Each segmentation results have several missing pixels at the bottom-right corner, what’s the reason for these missing pixels? Furthermore, What about the robustness of the proposed segmentation method? When there are Gaussian noise, Salt-and-pepper noise, and other pixel noises, what are the influences of the segmentation results? Have you conducted any analysis experiments? Experiments and analysis should be conducted in the paper to show the generalization of the proposed pixel sequence segmentation.\n\nThe pixels at the bottom-right corner are not missing, and they are in the next segmentation of the pixel sequence. The smallest unit of a segment is a pixel, and the length of a segment may not be an integer multiple of the height.\n\nTake 5000 iteration in the Figure 5 as an example, the first segmentation containing the image of 'M', and the pixels at the botton-right corner of 'M' are in the second segmentation (containing the image of 'or').The segmentation is learned using the method introduced in sub-section 4.4, and the visualization is the origin segmentation result without any edition.\n\nWe conduct experiments on images with pixel noises, and the results show that pixel noises will influence the performance of segmentation. However, after applying denoising, the performance recovers. \n\nThanks for your comment, we have added detailed explaination of Figure 5.\n\n**Question 6:**\n\n> In sub-section 6.4, there is no comparison and analysis of end-to-end models with cascade models. As so, error propagation indeed exists in cascade methods, and can your proposed end-to-end methods address this problem? What's the relationship between the analysis of error propagation of the cascade method with your proposed end-to-end method? Furthermore, the case study in Table 5 lacks the results of your proposed end-to-end model, which cannot prove your end-to-end model can reduce the influence of error propagation.\n\nThe BLEU score and COMET score of our end-to-end model are higher than the cascade method on test sets (Table 1), which demonstrates the end-to-end model has a better translation quality. The error propagation of cascade method is between OCR and NMT models (sub-section 6.4). Specifically, the error in the OCR results will have an influence on the BPE results, and will damage the NMT performance furthermore.  However, for an end-to-end model, there is no passing of results and cascading of models, which does not have the phenomenon of error propagation. \n\nWe also check the output image of our end-to-end model with the input image of Table 5, which contains *\"The perpetrators also shot a bus stop.\"*. Compared with the output of the cascade method (*\"The perpetrators shot* *Berdem* *on a bus stop .\"*), our end-to-end model mitigates the error propagation.\n\nThanks for your comment, we have added the output of our end-to-end model to Table 5. \n\n**Question 7:**\n\n> For image encoding and decoding, the pixel sequence transformation and pixel sequence segmentation might be one of your contributions. However, there is no comparison with vanilla patch-based image encoding and decoding methods, which is a quite vital baseline for your work.\n\nThe patch-based model (Vision Transformer, ViT) is a widely used vision model, which is a vital baseline. The previous work [1] used vision model (CNN), and the performance of our model is better than their model (Table 2).\n\nFurthermore, the shape of our input images is not fixed (HxWxC, W is proportional to the length of the text), and patch-based model and other conventional vision models usually take image with fixed shape as input (for example, the shape of input images of [1] is 32x1024). Thanks for your reminder, we will add this baseline in the new version.\n\n**Missing References:**\n\n> Path-based image encoding should be considered as a vital baseline for this paper: [1] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021\n\nThanks for your reminder, we have added it in our new version.\n\n**Presentation Improvements:**\n\n> What's the meaning of T in Equation (3)? Meanwhile, the detailed clarification of function F_{ITS}(x) is missing. Subsection 4.2 only introduces 1) transforming the RGB image into a grayscale map and 2) quantizing values in the grayscale map into a character map. I believe that there must have a step 3) transforming character map into pixel sequences but it is missing in the paper. If there is no step 3), the character map of length H\\times W (Line 203 in the paper) is different from x_{tok}\\in\\mathbb{R}^{T\\times 1} as in Equation (3).\n\nT is the length of pixel sequence, which is equal to HxW. x is the input RGB image, and x_{tok} is the pixel sequence corresponding to x, F_{ITS}(x) is the mapping form x to x_{tok}.  In sub-section 4.2, we describe the process of mapping. \n\nLike you said, there is a step transforming character map into pixel sequence (step 3). At Line 202-Line 203, we describe the character map that is converted into a sequence of length HxW.  We may not describe this step clearly. Specifically, the character map is organized as a sequence from top to bottom and then from left to right, and the character map with a size of HxW is converted into the 1-D sequence of length T (T=HxW). Thanks for your question, and we have described this step more clearly in the new version.\n\nThanks for your comments to help us improve our work, and we will continue to polish the paper to make it better.\n\n[1] Mansimov E, Stern M, Chen M, et al. Towards end-to-end in-image neural machine translation[J]. arXiv preprint arXiv:2010.10648, 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experiment results show the proposed method can outperform cascade model on in-domain and out-domain test sets.", "reasons_to_reject": "This paper has several shortcomings that should be rejected:\n1. The description of the pixel sequence-based in-image machine translation model is unclear, such as 1) how to transform the image into pixel sequence is incomplete, 2) the method of transforming RGB to grayscale map is unclear, and 3) the training and inference details are missing, and the optimization object is not mentioned.\n2. The constructed dataset in this paper is quite simple, which just considers the black font and white background image. As so, the experiments and analysis are based on a toy task in the in-image translation area, which has limited influence in real-world applications. Various text image effects should be considered in ablation studies to show the generalization of the proposed methods.\n3. There are certain problems with paper writing. Some explanations for variables and symbols are missing. Thus, this paper should take a major revision before publication.\n", "questions_to_authors": "1. In the task formulation section, what's the meaning of \\hat{Y}? Although the readers might guess the meaning of \\hat{Y}, the authors should clarify to eliminate misunderstandings.\n2. The data construction in this paper just renders texts in the parallel corpora into images with white background, which is quite easy data distribution of text images. The authors mention their method has strong scalability, I wonder whether authors conduct any experiments on other data distribution of text images like changing the font, font colors, background colors, and background images?\n3. What's the method of converting an RGB image into a grayscale map in your implementation (Line 186-188 in the paper)? There are multiple methods of converting an RGB image into a grayscale map, what's your implementation? PIL.image.convert(“L”) or pixel averaging through channels or any other methods? If the channel conversion is conducted using open-source packages or through existing methods, please provide the conversion formula, reference citation, or footnote with a reference link to the open-source package.\n4. Since the dataset in your work is images with black font and white background of the one-line text. Is the RGB image necessary and do you conduct any ablation study?\n5. Figure 5 shows the Visualization of segmentation with different iteration times. Each segmentation results have several missing pixels at the bottom-right corner, what’s the reason for these missing pixels? Furthermore, What about the robustness of the proposed segmentation method? When there are Gaussian noise, Salt-and-pepper noise, and other pixel noises, what are the influences of the segmentation results? Have you conducted any analysis experiments? Experiments and analysis should be conducted in the paper to show the generalization of the proposed pixel sequence segmentation. \n6. In sub-section 6.4, there is no comparison and analysis of end-to-end models with cascade models. As so, error propagation indeed exists in cascade methods, and can your proposed end-to-end methods address this problem? What's the relationship between the analysis of error propagation of the cascade method with your proposed end-to-end method? Furthermore, the case study in Table 5 lacks the results of your proposed end-to-end model, which cannot prove your end-to-end model can reduce the influence of error propagation.\n7. For image encoding and decoding, the pixel sequence transformation and pixel sequence segmentation might be one of your contributions. However, there is no comparison with vanilla patch-based image encoding and decoding methods, which is a quite vital baseline for your work.\n", "paper_topic_and_main_contributions": "This paper proposes an end-to-end image-to-image machine translation model. The main contribution of this paper is using a pixel sequence-based method to transform source language text images into target language text images.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Path-based image encoding should be considered as a vital baseline for this paper:\n[1] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021\n", "Typos_Grammar_Style_and_Presentation_Improvements": "What's the meaning of T in Equation (3)? Meanwhile, the detailed clarification of function F_{ITS}(x) is missing. Subsection 4.2 only introduces 1) transforming the RGB image into a grayscale map and 2) quantizing values in the grayscale map into a character map. I believe that there must have a step 3) transforming character map into pixel sequences but it is missing in the paper. If there is no step 3), the character map of length H\\times W (Line 203 in the paper) is different from x_{tok}\\in\\mathbb{R}^{T\\times 1} as in Equation (3)."}}, {"review_id": "6362vmlW59", "reviewer": "Reviewer_6iiA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments, and here are our responses.\n\n**Reason To Reject 1:**\n\n> My main concern is that the constructed dataset is rather artificial and can have limited application scenarios. Specifically, they directly render texts into images with white background, which cannot well represent the diverse real-world scenarios.\n\nOur dataset is close to real-world scenarios that contain a lot of text, such as webpage screenshots and scanned documents. We have constructed datasets with different fonts and font sizes, and our end-to-end models reach similar results. We will add the results in our new version.\n\nTo the best of our knowledge, there is no public available dataset for in-image machine translation, and our dataset serves as a good test-bed for initial attempts. Our work is a preliminary study on in-image machine translation, and we achieve better translation quality than cascade method in this scenario. We hope our method and dataset can facilitate the study of in-image machine translation, and there will be more further research.\n\n**Reason To Reject 2:**\n\n> Some experimental results can be questionable and do not provide readers with much insight. For example, I am not sure why would their method can be better at translating incomplete texts in images compared to pipeline methods.\n\nFor images with incomplete texts, some words in the images are incomplete which will influence the output of OCR model, and we conduct experiments to calculate the WER of OCR model for each type of images. For the origin images, the WER for newstest-2013 and newstest-2014 are 1.6% and 1.4%. For the incomplete images, the WER for newstest-2013 and newstest-2014 are 3.7% and 3.6%. More errors in the OCR output influence the BPE output more seriously, which will damage the NMT performance a lot.\n\nFor our end-to-end model, there is a decrease in translation quality, but it is less affected than the cascade method. The experiment results are shown in Table 3, RQ2. Thanks for your question, and we have added more detailed analysis at sub-section 6.4.\n\n**Reason To Reject 3:**\n\n> It would be better to include a model that take both the original image and the OCR results as inputs.\n\nThanks for your suggestion, we will try to conduct experiments on this model, and compare with our method.\n\nThanks for your comments to help us improve our work, and we will continue to polish the paper to make it better.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Building end-to-end in-image machine translation models is technically sound and can alleviate the error propagation issue.\n2. The constructed dataset can be helpful for future research.\n3. The idea is simple and effective.", "reasons_to_reject": "1. My main concern is that the constructed dataset is rather artificial and can have limited application scenarios. Specifically, they directly render texts into images with white background, which cannot well represent the diverse real-world scenarios.\n2. Some experimental results can be questionable and do not provide readers with much insight. For example, I am not sure why would their method can be better at translating incomplete texts in images compared to pipeline methods.\n3. It would be better to include a model that take both the original image and the OCR results as inputs.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper focuses on in-image neural machine translation. Unlike previous cascade approaches that first perform OCR, then do text translation and text rendering, this paper constructs an image-to-image translation dataset and directly train an end-to-end translation model. They demonstrate that this can alleviate the error propagation issue and improve the overall translation performance.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eOZQEH74lU", "reviewer": "Reviewer_9Cpu", "summary": "", "questions": "- It is not entirely clear how an image with a size of HxW is converted into a sequence of length T in the proposed method. However, it appears that the image is organized as a sequence from top to bottom and then from left to right, as shown in Figure 5. More explanation is needed in the paper to clarify this process.\n- The BLEU score of the Gloden NMT model on the newstest-2014 seems slightly higher than results reported in previous NMT paper. What about the details of your NMT model?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments, and here are our responses.\n\n**Reason To Reject 1:**\n\n> The created dataset primarily contains images with black text and white backgrounds, which may be simplified compared to real-world scenarios. It is unclear if the proposed method will outperform the cascaded method in more complex scenarios.\n\nOur dataset is close to real-world scenarios that contain a lot of text, such as webpage screenshots and scanned documents. We have constructed datasets with different fonts and font sizes, and our end-to-end models reach similar results. We will add the results in our new version.\n\nTo the best of our knowledge, there is no public available dataset for in-image machine translation, and our dataset serves as a good test-bed for initial attempts. Our work is a preliminary study on in-image machine translation, and we achieve better translation quality than cascade method in this scenario. We hope our method and dataset can facilitate the study of in-image machine translation, and there will be more further research.\n\n**Question 1:**\n\n> It is not entirely clear how an image with a size of HxW is converted into a sequence of length T in the proposed method. However, it appears that the image is organized as a sequence from top to bottom and then from left to right, as shown in Figure 5. More explanation is needed in the paper to clarify this process.\n\nWe describe the conversion of an image into a sequence in sub-section 4.2, which may not be very detailed. The actual process is what you said. The grayscale map is firstly converted into the character map (Line 191-Line 200). Then the character map is organized as a sequence from top to bottom and then from left to right, and the character map with a size of HxW is converted into a sequence of length T (T=HxW) (Line 202-Line 203.) Thanks for your suggestion, we have described it more clearly in the new version to eliminate misunderstandings.\n\n**Question 2:**\n\n> The BLEU score of the Gloden NMT model on the newstest-2014 seems slightly higher than results reported in previous NMT paper. What about the details of your NMT model?\n\nWe conduct experiments on newstest-2014 De-En (Line 262-Line 269), and the settings of Golden NMT is same as the NMT in the cascade method (Transformer Big, 32K BPE and beam 5, described at Line 318-Line 324). \n\nThere are two translation directions of newstest-2014, including De-En and En-De. For De-En, the BLEU score with similar settings is 31.4 [1]. For En-De, the BLEU score is 28.1 [1]. \n\nThe BLEU score of De-En is higher than our Golden NMT (29.7). The reason is to make our systems comparable, we render the translated text obtained by Golden NMT into the image, and use the same OCR model to recognize the text (Line 442 - Line 446). Because of errors in OCR results, the BLEU score is lower than other work on newstest-2014 De-En.\n\n**Missing References:**\n\n> Robust Open­Vocabulary Translation from Visual Text Representations\n\nThanks for your reminder, we have added it in our new version.\n\nThanks for your comments to help us improve our work, and we will continue to polish the paper to make it better.\n\n[1] Rothe S, Narayan S, Severyn A. Leveraging pre-trained checkpoints for sequence generation tasks[J]. Transactions of the Association for Computational Linguistics, 2020, 8: 264-280.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed framework is surprisingly effective in translating the source image to the target image without any textual supervision during training, outperforming the cascaded OCR+MT baseline. It demonstrates that the segmented pixel sequence is a simple yet effective intermediate representation for this task.\n- The paper is well written and easy to follow. The authors did detailed analysis of the proposed method.", "reasons_to_reject": "The created dataset primarily contains images with black text and white backgrounds, which may be simplified compared to real-world scenarios. It is unclear if the proposed method will outperform the cascaded method in more complex scenarios.", "questions_to_authors": "- It is not entirely clear how an image with a size of HxW is converted into a sequence of length T in the proposed method. However, it appears that the image is organized as a sequence from top to bottom and then from left to right, as shown in Figure 5. More explanation is needed in the paper to clarify this process.\n- The BLEU score of the Gloden NMT model on the newstest-2014 seems slightly higher than results reported in previous NMT paper. What about the details of your NMT model?", "paper_topic_and_main_contributions": "This paper focuses on a new task, In-Image Machine Translation (IIMT), and proposes a novel and effective solution. The input and output images are converted into pixel sequences, and the grayscale values are mapped to discrete symbols (1-10). A BPE vocabulary is learned on the symbol sequence, and a sequence-to-sequence Transformer model is used to learn the mapping between the source and target sequences. Experimental results on a created benchmark demonstrate the effectiveness of the proposed method.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Robust Open­Vocabulary Translation from Visual Text Representations"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a pixel sequence-to-sequence model that allows them to translate text in images with an end-to-end model, instead of a cascade between OCR and translation. Experiments show improvements over the cascade on both in- and out-of-domain test sets. The reviewers initially had many concerns with the presentation and experiments, but it seems like most of these have been resolved during the response. The most pressing remaining concern is the realism of the training and test sets, which are formed from black text on a white background, calling into question the generality of the in-image translations that are tested in this paper. The authors correctly point out that, as this is a new task, there are no better datasets, and that this does address a common scenario of images consisting primarily of text. But all three reviewers still seem somewhat disappointed. Given the framing of the paper, it would be much stronger with more realistic data.
Decision: Accept-Findings