Exploring Better Text Image Translation with Multimodal Codebook
Year: 2023

Abstract:
Text image translation (TIT) aims to translate the source texts embedded in the image to target translations, which has a wide range of applications and thus has important research value. However, current studies on TIT are confronted with two main bottlenecks: 1) this task lacks a publicly available TIT dataset, 2) dominant models are constructed in a cascaded manner, which tends to suffer from the error propagation of optical character recognition (OCR). In this work, we first annotate a Chinese-English TIT dataset named OCRMT30K, providing convenience for subsequent studies. Then, we propose a TIT model with a multimodal codebook, which is able to associate the image with relevant texts, providing useful supplementary information for translation. Moreover, we present a multi-stage training framework involving text machine translation, image-text alignment, and TIT tasks, which fully exploits additional bilingual texts, OCR dataset and our OCRMT30K dataset to train our model. Extensive experiments and in-depth analyses strongly demonstrate the effectiveness of our proposed model and training framework.1

heading: 1 Introduction
text: In recent years, multimodal machine translation (MMT) has achieved great progress and thus received increasing attention. Current studies on MMT mainly focus on the text machine translation with scene images (Elliott et al., 2016; Calixto et al., 2017a; Elliott and Kádár, 2017; Libovický et al., 2018; Ive et al., 2019; Zhang et al., 2020; Sulubacak et al., 2020). However, a more common requirement for MMT in real-world applications is text image translation (TIT) (Ma et al., 2022), which aims to translate the source texts embedded in the image to target translations. Due to its wide
∗Equal contribution. †Corresponding author.
1Our code and dataset can be found at https://github. com/DeepLearnXMU/mc_tit
applications, the industry has developed multiple services to support this task, such as Google Camera Translation.
Current studies on TIT face two main bottlenecks. First, this task lacks a publicly available TIT dataset. Second, the common practice is to adopt a cascaded translation system, where the texts embedded in the input image are firstly recognized by an optical character recognition (OCR) model, and then the recognition results are fed into a textonly neural machine translation (NMT) model for translation. However, such a method tends to suffer from the problem of OCR error propagation, and thus often generates unsatisfactory translations. As shown in Figure 1, “富锦消防” ("fu jin xiao fang”) in the image is incorrectly recognized as “富锦消阳” (“fu jin xiao yang”). Consequently, the text-only NMT model incorrectly translates it into “Fujin Xiaoyang”. Furthermore, we use the commonly-used PaddleOCR2 to handle several OCR benchmark datasets. As reported in Table 1, we observe that the highest recognition accuracy at the image level is less than 67% and that at the sentence level is not higher than 81%. It can be said that OCR errors are very common, thus they have a serious negative impact on subsequent translation.
In this paper, we first manually annotate a Chinese-English TIT dataset named OCRMT30K,
2https://github.com/PaddlePaddle/PaddleOCR.
3479
providing convenience for subsequent studies. This dataset is developed based on five Chinese OCR datasets, including about 30,000 image-text pairs.
Besides, we propose a TIT model with a multimodal codebook to alleviate the OCR error propagation problem. The basic intuition behind our model is that when humans observe the incorrectly recognized text in an image, they can still associate the image with relevant or correct texts, which can provide useful supplementary information for translation. Figure 3 shows the basic architecture of our model, which mainly consists of four modules: 1) a text encoder that converts the input text into a hidden state sequence; 2) an image encoder encoding the input image as a visual vector sequence; 3) a multimodal codebook. This module can be described as a vocabulary comprising latent codes, each of which represents a cluster. It is trained to map the input images and ground-truth texts into the shared semantic space of latent codes. During inference, this module is fed with the input image and then outputs latent codes containing the text information related to ground-truth texts. 4) a text decoder that is fed with the combined representation of the recognized text and the outputted latent codes, and then generates the final translation.
Moreover, we propose a multi-stage training framework for our TIT model, which can fully exploit additional bilingual texts and OCR data for model training. Specifically, our framework consists of four stages. First, we use a large-scale bilingual corpus to pretrain the text encoder and text decoder. Second, we pretrain the newly added multimodal codebook on a large-scale monolingual corpus. Third, we further introduce an image encoder that includes a pretrained vision Transformer with fixed parameters to extract visual features, and continue to train the multimodal codebook. Additionally, we introduce an image-text alignment task
to enhance the ability of the multimodal codebook in associating images with related texts. Finally, we finetune the entire model on the OCRMT30K dataset. Particularly, we maintain the image-text alignment task at this stage to reduce the gap between the third and fourth training stages.
Our main contributions are as follows:
• We release an OCRMT30K dataset, which is the first Chinese-English TIT dataset, prompting the subsequent studies.
• We present a TIT model with a multimodal codebook, which can leverage the input image to generate the information of relevant or correct texts, providing useful information for the subsequent translation.
• We propose a multi-stage training framework for our model, which effectively leverages additional bilingual texts and OCR data to enhance the model training.
• Extensive experiments and analyses demonstrate the effectiveness of our model and training framework.

heading: 2 Related Work
text: In MMT, most early attempts exploit visual context via attention mechanisms (Caglayan et al., 2016; Huang et al., 2016; Calixto et al., 2017a; Libovický and Helcl, 2017; Calixto and Liu, 2017; Su et al., 2021). Afterwards, Ive et al. (2019) employ a translate-and-refine approach to improve translation drafts with visual context. Meanwhile, Calixto et al. (2019) incorporate visual context into MMT model through latent variables. Different from these studies focusing on coarse-grained visual-text alignment information, Yin et al. (2020) propose a unified multimodal graph based encoder to capture various semantic relationships between tokens and visual objects. Lin et al. (2020) present a dynamic context-guided capsule network to effectively capture visual features at different granularities for MMT.
Obviously, the effectiveness of conventional MMT heavily relies on the availability of bilingual texts with images, which restricts its wide applicability. To address this issue, Zhang et al. (2020) first build a token-image lookup table from an image-text dataset, and then retrieve images matching the source keywords to benefit the predictions of target translation. Recently, Fang and
Feng (2022) present a phrase-level retrieval-based method that learns visual information from the pairs of source phrases and grounded regions.
Besides, researchers investigate whether visual information is really useful for machine translation. Elliott (2018) finds that irrelevant images have little impact on translation quality. Wu et al. (2021) attribute the gain of MMT to the regularization effect. Unlike these conclusions, Caglayan et al. (2019) and Li et al. (2021) observe that MMT models rely more on images when textual ambiguity is high or textual information is insufficient.
To break the limitation that MMT requires sentence-image pairs during inference, researchers introduce different modules, such as image prediction decoder (Elliott and Kádár, 2017), generative imagination network (Long et al., 2021), autoregressive hallucination Transformer (Li et al., 2022b), to produce a visual vector sequence that is associated with the input sentence.
Significantly different from the above studies on MMT with scene images, several works also explore different directions in MMT. For instance, Calixto et al. (2017b) and Song et al. (2021) investigate product-oriented machine translation, and other researchers focus on multimodal simultaneous machine translation (Caglayan et al., 2020; Ive et al., 2021). Moreover, there is a growing body of studies on video-guided machine translation (Wang et al., 2019; Gu et al., 2021; Kang et al., 2023). These studies demonstrate the diverse applications and potential of MMT beyond scene images.
In this work, we mainly focus on TIT, which suffers from incorrectly recognized text information and is more practicable in real scenarios. The most related work to ours mainly includes (Mansimov et al., 2020; Jain et al., 2021; Ma et al., 2022). Mansimov et al. (2020) first explore in-image translation task, which transforms an image containing the source text into an image with the target translation. They not only build a synthetic in-image translation dataset but also put forward an end-toend model combining a self-attention encoder with two convolutional encoders and a convolutional decoder. Jain et al. (2021) focus on the TIT task, and propose to combine OCR and NMT into an endto-end model with a convolutional encoder and an autoregressive Transformer decoder. Along this line, Ma et al. (2022) apply multi-task learning to this task, where MT, TIT, and OCR are jointly trained. However, these studies only center around
synthetic TIT datasets, which are far from the real scenario.

heading: 3 Dataset and Annotation
text: To the best of our knowledge, there is no publicly available dataset for the task of TIT. Thus we first manually annotate a Chinese-English TIT dataset named OCRMT30K, which is based on five commonly-used Chinese OCR datasets: RCTW-17 (Shi et al., 2017), CASIA-10K (He et al., 2018), ICDAR19-MLT (Nayef et al., 2019), ICDAR19LSVT (Sun et al., 2019) and ICDAR19-ArT (Chng et al., 2019). We hire eight professional translators for annotation over five months and each translator is responsible for annotating 25 images per day to prevent fatigue. Translators are shown an image with several Chinese texts and are required to produce correct and fluent translations for them in English. In addition, we hire a professional translator to sample and check the annotated instances for quality control. We totally annotate 30,186 instances and the number of parallel sentence pairs is 164,674. Figure 2 presents an example of our dataset.

heading: 4 Our Model

heading: 4.1 Task Formulation
text: In this work, following common practices (Afli and Way, 2016; Ma et al., 2022), we first use an OCR model to recognize texts from the input image v. Then, we fed both v and each recognized text x̂ into our TIT model, producing the target translation y. In addition, x is used to denote the ground-truth text of x̂ recognized from v.
To train our TIT model, we will focus on establishing the following conditional predictive proba-
bility distribution:
P (y|v, x̂;θ) = |y|∏
t=1
P (yt|v, x̂,y<t;θ), (1)
where θ denotes the model parameters.

heading: 4.2 Model Architecture
text: As shown in Figure 3, our model includes four modules: 1) a text encoder converting the input text into a hidden state sequence; 2) an image encoder encoding the input image as a visual vector sequence; 3) a multimodal codebook that is fed with the image representation and then outputs latent codes containing the text information related to the ground-truth text; and 4) a text decoder that generates the final translation under the semantic guides of text encoder hidden states and outputted latent codes. All these modules will be elaborated in the following.
Text Encoder. Similar to dominant NMT models, our text encoder is based on the Transformer (Vaswani et al., 2017) encoder. It stacks Le identical layers, each of which contains a self-attention sub-layer and a feed-forward network (FFN) sublayer.
Let H(l)e = h (l) e,1, h (l) e,2, ..., h (l) e,Ne
denotes the hidden states of the l-th encoder layer, where Ne is the length of the hidden states H(l)e . Formally, H (l) e is calculated in the following way:
H(l)e = FFN(MHA(H (l−1) e ,H (l−1) e ,H (l−1) e )),
(2) where MHA(·, ·, ·) denotes a multi-head attention function (Vaswani et al., 2017). Particularly, H(0)e
is the sum of word embeddings and position embeddings. Note that we follow Vaswani et al. (2017) to use residual connection and layer normalization (LN) in each sub-layer, of which descriptions are omitted for simplicity. During training, the text encoder is utilized to encode both the ground-truth text x and the recognized text x̂, so we use Ĥ(l)e to denote the hidden state of recognized text for clarity. In contrast, during inference, the text encoder only encodes the recognized text x̂, refer to Section 4.3 for more details.
Image Encoder. As a common practice, we use ViT (Dosovitskiy et al., 2021) to construct our image encoder. Similar to the Transformer encoder, ViT also consists of Lv stacked layers, each of which includes a self-attention sub-layer and an FFN sub-layer. One key difference between the Transformer encoder and ViT is the placement of LN, where pre-norm is applied in ViT.
Given the image input v, the visual vector sequence H(Lv)v = h (Lv) v,1 , h (Lv) v,2 , ..., h (Lv) v,Nv
output by the image encoder can be formulated as
H(Lv)v = MHA(H (Le) e ,WvViT(v),WvViT(v)), (3) where Nv is the length of the hidden states H (Lv) v and Wv is a projection matrix to convert the dimension of ViT(v) into that of H(Le)e .
Multimodal Codebook. It is the core module of our model. The multimodal codebook is essentially a vocabulary with K latent codes, each of which is represented by a d-dimensional vector ek like word embeddings. Note that we always set the dimension of the latent code equal to that of the text encoder, so as to facilitate the subsequent calculation in Equation 11.
With the multimodal codebook, we can quantize the hidden state sequence H(Le)e = h (Le) e,1 , h (Le) e,2 , ..., h (Le) e,Ne or the visual vector sequence H (Lv) v = h (Lv) v,1 , h (Lv) v,2 , ..., h (Lv) v,Nv
to latent codes via a quantizer zq(·). Formally, the quantizer looks up the nearest latent code for each input, as shown in the following:
zq(h (Le) e,i ) = argmin e k ′ ||h(Le)e,i − ek′ ||2, (4)
zq(h (Lv) v,j ) = argmin e k ′′ ||h(Lv)v,j − ek′′ ||2. (5)
By doing so, both text and image representations are mapped into the shared semantic space of latent codes.
Text Decoder. This decoder is also based on the Transformer decoder, with Ld identical layers. In addition to self-attention and FFN sub-layers, each decoder layer is equipped with a cross-attention sub-layer to exploit recognized text hidden states Ĥ (Le) e and latent codes zq(H (Lv) v ).
The hidden states of the l-th decoder layer are denoted by H(l)d = h (l) d,1, h (l) d,2, ..., h (l) d,Nd
, where Nd represents the total number of hidden states. These hidden states are calculated using the following equations:
C (l) d = MHA(H (l−1) d ,H (l−1) d ,H (l−1) d ), (6)
T (l) d = [Ĥ (Le) e ; zq(H (Lv) v )], (7)
H (l) d = FFN(MHA(C (l) d ,T (l) d ,T (l) d )). (8)
Finally, at each decoding timestep t, the probability distribution of generating the next target token yt is defined as follows:
P (yt|v, x̂,y<t;θ) = softmax(Woh(Ld)d,t + bo), (9) where Wo and bo are trainable model parameters.

heading: 4.3 Multi-stage Training Framework
text: In this section, we present in detail the procedures of our proposed multi-stage training framework. As shown in Figure 4, it totally consists of four stages: 1) pretraining the text encoder and text decoder on a large-scale bilingual corpus; 2) pretraining the multimodal codebook on a large-scale monolingual corpus; 3) using additional OCR data to train the image encoder and multimodal codebook via an image-text alignment task; 4) finetuning the whole model on our released TIT dataset.
Stage 1. We first pretrain the text encoder and text decoder on a large-scale bilingual corpus Dbc in the way of a vanilla machine translation. Formally, for each parallel sentence (x,y)∈Dbc, we define the following training objective for this stage:
L1(θte,θtd) = − |y|∑
t=1
log(p(yt|x,y<t)), (10)
where θte and θtd denote the trainable parameters of the text encoder and text decoder, respectively.
Stage 2. This stage serves as an intermediate phase, where we exploit monolingual data to pretrain the multimodal codebook. Through this stage of training, we will learn a clustering representation for each latent code of the multimodal codebook.
Concretely, we utilize the same dataset as the first stage but only use its source texts. Following van den Oord et al. (2017), we update the multimodal codebook with an exponential moving average (EMA), where a decay factor determines the degree to which past values affect the current average. Formally, the latent code embedding ek is updated as follows:
ck =
Ne∑
i=1
I(zq(h (Le) e,i ) = ek),
hk =
Ne∑
i=1
I(zq(h (Le) e,i ) = ek)h (Le) e,i ,
nk ← γnk + (1− γ)ck,
ek ← 1
nk (γek + (1− γ)hk),
(11)
where I(·) is the indicator function and γ is a decay factor we set to 0.99, as implemented in (van den
Oord et al., 2017). ck counts the number of text encoder hidden states that are clustered into the kth latent code, hk denotes the sum of these hidden states, and nk represents the sum of the past exponentially weighted average and the current value ck. Particularly, nk is set to 0 at the beginning.
Stage 3. During this stage, we introduce an image-text alignment task involving an additional OCR dataset Docr to further train the image encoder and multimodal codebook. Through this stage of training, we expect to endow the multimodal codebook with the preliminary capability of associating images with related texts.
Given an image-text training instance (v,x) ∈ Docr, we define the training objective at this stage as
L3 = Lita + αLic, (12) Lita(θie) = ||zq(H(Lv)v )− sg(zq(H(Le)e ))||22,
(13)
Lic(θie) = ||H(Lv)v − sg(zq(H(Lv)v ))||22, (14) where sg(·) refers to a stop-gradient operation and θie is the parameters of the image encoder except the ViT module. Specifically, zq(H (Lv) v ) is calculated as 1Nv ∑Nv j=1 zq(h (Lv) v,j ) and zq(H (Le) e ) is calculated as 1Ne ∑Ne i=1 zq(h (Le) e,i ), which represent the semantic information of image and text respectively. Via Lita, we expect to enable both image and text representations to be quantized into the same latent codes. Meanwhile, following van den Oord et al. (2017), we use the commitment loss Lic to ensure that the output hidden states of image encoder stay close to the chosen latent code embedding, preventing it fluctuating frequently from one latent code to another, and α is a hyperparameter to control the effect of Lic. Note that at this stage, we continue to update the parameters of the multimodal codebook using Equation 11.
Stage 4. Finally, we use the TIT dataset Dtit to finetune the whole model. Notably, L3 is still involved, which maintains the training consistency and makes finetuning smoothing.
Given a TIT training instance (v, x̂,x,y)∈Dtit, we optimize the whole model through the following objective:
L4 = L3 + Ltit + βLtc, (15)
Ltit(θte,θie,θtd) = − |y|∑
t=1
log(p(yt|v, x̂,y<t)),
(16)
Ltc(θte) = ||H(Le)e − sg(zq(H(Le)e ))||22, (17)
where Ltc is also a commitment loss proposed for the text encoder, and β is a hyperparameter quantifying its effect. Note that x̂ is only used as an input for Ltit to ensure the consistency between the model training and inference, and x is used as an input for image-text alignment task to train the ability of the multimodal codebook in associating the input image with the ground-truth text. Besides, we still update the multimodal codebook with EMA.

heading: 5 Experiments

heading: 5.1 Datasets
text: Our proposed training framework consists of four stages, involving the following three datasets:
WMT22 ZH-EN3. This large-scale parallel corpus contains about 28M parallel sentence pairs and we sample 2M parallel sentence pairs from the original whole corpus. During the first and second training stages, we use the sampled dataset to pretrain our text encoder and text decoder.
ICDAR19-LSVT. It is an OCR dataset including 450, 000 images with texts that are freely captured in the streets, e.g., storefronts and landmarks. In this dataset, 50,000 fully-annotated images are partially selected to construct the OCRMT30K dataset, and the remaining 400,000 images are weakly annotated, where only the text-of-interest in these images are provided as ground truths without location annotations. In the third training stage, we use these weakly annotated data to train the image encoder and multimodal codebook via the image-text alignment task.
OCRMT30K. As mentioned previously, our OCRMT30K dataset involves five Chinese OCR datasets: RCTW-17, CASIA-10K, ICDAR19-MLT, ICDAR19-LSVT, and ICDAR19-ArT. It totally contains about 30,000 instances, where each instance involves an image paired with several Chinese texts and their corresponding English translations. In the experiments, we choose 1,000 instances for development, 1,000 for evaluation, and the remaining instances for training. Besides, We use the commonly-used PaddleOCR to handle our dataset and obtain the recognized texts. In the final training stage, we use the training set of OCRMT30K to finetune our whole model.
3https://www.statmt.org/wmt22/translation-task.html

heading: 5.2 Settings
text: We use the standard ViT-B/16 (Dosovitskiy et al., 2021) to model our image encoder. Both our text encoder and text decoder consist of 6 layers, each of which has 512-dimensional hidden sizes, 8 attention heads, and 2,048 feed-forward hidden units. Particularly, a 512-dimensional word embedding layer is shared across the text encoder and the text decoder. We set the size of the multimodal codebook to 2,048.
During the third stage, following van den Oord et al. (2017), we set α in Equation 12 to 0.25. During the final training stage, we set α to 0.75 and β in Equation 15 to 0.25 determined by a grid search on the validation set, both of which are varied from 0.25 to 1 with an interval of 0.25. We use the batch size of 32,768 tokens in the first and second training stages and 4,096 tokens in the third and final training stages. In all stages, we apply the Adam optimizer (Kingma and Ba, 2015) with β1 = 0.9, β2 = 0.98 to train the model, where the inverse square root schedule algorithm and warmup strategy are adopted for the learning rate. Besides, we set the dropout to 0.1 in the first three training stages and 0.3 in the final training stage, and the value of label smoothing to 0.1 in all stages.
During inference, we use beam search with a beam size of 5. Finally, we employ BLEU (Papineni et al., 2002) calculated by SacreBLEU4 (Post, 2018) and COMET5 (Rei et al., 2020) to evaluate the model performance.

heading: 5.3 Baselines
text: In addition to the text-only Transformer (Vaswani et al., 2017), our baselines include:
• Doubly-ATT (Calixto et al., 2017a). This model uses two attention mechanisms to exploit the image and text representations for translation, respectively.
• Imagination (Elliott and Kádár, 2017). It trains an image prediction decoder to predict a global visual feature vector that is associated with the input sentence.
• Gated Fusion (Wu et al., 2021). This model uses a gated vector to fuse image and text representations, and then feeds them to a decoder for translation.
4https://github.com/mjpost/sacrebleu 5https://github.com/Unbabel/COMET
• Selective Attn (Li et al., 2022a). It is similar to Gated Fusion, but uses a selective attention mechanism to make better use of the patchlevel image representation.
• VALHALLA (Li et al., 2022b). This model uses an autoregressive hallucination Transformer to predict discrete visual representations from the input text, which are then combined with text representations to obtain the target translation.
• E2E-TIT (Ma et al., 2022). It applies a multitask learning framework to train an end-toend TIT model, where MT and OCR serve as auxiliary tasks. Note that except for E2E-TIT, all other models are cascaded ones. Unlike other cascaded models that take recognized text and the entire image as input, the input to this end-to-end model is an image cropped from the text bounding box.
To ensure fair comparisons, we pretrain all these baselines on the same large-scale bilingual corpus.

heading: 5.4 Results
text: Table 2 reports the performance of all models. We can observe that our model outperforms all baselines, achieving state-of-the-art results. Moreover, we draw the following interesting conclusions:
First, all cascaded models exhibit better performance than E2E-TIT. For this result, we speculate that as an end-to-end model, E2E-TIT may struggle to distinguish text from the surrounding background in the image when the background exhibits visual characteristics similar to the text.
Second, our model outperforms Doubly-ATT, Gated Fusion, and Selective Attn, all of which adopt attention mechanisms to exploit image information for translation. The underlying reason is that each input image and its texts are mapped into the shared semantic space of latent codes, reducing the modality gap and thus enabling the model to effectively utilize image information.
Third, our model also surpasses Imagination and VALHALLA, both of which use the input text to generate the representations of related images. We conjecture that in the TIT task, it may be challenging for the model to generate useful image representations from the incorrectly recognized text. In contrast, our model utilizes the input image to generate related text representations, which is more suitable for the TIT task.
Inspired by E2E-TIT, we also compare other baselines with the cropped image as input. Table 3 reports the results of our model compared with other baselines using the cropped image as input. We can observe that our model still achieves stateof-the-art results.

heading: 5.5 Ablation Study
text: To investigate the effectiveness of different stages and modules, we further compare our model with several variants in Table 4:
w/o Stage 2. We remove the second training stage in this variant. The result in line 2 shows that this change causes a significant performance decline. It suggests that pretraining the clustering representations of latent codes in the multimodal codebook is indeed helpful for the model training.
w/o Stage 3. In this variant, we remove the third
stage of training. The result in line 3 indicates that this removal leads to a performance drop. The result confirms our previous assumption that training the preliminary capability of associating images and related texts indeed enhances the TIT model.
w/o L3 in Stage 4. When constructing this variant, we remove the loss itemL3 from stage 4. From line 4, we can observe that preserving L3 in the fourth stage makes the transition from the third to the fourth stage smoother, which further alleviates the training discrepancy.
w/o multimodal codebook. We remove the multimodal codebook in this variant, and the visual features extracted through the image encoder are utilized in its place. Apparently, the performance drop drastically as reported in line 5, demonstrating the effectiveness of the multimodal codebook.
w/ randomly sampling latent codes. Instead of employing quantization, we randomly sample latent codes from the multimodal codebook in this variant. Line 6 shows that such sampling leads to a substantial performance decline. Thus, we confirm that latent codes generated from the input image indeed benefits the subquent translation.

heading: 5.6 Analysis
text: To further reveal the effect of the multimodal book, we provide a translation example in Figure 5(a), listing the OCR result and translations produced by ours and Gated Fusion, which is the most competitive baseline. It can be seen that “用品商店” (“supplies store”) is incorrectly recognized as “用 品高店” (“supplies high store”), resulting in the incorrect translation even for Gated Fusion. By contrast, our model can output the correct translation with the help of the multimodal codebook.
During decoding for “supplies store”, latent code 1368 demonstrated the highest cross-attention weight in comparison to other codes. Therefore, we only visualize the latent code 1368 for analysis. In Figure 5(b), since tokens may be duplicated and all images are different, we provide the five
most frequent tokens and five randomly-selected images from this latent code, and find that all these tokens and images are highly related to the topic of business. Thus, intuitively, the clustering vector of this latent code will fully encode the information related to the business, and thus can provide useful information to help the model conduct the correct translation.

heading: 6 Conclusion
text: In this paper, we release a Chinese-English TIT dataset named OCRMT30K, which is the first publicly available TIT dataset. Then, we propose a novel TIT model with a multimodal codebook. Typically, our model can leverage the input image to predict latent codes associated with the input sentence via the multimodal codebook, providing supplementary information for the subsequent translation. Moreover, we present a multi-stage training framework that effectively utilizes additional bilingual texts and OCR data to refine the training of our model.
In the future, we intend to construct a larger dataset and explore the potential applications of our method in other multimodal tasks, such as videoguided machine translation.
Limitations
Since our model involves an additional step of OCR, it is less efficient than the end-to-end TIT model, although it can achieve significantly better performance. Besides, with the incorporation of image information, our model is still unable to completely address the issue of error propagation caused by OCR.
Ethics Statement
This paper proposes a TIT model and a multi-stage training framework. We take ethical considerations seriously and ensure that the methods used in this study are conducted in a responsible and ethical manner. We also release a Chinese-English TIT dataset named OCRMT30K, which is annotated based on five publicly available Chinese OCR datasets, and are used to support scholars in doing research and not for commercial use, thus there exists not any ethical concern.

heading: Acknowledgments
text: The project was supported by National Key Research and Development Program of China (No. 2020AAA0108004), National Natural Science Foundation of China (No. 62276219), and Natural Science Foundation of Fujian Province of China (No. 2020J06001). We also thank the reviewers for their insightful comments.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: In the limitations section

heading: 3 A2. Did you discuss any potential risks of your work?
text: In the ethics statement section
3 A3. Do the abstract and introduction summarize the paper’s main claims? 6
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 1,5
3 B1. Did you cite the creators of artifacts you used? 5
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We follow license but do not discuss

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 5
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We use existing datasets
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? We use existing artifacts
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3,5
C 7 Did you run computational experiments? Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Not applicable. Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Not applicable. Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 3
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? We provide it, but do not describe in the paper
7 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? This is not the focus of our paper
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? We have obtained consent but not described it in the paper
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Due to personal privacy, we did not describe it in the paper

heading: Future Work
text: In the future, we intend to construct a larger dataset and explore the potential applications of our method in other multimodal tasks, such as videoguided machine translation. Limitations
Since our model involves an additional step of OCR, it is less efficient than the end-to-end TIT model, although it can achieve significantly better performance. Besides, with the incorporation of image information, our model is still unable to completely address the issue of error propagation caused by OCR