English-to-Japanese Multimodal Machine Translation Based on Image-Text Matching of Lecture Videos (2024)

Abstract: We work on a multimodal machine translation of the audio contained in English lecture videos to generate Japanese subtitles. Image-guided multimodal machine translation is promising for error correction in speech recognition and for text disambiguation. In our situation, lecture videos provide a variety of images. Images of presentation materials can complement information not available from audio and may help improve translation quality. However, images of speakers or audiences would not directly affect the translation quality. We construct a multimodal parallel corpus with automatic speech recognition text and multiple images for a transcribed parallel corpus of lecture videos, and propose a method to select the most relevant ones from the multiple images with the speech text for improving the performance of image-guided multimodal machine translation. Experimental results on translating automatic speech recognition or transcribed English text into Japanese show the effectiveness of our method to select a relevant image.

heading: 1 Introduction
text: Multimodal machine translation (Sulubacak et al., 2020) is a machine translation (MT) approach that combines information from modalities other than text, such as audio and images. Since images provide visual information that is not included in audio or text, it is expected to improve translation quality by correcting errors in automatic speech recognition (ASR) or by complementing information in ambiguous text.
This study tackles the task of translating English audio or subtitles from lecture videos into Japanese. In such situations, since useful information can be obtained from the images in the presentation materials, image-guided MT can improve translation quality over text-only MT. However, some of the images derived from lecture videos are
when you do that here's what I can promise you're going to be the 800- pound gorilla in the forest ASR text (English)
So when you do that, here's what I can promise: You're going to be the 800 pound gorilla in the forest. Transcribed text (English)
not directly related to the subtitle text, such as the image shown on the left in Figure 1, which shows only the speaker. No improvement in translation quality can be expected from such images.
To improve the performance of image-guided MT, we propose a method to select the image most relevant to the text among multiple images that correspond in time to the subtitle text. Additionally, to evaluate our method, we construct a multimodal parallel corpus, TAIL1 (Englishto-Japanese Translation Corpus with Audio and Images from Lecture Videos), consisting of English subtitles of lecture videos and their Japanese translations. Experimental results on translating ASR or transcribed English text into Japanese subtitles show the effectiveness of our method to select a relevant image.
1https://github.com/EhimeNLP/TAIL
86

heading: 2 Related Work

heading: 2.1 Multimodal Parallel Corpus
text: Previous studies of multimodal MT have often involved adding some one modality to the text, such as MT from speech (Di Gangi et al., 2019; Wang et al., 2020; Salesky et al., 2021) or imageguided MT (Elliott et al., 2016; Parida et al., 2019; Thapliyal et al., 2022). Furthermore, we can expect further improvement in translation quality by combining three modalities of text, audio, and images. Previous studies combining the three modalities include video-guided MT, such as How-2 (Sanabria et al., 2018) and QED (Abdelali et al., 2014). However, these are limited in scope because How-2 only covers English-Portuguese language pairs and QED only covers education domain. To cover English-Japanese lecture subtitles, we need to expand the multimodal parallel corpus.

heading: 2.2 Image-guided Machine Translation
text: Image-guided MT (Specia et al., 2016) improves translation quality by complementing textual ambiguity with visual information derived from images. Early studies (Caglayan et al., 2016; Libovický and Helcl, 2017; Calixto and Liu, 2017) combined CNN-based visual representations with textual representations in RNN-based encoderdecoder models. In the modern approach (Li et al., 2022a), both vision and language inputs are encoded by the Transformer (Vaswani et al., 2017; Dosovitskiy et al., 2021), integrated by selective attention, and fed to the Transformer decoder. The image-guided machine translation model, based on the powerful Vision Transformer (Dosovitskiy et al., 2021), achieves higher translation quality with images that are more relevant to the text (Yuasa et al., 2023). Therefore, in situations where multiple images are available, translation quality can be improved by selecting images that are more relevant to the text.

heading: 2.3 Vision and Language Pre-training
text: In image-text matching, CLIP (Radford et al., 2021) and BLIP (Li et al., 2022b), trained by multimodal contrastive learning, have achieved state-of-the-art performance. Especially, BLIP is trained in a multi-task learning manner of imagetext matching and image caption generation as well as contrastive learning, which allows a single model to perform both understanding and generating on vision and language tasks.

heading: 3 TAIL Corpus
text: For English-to-Japanese multimodal MT of lecture subtitles, we construct a corpus consisting of five sets of images, audio in English, ASR sentences in English, transcribed sentences in English, and reference translations in Japanese for lecture videos from TED.2 Since an EnglishJapanese parallel corpus consisting of transcribed sentences for TED lecture videos has been released in the IWSLT2017 competition (Cettolo et al., 2017), we annotate it with images, audio, and ASR sentences, as shown in Figure 2.

heading: 3.1 Audio Annotation
text: First, we annotate both audio and ASR sentences in English on top of the IWSLT2017 En-Ja corpus.
Audio Acquisition We downloaded the lecture videos in MP4 format from the URLs provided in the metadata of the IWSLT2017 En-Ja corpus. These videos are converted to audio in FLAC format with ffmpeg converter.3 (Step 1 in Figure 2)
Forced Alignment For each lecture video, the transcribed English sentences from the IWSLT2017 En-Ja corpus and the audio from Step 1 are aligned by aeneas toolkit.4 Here, both
2https://www.ted.com 3https://ffmpeg.org 4https://github.com/readbeyond/aeneas
start and end timestamps are recorded for each sentence (Step 2 in Figure 2), and the audio is segmented by ffmpeg (Step 3 in Figure 2).
Automatic Speech Recognition The audio, which was segmented into sentences in Step 3, is converted into text using Google Speech Recognition.5 (Step 4 in Figure 2)

heading: 3.2 Image Annotation
text: In this section, we further annotate images on top of our corpus to construct five sets. Since some scenes in TED videos do not represent the content of the lecture, such as scenes showing only the speaker, we collect multiple images for each sentence. Specifically, we use three images corresponding to the beginning, middle, and end of the timestamp of each sentence. Three images per sentence were extracted using OpenCV library6 with video and timestamps for each lecture video.

heading: 3.3 Parallel Corpus Filtering
text: The IWSLT2017 En-Ja corpus originally released 223k sentence pairs, but only 212k sentence pairs allowed us to access the videos from the URLs. To reduce noise in the corpus due to errors in timestamping and alignment, we automatically filter our parallel corpus. We filter out noisy sentence pairs by both sentence length difference and word error rate (WER) between automatically recognized (ASR) and manually transcribed (REF) English sentences. We keep 0.8 ≤ len(ASR)/len(REF) ≤ 1.2 cases with small sentence length differences. Where len(·) is the number of words in the sentence. It also keeps WER(ASR, REF) ≤ 0.5 cases with small WER. In the WER calculation, text was lowercased and symbols were removed as a preprocessing step. This left 102k sentence pairs.
We further exclude pairs where all images are unrelated to the text, which is not beneficial to the
5https://github.com/Uberi/speech_recognition 6https://opencv.org
image-guided MT. We compute the cosine similarity between the text and each of the three images assigned to it, and employ the 70,000 sentence pairs in descending order of their maximum value for our experiment. Here, BLIP-based multimodal embeddings (Li et al., 2022b) are used for similarity calculations, as in the next section.

heading: 4 Image-guided Machine Translation
text: In this study, as shown in Figure 1, we are given an English sentence that has been automatically recognized or manually transcribed from a lecture subtitle as well as three images that correspond in time to the text. The image-guided machine translation that we are working on is the task of inputting one image selected from among three images along with its English text and translating it into Japanese subtitles.
To select the image related to a given English sentence, we estimate the semantic similarity between vision and language. Both of the following two proposed methods are based on BLIP (Li et al., 2022b), a pre-trained multimodal model.
• Embedding-based method: Encode each given text and image with BLIP and then rank multiple images by the cosine similarity between their embeddings.
• Captioning-based method: Generate an English caption with BLIP from a given image and rank multiple images by the BLEU (Papineni et al., 2002) between the input text and the caption. (Right side of Figure 3)

heading: 5 Evaluation

heading: 5.1 Setting
text: Model Our multimodal MT model employed the Selective Attention model7 (Li et al., 2022a).
7https://github.com/libeineu/fairseq_mmt
This model is a 4-layer, 128-dimensional Transformer (Vaswani et al., 2017) combined with image features from the Vision Transformer (vit_tiny_patch16_384) (Dosovitskiy et al., 2021). RAdam (Liu et al., 2020) was used for optimization and trained with a batch size of 4, 096 tokens and a learning rate of 1e− 4. Training was terminated when the cross-entropy loss in the validation dataset was not updated 10 times.
Data The TAIL corpus described in Section 3 was used for our experiments. We used 70,000 sentence pairs for training, 2,669 for validation, and 2,371 for evaluation. As a preprocessing, MosesTokenizer8 (Koehn et al., 2007) and MeCab9 (IPADIC) (Kudo et al., 2004) were used for word segmentation for English and Japanese, respectively. Subsequently, a subword segmentation with a vocabulary size of 16, 000 was performed by fastBPE10 (Sennrich et al., 2016).
Comparison We evaluate the effectiveness of our image-text matching for image-guided MT by comparing it to the following three baseline models. Each model is trained three times with changing random seed, and the averaged BLEU (Papineni et al., 2002) is reported.
• w/o Image baseline: Text-only MT model. We discuss the effectiveness of the imageguided MT in comparison to this baseline.
• w/ Random Image baseline: An imageguided MT model that uses a randomly selected image from the entire dataset. We discuss the effectiveness of the use of related images in comparison to this baseline.
• w/ Related Image baseline: An imageguided MT model that uses a randomly selected image from a set of three images that correspond in time to given sentence. We discuss the effectiveness of the use of the most related images in comparison to this baseline.

heading: 5.2 Results
text: Automatic Evaluation Experimental results are shown in the BLEU columns of Table 1. Note that the ASR column is the translation quality for automatically recognized English sentences, while
8https://github.com/moses-smt/mosesdecoder 9https://taku910.github.io/mecab/
10https://github.com/glample/fastBPE
the IWSLT column is for manually transcribed English sentences. Compared to the baseline model without images, the other image-guided MT models achieved significantly higher translation quality. This suggests the effectiveness of complementing MT of lecture subtitles with images.
Two baselines of image-guided MT (w/ Random Image and w/ Related Image) achieved comparable translation quality. This suggests that simply using images that correspond in time to the input text does not necessarily result in high performance. In contrast, our embedding-based method of selecting images to match text achieved the best performance for both ASR and IWSLT text.
Human Evaluation The Accuracy column in Table 1 shows the human evaluation of the accuracy of image selection for randomly sampled 200 texts. Note that these samples do not include cases where all images are related to or unrelated to the text. As with translation quality, our embedding-based method achieved the best performance. These results reveal a strong correlation between the performance of image-text matching and translation quality. It is suggested that multimodal MT performance can be improved by selecting images that are well related to the text.

heading: 6 Conclusion
text: In this study, we constructed a multimodal parallel corpus of images, audio in English, ASR sentences in English, transcribed sentences in English, and reference translations in Japanese of approximately 75k sentence pairs to generate crosslingual subtitles from lecture videos. Experimental results reveal that our embedding-based imagetext matching method contributes to improved performance of image-guided machine translation. Our future work includes further improvement of translation quality by combining multiple images.

heading: Acknowledgments
text: These research results were obtained from the commissioned research (No.22501) by National Institute of Information and Communications Technology (NICT), Japan.

heading: Future Work
text: Our future work includes further improvement of translation quality by combining multiple images