Video-Helpful Multimodal Machine Translation
Authors: Yihang Li
Shuichiro Shimizu
Chenhui Chu
Sadao Kurohashi
Wei Li
Venue: EMNLP 2023 Main (2023)
Keywords: [Multimodal machine translation
Video]

Abstract: Existing multimodal machine translation (MMT) datasets consist of images and video captions or instructional video subtitles, which rarely contain linguistic ambiguity, making visual information ineffective in generating appropriate translations. Recent work has constructed an ambiguous subtitles dataset to alleviate this problem but is still limited to the problem that videos do not necessarily contribute to disambiguation.

We introduce EVA (Extensive training set and Video-helpful evaluation set for Ambiguous subtitles translation), an MMT dataset containing 852k Japanese-English parallel subtitle pairs, 520k Chinese-English parallel subtitle pairs, and corresponding video clips collected from movies and TV episodes. In addition to the extensive training set, EVA contains a video-helpful evaluation set in which subtitles are ambiguous, and videos are guaranteed helpful for disambiguation.

Furthermore, we propose SAFA, an MMT model based on the Selective Attention model with two novel methods: Frame attention loss and Ambiguity augmentation, aiming to use videos in EVA for disambiguation fully. Experiments on EVA show that visual information and the proposed methods can boost translation performance, and our model performs significantly better than existing MMT models.

Video-Helpful Multimodal Machine Translation
Yihang Li Shuichiro Shimizu', Chenhui Chu', Sadao Kurohashi', Wei Li² 'yyoto University, *Google Research (liyh, sshimizu, chu, kuro}ēnlp. ist. i. kyoto-u...... ac. jc.jp, mwe mweeil@@ooogee.om
Abstract
Corresponding Video Clips
Translation set
Boutse
Tarpol
Existing multimodal machine translation (MMT) datasets consist of images and video captions or instructional video subtitles, which rarely contain linguistic ambiguity, making visual information ineffective in generating appropriate translations. Recent work has constructed ambiguous subtitles dataset to alleviate this problem but is still limited to the problem that videos do not necessar- ily contribute disambiguation. We intro- duce EVA (Extensive training set and Video- helpful evaluation set for Ambiguous subtitles translation), an MMT dataset containing 852k Jangnsse-English (Ja-En) parallel subtitle pairs, 520k Chinese-Enge English (Zh-En) parallel subti- tle pairs, and corresponding video clips col- lected from movies and TV episodes. dition the extensive training set, EVA con- tains a video-helpful evaluation set in which subtitles are ambiguous, and videos are guar- anteed helpful for disambiguation. Further- more, we propose SAFA, an MMT model based on the Selective Attention model with two novel methods: Frame attention loss and Am- biguity augmentation, aiming to use videos in EVA for disambiguation fully. Experi- ments on EVA show that visual information and the proposed methods can boost transla- tion performance, and model performs sig- nificantly better than existing MMT models. The EVA dataset and the SAFA model are available at: MM..ful-MMT.git.
Go
Hit it
(go)
Move
Forward
Figure 1: translation and the corresponding video clips. We combine parallel subtitles with source sub- titles Japanese and target subtitles in English into translation set.
factors such polysemy but also to different possi- ble translations caused by factors such emotion, politeness, and omission that needs multimodal in- formation for disambiguation. For a source text, if there are multiple possible translations and some them are more appropriate than others under certain visual scenes, we call it ambiguous. Multimodal machine translation (MMT) (Specia et al., 201 6; Sulubacak et al., 2020a) uses visual data auxil- iary information to tackle the ambiguity problem. The contextual information in the visual data helps to resolve the ambiguity in the source text data.
ad-
Previous MMT studies have mainly focused on the image-guided machine translation (IMT) task (Elliott et al., 2016; Zhao et al., 2020; Li et al., 2022a), where, given an image and source sen- tence, the goal is to enhance the quality of transla- tion by leveraging their semantic correspondence to the image. Resolving ambiguities through visual cues is one of the main motivations behind this task. Compared with images, videos contain ordered se- quences of frames and can provide richer visual fcatures such as motion features. Recently, some studies have started to focus on the video-guided machine translation (VMT) task (Wang et al., 2019; Gu al., 2021).
1 Introduction
Neural machine translation (NMT) (Bahdanau et al., 2015; Wu et al., 2016) models relying on text data have achieved nate-of-the-ar performance. However, in many cases, the text is insufficient provide the information needed for appropriate translation, especially when the source text is am- biguous. In this work, "Ambiguous" refers not only to ambiguity in a narrow sense caused by
VMT faces the problem of data scarcity. The How2 (Sanabria et al., 2018) and VATEX (Wang et al., 2019) datasets are recent efforts to allevi-

---

ale the problem. In addition, previous datasets are limited subtitles of instructional vidcos or video captions that describe the video clips. It has been shown that caption MMT essentially does not re- quire visual information due to the lack of language ambiguity in captions (Caglayan et al., 2019). At the same time, the subtitles of the instructional videos are similar to captions and also lack ambigu- ity. The VISA (Li et al., 2022b) dataset has made efforts to solve this problem. VISA contains paral- lel subtitles and corresponding video clips collected from movies or TV episodes in which the source subtitles are ambiguous. However, has a limita- tion that the videos do not necessarily contribute to disambiguation.
biguity and the need for the training contain- ing possible translation patterns, we collect video- helpful data using translation sets, which are sets of parallel subtitles that have the same source sub- titles but different target subtitles. An example of translation set is shown in Figure 1. Each pair of parallel subtitles belongs to video clip. As a translation task, the video clip can help us trans- late the source subtitle. The first video clip shows two women escaping, suggesting "go" translation, while the last video clip portrays a scene involving an army, suggesting a "forward" translation. We construct the dataset by collecting parallel subti- tles and corresponding video clips, constructing an evaluation set with translation sets and crowd- sourcing, and then using the remaining part as the training set.
To address the problems of previous VMT datasets, we construct new large-scale VMT dataset EVA (Extensive training set and Video- helpful evaluation set for Ambiguous subtitles translation) for VMT research. EVA has an exten- sive training set, and a video-helpful evaluation set in which source subtitles are ambiguous and videos are guaranteed to be helpful for disambiguation. In total, EVA contains 852k Ja-En parallel subti- tle pairs, 520k Zh-Lön parallel subtitle pairs, and corresponding video clips collected from movies and TV episodes, where each pair of parallel subti- tles has a corresponding video clip. Subtitles from movies and TV episodes are essentially dialogues and short (7.34 English words in our case), which makes subtitles have many possible interpretations and thus makes videos helpful.
Furthermore, we propose an MMT model SAFA (Selective Attention model with Frame attention loss and Ambiguity augmentation) for the VMT task. Based on the selective attention model pro- posed (Li et al., 2022a) for the IMT task, we 1) use CLIP4Clip (Luo et al., 2022) model to extract video features, 2) propose frame attention loss to make the model focus more the central frames where the subtitles occur, and 3) propose ambiguity augmentation make the model put more weights on the EVA show that SAFA achieves 15.41 BLEU score and 35.86 METEOR score for Ja-En translation, and 27.62 BLEU score and 48.74 METEOR score for Zh-En translation, which are significantly bet- ter than existing MMT models. Furthermore, our proposed methods significantly improve MT perfor- mance when incorporating videos, with relative im- provements of 9.99% and 4.94% in BLEU scores for Ja-En and Zh-En translations, respectively. In summary, our contributions are three-fold:
data. Experiments on
To train a VMT model that can disambiguate translations, necessary for the training set to contain possible translation patterns. To achieve this, a simple yet effective way is to make the train- ing set as large as possible. The training set of EVA contains 848k Ja-En parallel subtitle pairs, 51 7k Zh-En parallel subtitle pairs, and corresponding video clips, which collected from 763 movies and 1, 36:16117 TV episodes with total length 3, hours. The training set much larger than existing VMT datasets and may cover more subtitle transla- tion patterns.
We construct EVA, large-scale parallel subti- tles and video clips dataset, to promote VMT research.
3, 791
We propose the SAFA model with frame at- tention loss and ambiguity augmentation.
We conduct substantial experiments on the EVA dataset with SAFA set benchmark of the dataset.
For testing VMT models, it is inappropriate to general data because many translations do not require visual information. Therefore, we select video-helpful data construct an evaluation set that contains 4, 276 Ja-En parallel subtitle pairs, 2, 940 Zh-En En parallel subtitle pairs, and correspond- ing video clips. Considering the definition of am-
Related Work
Multimodal Machine Translation. MMT in- volves drawing information from multiple modal-

---

Dataset
Domain
Language
Duration
#sent
#video
How2
instruction
En-Pt
90s
189,276
13,662
VATEX
caption
En -n-Zh
10s
349,910
34,991
VISA
subtitle
Ja-En
10s
39,880
39,880
EVA (Ours)
subtitle
Ja-En, Zh-En
10s
1,372,113
1,372,113
Table I: Statistics of VMT datasets. #sent stands for the number of parallel sentences. The duration of videos in the How2 dataset is average duration.
ities, assuming that they should contain useful al- ternative views of the input data (Sulubacak et al.. 2020b). Previous studies mainly focus on IMT us- ing images visual modality to help machine translation (Specia et al., 2016; Elliott al., 2017; Barrault al., 2018; Su et al., 2019; Yang al., 2020; Li et al., 2022a). The usefulness of the visual modality has recently been disputed under specific datasets or task conditions (Elliott, 2018; Caglayan et al., 2019; Wu et al., 2021). However, using im- ages in captions translation is theoretically helpful for handling grammatical characteristics and re- solving ambiguitics when translating between dis- similar languages (Sulubacak et al., 2020b). VMT is a MMT task similar to IMT but focuses on video clips rather than images associated with the textual input. Existing VMT models mainly focus on video caption translation and use video motion features extracted with the I3D (Carreira and Zisserman, 2017) model to help translation (Wang et al., 2019; Gu et al., 2021). The hierarchical attention net- work (Gu et al., 2021) has been proposed further to combine motion features, and object features (Ren al., 2015).
beauthi the
Je PI
Ju #uuvve
Collect Paraliel Subeites and Video Clips
Collect Transiation
Select Video-tteipful Crowdsourcing
Select Ambiguous
Translation Sets
BE You look besutiul
You beautnd
Figure 2: The pipeline of dataset construction.
of the pipeline.
Dataset
In this section, outline the construction of EVA, which is VMT dataset combined with parallel sub- titles and corresponding video clips. The dataset contains a large-scale training set and a small video- helpful evaluation set. For the former, we make it large enough to cover different translation patterns. For the latter, we ensure that the source subtitles are ambiguous and the video clips can help disam- biguate the source subtitles.
VMT Datasets. The scarcity of datasets one the largest obstacles to the advancement of VMT.
3.1 Pipeline
Recent efforts to compile freely accessible data for
To construct the dataset, we first collect a large number of parallel subtitles and one-by-one corre- sponding video clips, where each pair of parallel subtitles matches video clip. For these data, we select some vidco-heipful data to construct the eval- uation set and use the remaining data as the training set. To select video-helpful data, we collect trans- lation sets from the parallel subtitles, select am- biguous translation sets, and do crowdsourcing to further select video-helpful data. We construct the evaluation set with the goal of reaching a specific number instead of covering all video-helpful data. In this way, we can both get video-helpful evalu- ation set and keep the most ambiguous data in the training set. As the dataset construction pipeline is can be extended to other language pairs. The pipeline is shown in Figure 2.
VMT, such as the How2 (Sanabria et al., 2018), VATEX (Wang et al., 2019) and VISA (Li et al.. 2022b) datasets, have begun to alleviate this bottle- neck. Table shows the statistics of existing VMT datasets and EVA. EVA is the largest VMT dataset video hours and a number of video clips and par- allel sentences. Very recently, the BigVides (Kang et al., 2023) dataset consisting of 1.5 million Zh-En sentence pairs and 9, 981 hours of YouTube videos is proposed to facilitate the study of MMT. We consider this work contemporancous our study. The evaluation set of Big Video is annotated by pro- fessional speakers in both Chinese and English to enhance the quality, while we design a language- independent pipeline with translation sets to ensure the scalability of the evaluation set and reusability

---

3.1.1 Collect parallel subtitles and video clips
teadion
reldiion
You look beautifll
We collect parallel subtitles and corresponding video clips following the method in (Li et al., 2022b). On the one hand, the collected data can be used to construct extensive training set. On the other hand, the collected data can be used to further extract the video-helpful evaluation set.
Wow
You look beautifu
ShUVE (oesutiful)
[besutifu)
Noco
Figure 3: The selection of an ambiguous translation set.
Regarding the parallel subtitles, collect Arranese-Engllite and Zh-Ein parallel subtitles from the OpenSubiitles(Lsoon and Tiedemann, 2016) dataset. OpenSubtitles is a subtitles dataset com- piled from an extensive database of film and TV subtitles which includes a total of 1, 689 bitexts spanning 2.6 billion sentences across 60 languages. From OpenSubtitles, we can also collect subti- te timestamps and the Internet Movie Database (IMDb) ids of the video sources.
3.1.3 Select ambiguous translation sets
Then we select ambiguous translation sets with sentence similarity. An ambiguous translation set contains source subtitle and two targer subtitles with different meanings. As shown in Figure 3, select ambiguous translation sets considering two points. On the one hand, we ensure the target subtitles have different meanings. On the other hand, we ensure the target subtitles are parallel with the source subtitle. Subtitles in the OpenSubtitles dataset are provided by volunteers and therefore contain some non non-parallel subtitles. If we only focus on the first point, we might often select the non-parallel target subtitles because they tend to have totally different meaning from other target parallel subtitles.
To collect corresponding video clips, fix sub- title timestamps and crop video clips according to these timestamps. More details can be found in Appendix B.1, Based on accurate timestamps, we crop 10-second 25-fps video clips for parallel sub- titles following (Wang et al., 2019; Li et al., 2022b). From the midpoint of each subtitle's 's period, each video clip takes 5 seconds before and after, respec- tively. The audios of most videos are in English, which is the same as the translation target language and may interfere with translation. Therefore, only keep the video content of video clips and re- move the audio content.
We propose a method to balance the two points discussed above. The main idea is as the following, We first keep the parallel subtitles similarity as high as possible. Then select the most different tar- get subtitles pair. If they are different enough, we combine them and the source subtitle into an am- biguous translation set and discard the remaining data of the translation set. Otherwise, we relax the restriction on parallel subtitles' similarity and re- peat the process above. More details can be found in Appendix B.3. As result, we collected 10, 594 Ja-En ambiguous translation sets and 7, 102 Zh-En ambiguous translation sets.
As a result, we collected 852, 440 Ja-En paral- lel subtitles, 519, 673 Zh-En parallel subtitles, and corresponding video clips. This way, we can con- struet a VMT training set much more extensive than existing VMT datasets.
3.1.2 Collect translation sets
3.1.4 Select video-helpful data via crowdsourcing
After collecting parallel subtitles and correspond- ing video clips, we can collect translation sets from the parallel subtitles. Translation sets are sets of parallel subtitles that have the completely same source subtitles but different target subtitles (i.e., translations). Therefore, large number of parallel subtitles is prerequisite for collecting translation sets. Note that the parallel subtitles with similar but completely the same source subtitles are collected into the translation sets and therefore re- tained in the training set. In this step, we collected 26, 533 Ja-En En translation sets and 17, 642 Zh-En translation sets.
At last, we do crowdsourcing to further select video- helpful data. In order determine whether the video can help disambiguate the source subtitles, the best way should be to distribute tasks to work- ers who can understand both the source and target languages. However, in practice, is hard to find such workers. So we designed a scheme that only requires workers to be able to understand the target language.
In each task, we show two target subtitles from the same ambiguous translation set and one video

---

Split
train
validation
test
subtitle is ambiguous by just checking the source subtitles, and the ambiguity can be easier under- stood by comparing the different target subtitles. For example, " - 2_ (hear) I/2 $ h> (can) ?" is usually translated into Can you hear me?". How- ever, if a person is on the phone and this is his first sentence, "Is there anyone?" is more natural.
#sample (Ja-En)
848,164
2,138 1,470
2.138
#sample (Zh-En)
516,733
1,470
video-helpful
Table 2: EVA splits. "sample" denotes pair of parallel subtitles and a corresponding video clip.
clip that belongs to one of the two subtitles. Then we ask workers if there is any subtitle strongly related to the video content. And give workers four choices: (1) none of them; (2) only the first subtitle; (3) only the second subtitle; (4) both of them. this way, if the video content is only strongly related to the corresponding subtitle, we may estimate that according to the video content, the source subtitle can only be translated to the corresponding subtitle instead of the other one. We did crowdsoureing on Amazon Mechanical Turk. We distribute each task to three workers. Suppose at least two workers agree that the video content is only strongly related to one of the two subtitles, and the subtitle is the corresponding sub- title; in that case, we regard the video clip and corresponding parallel subtitles as video-helpful data. Other techniques improve crowdsourcing quality can be found in Appendix B.4. As a result, we constructed vidco-helpful evaluation con- taining 1, 276 Ja-En parallel subtitle pairs, 2, 940 Zh-En parallel subtitle pairs, and corresponding video clips. And we equally divide this set into a validation set and a test set.
4 Model
We present a new model, SAFA, for VMT based on the EVA dataset. Previous VMT models mainly focus on caption translation based on the exist- ing video caption datasets. In contrast, our model mainly focuses subtitle translation. We design SAFA based a selective attention model, pro- pose a frame attention loss to make the model focus on the central frames where the subtitles occur, and use ambiguity augmentation to make the model put more weight on the possibly-ambiguous data. The model overview is shown in Figure 4.
4.1 Selective attention model
There are few existing VMT models, and most of them based LSTM GRU models (Wang et al., 2019; Gu et al., 2021). The selective attention model is an IMT model recently proposed (Li et al., 2022a), which based on the Transformer model and has the advantage of simplicity and ef- ficiency. We replace the image feature extraction models with video feature extraction models to fit the VMT task. The model mainly consists of the following five modules.
We calculated the inter-annotator reliability to evaluate the crowdsourcing results. Krippendorff's alpha (Krippendorff, 2011) is 0.681 and 0.728 for Ja- En and Zh-En. Both of them achieve the sub- stantial agreement (Hughes, 2021).
Text transformer encoder. The text trans- former encoder follows the transformer encoder- decoder paradigm( (Vaswani et al., 2017). The input is the source text, and the output the text repre- sentation.
3.2 Dataset analyses
Video featurre extraction model. The video feature extraction model's input is the video frames, and the output is the video feature.
Table shows the splits of EVA. The evaluation set contains many kinds of ambiguities. We checked 50 samples. The most frequent causes are omission, emotion, and polysemy, with approximate propor- tions of 30%, 30%, and 20%. For the remaining, it is difficult to define the causes of ambiguity. Some- times, instances of ambiguity arise from a com- bination of various factors, thereby challenging precise classification. For example, "Ij !h! can be translated as both "Let me go!" and "Drop it!." This ambiguity could arise due the polysemy !" or it could stem from the omission of the object. And sometimes, is difficult to tell whether
Selective attention. Given the text representa- tion text and the video feature H the se- lective attention mechanism tion network to correlate words with vidco frames, where the query, key, and value are Hlext Hoiden and Hvideo respectively. The selective attention output H attn can be defined as:
single-head atten-
QKT
Hviden Softmax
(I)
Vdl.
where dy is a scaling factor.

---

Hedd
Attention
Taayy
Traneformor Decoder
video featuriis
fx
Video Feature Model
Trinsformar Encoder
You look beautfull
Poweliy- Ambiguuu
Video
Figure 4: The SAFA model with frame attention loss (top left in red) and ambiguity augmentation (right in green). The frame attention loss uses Gaussian distribution to guide the model to pay more attention to the central frames, while the ambiguity augmentation makes the model put more weight on the data with possibly ambiguous source subtitles.
Gated fusion The gated fusion mechanism is popular technique for fusing representations from different sources (Wu et al., 2021; Fang and Feng, 2022; Lin et al., 2020; Yin et al., 2020). The fused output is a weighted sum between the text represen- tation and the selective attention output, in which the weight is controlled with the gate X. The gate A E [0, 1] and the fused output can be defined as:
videos. Inspired by (Li et al., 2020), we propose frame attention loss that uses Gaussian distribution guide the attention on video features.
For a random variable X ' (H, o²) in which = and a = 1, we define fx (œ) as the prob- ability density function of X. We define z
22, 22. =M] in which 21 to ZM are M points equally spaced from -a to and M is the number of frames in a video feature. Then the frame atten- tion loss is defined the Beilbackk-eibblle (KL) divergence between the frame attention defined in Eq. (1) and the Gaussian distribution with the soft- max temperature (Hinton et al., 2015) mechanism:
X Sigmoid
(UHte + V Hyideo
Hout = A) Hlext A
Hvideo attn
_ = KK (Softmax
QK" vd,
Sootmaxx (fx(z))
where U and V are trainable variables. Then, the fused output yo is fed into the decoder.
(4)
where KT.() is the KI. divergence function and Softmax; the softmax temperature mechanism:
Transformer decoder The transformer de- coder also follows the transformer decoder paradigm. The difference is that the cross-atee vrossaateetiion block uses the fused output instead of the text rep- resentation as key and value.
Softmax, (x) = exp (a/TxT
(5)
Σ,exp exp (2z/T)
where T (0, is temperature parameter, When T gets smaller, the distribution tends to a Kronecker distribution (and is equivalent to a one- hot target vector), and the model will pay more attention to the central frames; when T gets larger, the distribution tends to uniform distribution, and the model will pay equal attention to all the frames. As we can substantially change the uniformity of the distribution by adjusting T, we fix reduce the number hyperparameters.
The loss function _ is cross entropy loss with label smoothing (Szegedy et al., 2015).
4.2
Frame attention loss
Unlike captions which describe the entire video clip, subtitles only occur for few seconds in video clip. Generally, the frames close to the subti- tles more associated with the subtitles and thus provide more information associated with the sub- titles. Therefore, we hope the model can pay more attention the frames close to the subtitles. In EVA, the subtitles occur the center of the video clips because we have aligned the subtitles to the
With the frame attention loss, the loss function can be defined as:
-- 5 +7CG
(6)

---

where y is hyperparameter.
the results of single experiments following the Spa- tial HAN model (Gu et al., 2021). Moreover, we reported the statistical significance of BLEU using bootstrap resampling (Koehn, 2004) over a merger of three test translation results.
4.3 Ambiguity augmentation
VMT, the video can help with translation only when the source subtitles are ambiguous. There- fore, we hope the model puts more weight on the data with ambiguous source subtitles.
5.2 Compare with SOTA
For VMT dataset X [21; xN 2t is data consisting of a pair of parallel sub- titles and corresponding video clip. We di- vide the dataset into
So far, there are very few VMT models. Because the Spatial HAN model (Wang et al., 2019) is not available, use the publicly available VMT model proposed in VATEX (Wang al., 2019) baseline. Existing VMT models (Wang et al., 2019; Gu et al., 2021) are mainly based on LSTM or GRU NMT model, while our model is based on the Transformer NMT model. As shown in Table 3, even the text-only model alone, which is the standard text-only Transformer model, per- forms much better than the previous VMT models. Therefore we focus the comparison between our model and the text-only model. Compared to the text-only model, SAFA has comparable pa- rameters while demonstrating significant perfor- mance gains. Specifically, for Zh-Ein translation. the achieves 9.99% improvement in BLEU score (absolute: 1.40) and 2.75% improvement in ME- TEOR score (absolute: 0.96). Furthermore, for Ja- En translation, the achieves 4.94% improvement in BLEU score (absolute: 1.30) and 3. 3.28% improve- ment in METEOR score (absolute: 1.55).
dataset X = {-f. 2z.... ......... ..p.. and dessibly-unambignn dataset X = { "r......... # according to whether the source subtitle of z; in a trans- lation set or not, where P and Q are the num- data and possibly- unambiguous data respectively. The loss function of the selective attention model is defined as:
bers of
N
Lr.
N
2=1
where is the number of data and C,, is the loss of the data 2ú. Then, with ambiguity augmentation, the loss function is defined as:
1 P L = w P
Q
C.n +
Σ Cag
(8)
where 1 is a weight increase the loss of possibly-anbiguous data, making the model put more weight on the possibly-ambiguuus data.
Furthermore, we did context translation exper- iment following (Tiedemann and Scherrer, 2017) to check whether local contextual information can help disambiguate the source subtitles. We com- bine each subtitle with the two subtitles before and after it to construct context data and train the text-only model context data. As shown in Ta- ble 3, the text-only model using context data per- forms similarly using single subtitle data. The reason why context does not enhance translation performance may be the lack of speaker identity. The results indicate that visual information is more helpful than local contextual information in this translation task.
5 Experiments
5.1 Settings
We conducted experiments with the Transformer configuration following (Li et al., 2022a). We train the Transformer from scratch without using external text data. For the video feature extrac- tion model, we used CLIP4Clip (Luo al., 2022). More details can be found in Appendix C.
We adopted BLEU (Papineni et al., 2002; Post, 2018) and METEOR (Banerjee and Lavie, 2005) as the evaluation metries. Subtitles are essentially dialogues that are often short. Therefore we intro- duced the METEOR score in addition to BLEU. For experiments in Sections 5.2 and 5.3, re- peated the experiment five times for each setting, discarded the maximum and minimum scores, and then took the average of the remaining three scores as a result. For experiments in Section 5.4, we used
5.3 Effectiveness of the two proposed methods In the SAFA block of Table 3, we conducted abla- tion studies by removing the frame attention loss (w/o Frame Attn), ambiguity augmentation (w/o Ambi Aug), and both (w/o Both). We can see that w/o Frame Attn decreases the performance more than w/o Ambi Aug, and w/o Both further decreases significantly. SAFA w/o Both (i.e., the

---

Ja-En
Zh-En
Method
#param
BLEU
METEOR
#param
BLEU
METEOR
Text-only
15.35M
14.01
34.90
12.71M
26.32
47.19
Text-only (context)
15,35M
14,10
33.57
12.71M
26.40
47.83
VATEX
52.41M
12.24
32.07
36.87M
22.15
44.68
SAFA
15,55M
15.41
35.86
12.91M
27.62
48.74
w/o Frame Attn
15.55M
14.98
35.66
12.91M
27.41
48.59
w/o Ambi Aug
15,55M
15.12
35.55
12.91M
26.81
48.30
w/o Both
15.55M
14.72
35.23
12.91M
26.55
48.47
Table 3: Experiments the EVA dataset. indicates that the result is significantly better than text-only, text-only (context), and VATEX at < 0.01, respectively.
Method
#param
BLEU
METEOR
SRC BELL (snouthe))
REP: You looe besuritu
NMT Boouthll
Text-only
8.13M
13.01
28.22
VMT You benuthu
VATEX
52.41M
10.50
25.31
BRC. In B.98EE-0 ST (Burkk))
REF: Thans you oir
Spatial HAN
57.78M
13.19
28.26
MMT Thank you
SAFA
8.33M
13.86
29.09
Trams
SRC RM0 (ddee))
REP: Cloaa the doot men
Table Experiments on the VISA dataset. t indicates
NMTI C Coea i.
VMTI Cleee
that the result is significantly better than text-only, VA- TEX, and Spatial HAN at < 0.01.
SRC: Is (1e) CCua (maybo)
REP: On no
NNTI: Youlre (sin VMT
Figure 5: Qualitative examples for NMT and VMT. The VMT model is the SAFA model.
selective attention model) characterized by its straightforwardness, but does not account for the significance of the central frames where subtitles occur, nor does it allocate additional attention to ambiguous data. Therefore, it cannot take full ad- vantage of the video information.
are not as large as that on EVA. On the one hand, due to the size of the VISA dataset, it only con- tains small number of translation sets and small number of possibly-ambigue the performance improvement of the model mainly comes from the frame attention loss method. On the other hand, the videos VISA's evaluation set do not necessarily contribute to disambiguation.
data. We think that
5.4 Effectiveness of SAFA on other VMT datasets
Considering the How2 dataset is an instruction sub- title dataset with long videos while VATEX is conducted additional experi-
caption dataset, ments on the VISA (Li al., 2022b) dataset that is also a subtitle dataset to test the model's gener- alization ability across datasets. Table shows the results. While the Spatial HAN model not avail- able, we obtained the translation results on VISA for comparison.
5.5 Case Study
Finally, we compare several real cases to see how visual information helps translation. We choose text-only and SAFA translation results. Figure 5 shows four qualitative examples in which the VMT model uses visual information to improve the trans- lation. Generally, subtitles from movies and TV episodes are usually short, video can help promote the interpretation of the subtitles. In the first ex- ample of omission ambiguity, the man speaks to the woman while looking at her. Therefore, we can infer that the subject pronoun "you" is omitted in the source caption rather than an object or scene. In the second example, two men in suits are talk- ing. Considering politeness, the address should be added. And considering the gender, the address should be "sir." In the third example, the video shows the door closed, so the VMT model did a more generative translation. In the last example emotion ambiguity. the woman shows a surprised
It's worth noting that the Spatial HAN model is based on the winning model in the VMT Challenge competition,' which is a GRU model. In contrast, the VATEX model based on LSTM. As a result, there a significant performance gap between the Spatial HAN and VATEX models. Because VISA is relatively small and thus more appropriate for models based on GRU instead of Transformer, our text-only model performed similarly to the Spatial HAN model (Gu et al., 2021) model. We can see that the SAFA model performs significantly bet- ter than other models, although the improvements

---

expression, and it is clear that the woman is not talking someone clse according other frames, Therefore, the source subtitle should be translated as an expression of surprise rather than a statement that another person is lying. In this case, the VMT model does not translate correctly, but its transla- tion contains emotional information. Appendix D shows the frame attention of the examples.
provide a self-declaration that the data is to be used solely for research purposes.
Acknowledgement
This work was supported by Google Research Scholar Award and JSPS KAKENHI Grant Num- ber JP23H03454.
6 Conclusion
The paper introduced a new VMT dataset called EVA, which contains an extensive training set and a vidco-helpful evaluation set in which videos are guaranteed to be helpful for disambiguating source subtitles. In addition, we proposed a novel VMT model called SAFA that incorporates selective at- tention with frame attention loss and ambiguity augmentation. Experiments on EVA demonstrated that visual information and the proposed methods can boost translation performance, and SAFA per- forms significantly better than previous VMT mod- els. We hope that this work will inspire further rescarch in this field.
References
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. Computer Vision-ECCV 2016, pages 382-398.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations. ICLR 2015. San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings.
Satanjeev Banerjee and Alon Lavie. 2005. ME- TEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACI. Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. pages 65-72. Ann Arbor, Michigan. Association for Computational Linguistics.
7 Limitations
The main limitations of our dataset are the follow- ing. First, the subtitles are from the OpenSubtitles datasct and provided by volunteers, which ensures the size of the dataset but results in the dataset containing some low-quality parallel subtitles. We used methods such as cross-lingual similarity to fil- ter high-quality parallel subtitles when constructing the evaluation set but could not wholly remove low- quality subtitles. Second, each task in the crowd- sourcing is only distributed to three workers due to financial constraints. If we distribute the tasks to more workers and expand the crowdsourcing scale, it is possible to obtain a higher-quality evaluation set.
Loic Barrault, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, and Stella Frank. 2018. Find- ings of the third shared task on multimodal machine translation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 304-323, 323, Belgium, Brussels. Association for Com- putational Linguistics.
Ozan Caglayan, Pranava Madhyastha, Lucia Spe- cia, and Loic Barrault. 2019. Probing the need for visual context in multimodal machine trans- lation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4159-4170, Minneapolis, Minnesota. Asso- ciation for Computational Linguistics.
8 Ethical Statements
We aim to facilitate MMT, so our dataset and codes will be publicly released. The subtitles utilized in this study are collected from publicly avail- able datasets, while the videos are extracted from movies and TV episodes. We only use 10-sccond video clips associated with subtitles address copyright concerns, with the audio removed. We will require all users to provide their academic af- filiation as a condition to access the data. Besides, we may ask users intending to access our data to
Joao Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? new model and the kinetics dataset. proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308
Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. 2021. Up-detr: Unsupervised pre-training for ob- ject detection with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1601-1610.

---

Desmond Elliott. 2018. Adversarial evaluation of mul- timodal machine translation. In Proceedings of the 2018 Conference on Empiriaal Methods in Natural Language Processing, pages 2974-2978, Brussels, Belgium. Association for Computational Linguistics.
Companion Volume Proceedings of the Demo and Poster Sessions, pages 177-180, Prague, Czech Re- public. Association for Computational Linguisties.
Klaus Krippendorff. 2011. Computing krippendorff's alpha-reliability
Desmond Elliott, Stella Frank, Loic Barrault, Fethi Bougares, and Lucia Specia. 2017. Findings of the second shared lask on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation, pages 215-233, Copenhagen, Denmark. Association for Computational Linguistics.
Bei Li, Chuanhao Lv, Zefan Zhou, Tao Zhou, Tong Xiao, Anxiang Ma, and JingBo Zhu. 2022a. On vision features in multimodal machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume
1: LLonn Papers), pages 6327 -377-337, Dublin, Ireland. Association for Computational Linguisties.
Desmond Elliott, Stella Frank, Khalil Sima'an an, and Lu- cia Specia. 2016. Multi30K: Multilliguul English- German image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pages 70- 74, Berlin, Germany, Association for Computational Linguistics.
Yihang Li, Shuichiro Shimizu, Weiqi Gu, Chenhui Chu, and Sadao Kurohashi. 2022b. VISA: An am- biguous subtitles dataset for visual scene-aware ma- chine translation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6735-6743, Marseille, France. European Lan- guage Resources Association.
Qingkai Fang and Yang Feng. 2022. Neural machine translation with phrase-level universal visual rep- resentations. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 5687- 5698, Dublin, Ireland. Association for Computational Linguistics.
7. Li, Rui Wang, Kehai Chen, Masso Utiyama, Ei- ichiro Sumita, Zhuosheng Zhang, and Hai Zhao. 2020. Data-dependent gaussian prior objective for language generation. In ICLR.
Weiqi Gu. Haiyue Song, Chenhui Chu, and Sadao Kurohashi. 2021. Video-guided machine transla- tion with spatial hierarchical attention network. In Proceedings of ACL-IJCNL
Huan Lin, Fandong Meng, Jinsong Su, Yongjing Yin. Zhengyuan Yang. Yubin Ge, Jie Zhou, and Jiebo Luo. 2020. Dynamic context-guided cap- sule network for multimodal machine translation. In Proceedings the 28th ACM International Conference on Multimedia, pages 1320-1329.
pages 87-92.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network.
John Hughes. 2021. krippendorffsalpha:: An R Pack- age for Measuring Agreement Using Krippendorff's Alpha Coefficient. The R Journal, 13(1):413-42
Pierre Lison and Jörg Tiedemann. 2016. OpenSub- titles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC' 16), pages 923-929, Por- torož, Slovenia. European Language Resources As- sociation (ELRA).
Liyan Kang. Luyang huang, Ningxin Peng, Peihao Zhu, Zewei Sun, Shanbo Cheng. Mingxuan Wang, Degen Huang, and Jinsong Su. 2023. Bigvideo: A large- scale video subtitle translation dataset for multimodal machine translation. ArXiv, abs/2305.182 18326.
Huaishao Luo, Lei Ji. Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. 2022. Clip4elip: An empirical study clip for end to end video clip retrieval and captioning. Neurocomputing, 508:293- 304.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR (Poster).
Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388-395, Barcelona, Spain. Association for Computational Lin- guistics.
Christopher Manning, Mihai Surdcanu, John Bauer, Jenny Finkel, Steven Bethard, and David Mc- Closky. 2014. The Stanford CoreNIP natu- ral language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55 60, Baltimore, Maryland. Association for Computational Linguistics.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondfej Bojar, Alexan- dra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine transla- tion. Proceedings the 45th Annual Meeting of the Association for Computational Linguistics
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.

---

Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 31 311-,18, Philadel- phia, Pennsylvania, USA. Association for Computa- tional Linguisties.
Jörg Tiedemann and Yves Scherrer. 2017. Neural machine translation with extended context. In Proceedings of the Third Workshop on Discourse in Machine Translation, pages 82-92, Copenhagen, Denmark. Association for Computational Linguis- tics.
Arseny Tolmachev, Daisuke Kawahara, and Sadao Kurohashi. 2018. Juman++: A morphologi- cal analysis toolkit for scriptio continua. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 54 59, Brussels, Belgium. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Belgium, Brussels. Association for Computa- tional Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNIP--JCNIP), pages 3982-39999 3992, Hong Kong. China. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need, In Advances in Neural Information Processing Systems, volume 30, Curran Associates, Inc.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image de- scription evaluation. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEFE.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object de- tection with region proposal networks. Advances in Neural Information Processing Systems, vol- ume 28. Curran Associates, Inc.
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan- Fang Wang, and William Yang Wang. 2019. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4581 4591.
Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loic Barrault, Lucia Specia, and Florian Metze. 2018. How2: A large-scale dataset for multimodal language understanding. In NeurIPS.
Yonghui Wu, Mike Schuster; Zhifeng Chen, Quoe Le, Mohammad Norouzi. Wolfgang Macherey, Maxim Krikun, Yuan Cao. Qin Gao, Klaus Macherey, al. 2016. Google's neural machine translation system: Bridging the gap between human and machine trams- lation. arXiv preprint
Lucia Specia, Stella Frank, Khalil Sima'an, an, and Desmond Elliou.. 2016. A shared task on multi- modal machine translation and crosslingual image de- scription. In Proceedings of the First Conference on Machine Tramslation: Volume 2, Shared Task Papers. pages 543-553, Berlin, Germany, Association for Computational Linguistics.
144.
Zhiyong Wu, Lingpeng Kong, Wei Bi. Xiang Li. and Ben Kao. 2021. Good for misconceived rea- sons: An empirical revisiting on the need for vi- sual context in multimodal machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6153-6166, Online. Association for Computa- tional Linguistics.
Yuanhang Su, Kai Fan. Nguyen Bach, C.-C. Jay Kuo, and Fei Huang. 2019. Unsupervised multi-modal neural machine translation. In Proceedings of CVPR, pages 10491.
Umut Sulubacak, Ozan Caglayan, Stig-Ame Grönroos, Aku Rouhe, Desmond Elliott, Lucia Specia, and Jörg Tiedemann, 2020a. Multimodal machine translation through visuals and speech. Machine Translation, 34(2):97-447.
Pengcheng Yang. Boxing Chen, Pei Zhang, and Xu Sun. 2020. Visual agreement regularized training for multi-modal machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 34, pages 9418 1448-5...
Umut Sulubacak, Ozan Caglayan, Stig-Are Grönroos, Aku Rouhe, Desmond Elliott, Lucia Specia, and Jörg Tiedemann. 2020b. Multimodal machine translation through visuals and speech. Machine Translation, 34(2):97-447.
Yongjing Yin, Fandong Meng, Jinsong Su, Chulun Zhou, Zhengyuan Yang, Jie Zhou, and Jiebo Luo. 2020. novel graph- based multi-moda fusion en- coder for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3025-3035, On- line. Association for Computational Linguistics.
Christian Szegedy, Vincent Vanhoucke, Sergey Toffe, Jonathon Shiens, and Zbigniew Wojna. 2015. Re- thinking the inception architecture for computer vi- sion. CoRR, 10.0157.

---

Zhuosheng Zhang, Kehai Chen, Rui Wang, Masao Utyama, Eiichiro Sumita, Zuchao Li, and Hai Zhao. 2020. Neural machine translation with universal vi- sual representation. In International Conference on Learning Representations.
Ja-En
Zh-En
Method
CIDEr
SPICE
CIDEr
SPICE
Text-only
1.3167
6.56
2.0697
11.24
Text-only (context)
1.2571
6.40
2.0805
11.50
VATEX
1.0919
5.64
1.8516
9.39
Yuting Zhao, Mamoru Komachi, Tomoyuki Kajiwara, and Chenhui Chu. 2020. Double attention-basee multimodal neural machine translation with seman- tic image regions. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 105-1 14, Lisboa, Portu- gal. European Association for Machine Translation.
SAFA
1.3942
6,80
2.1113
11.34
w/o Frame Attn
1.3726
6,80
2.1346
11,49
w/o Ambi Aug
1.3758
7.09
2.1143
11.12
w/o Both
1,3380
6.64
2.1270
11.14
Table 5: Experiments on the EVA dataset.
A
Other Experiments
A.3 Experiments on other VMT evaluation set
We conducted experiments on VISA's training set and EVA's evaluation set to check the necessity the EVA's training set. The BLEU and METEOR scores of the SAFA model are 6.77 and 25.17, re- spectively. Both are significantly lower than the SAFA results in Table 3. The results indicate that EVA's training set is more helpful.
A.1 Compare with IMT method
To compare the performance against image-based methods, we conducted IMT experiments using the Selective Attention model. In this setting, we extracted the central frame from each video clip and obtained the DETR (Dai et al., 2021) image feature of these frames guide translations. For Ja-En translation, the BLEU score and METEOR score are 14.63 62622222 and 34.93, respectively. For Zh-En translation, the BLEU score and METEOR score are 26.36 and 47 17.58, respectively. Compared with the results in Table 3, the IMT method has better performance than the text-only method but not as good the VMT method using CLIP4Clip fea- tures (w/o Both). Although the central frames strongly associated with the subtitles, they cannot adequately capture the contextual information nec- essary for interpreting the subtitles. Sometimes the central frame may solely display the speaker's s face, while relevant information, such as the object re- ferred to in the subtitle, may appear before or after it.
A.4 Other evaluation scores
We calculated the CIDEr (Vedantam et al., 2015) and SPICE (Anderson et al., 2016) scores for the main results. The results are shown in Table 5, Since we focus on the subtitles translation task instead of the video caption generation task, we add the results as a reference.
B Additional Details for Dataset
B.1 Fix subtitle timestamps
The subtitle timestamps collected from the Open- Subtitles dataset are provided by volunteers and may not match the video. Therefore we need to align subtitles to videos to fix the timestamps. In practice, we use alass² to align subtitles videos. Alass can perform subtitles alignment in two ways. One is to align subtitle files with incorrect times- tamps to subtitle files with correct timestamps, such as those extracted from videos. The other is to align the incorrect subtitle file with the correspond- ing video using voice activity detection (VAD). We combine the two methods to do alignment and man- ually check the results. In this way, make sure that the timestamps of the subtitles correspond ex- actly to the time when the subtitles appear.
A.2 Effect on randomly divided evaluation set
To check the performance of the model on a test set with the original distribution, we conducted experi- ments on the Ja-En part of EVA with a randomly divided training set and evaluation set instead of us- ing a video-helpful evaluation set. The size of each set was equivalent to that of EVA. The BLEU and METEOR scores of the text-only model are 13.64 and 41.58, respectively, while those of SAFA 13.77 and 11.41, respectively. We can that the models have similar performance. Many sam- ples in the randomly divided test set do not require disambiguation. Therefore, videos can not signifi- cantly help the model improve its performance.
Phaps:/giittub.oomkkaaailaass

---

B.4 Crowdsourcing
The number translalonn sels
15000
The crowdsourcing interface is shown in Figure 7. In the instructions, we tell workers that the subtitles occur in the center of the video clips. Because the language of most video clips is English, the workers may choose the subtitle according the character's 's lip movements. Therefore we especially tell the workers to choose based on the content of the videos rather than the characters' lip movement. Moreover, we state that the video only shows some people talking with each other and nothing special, it should not be regarded as strongly related to any subtitle.
22500
10000
7500
5000
2500
12
The size of transtation sets
Figure 6: The size distribution of translation sets. (Until size of 15)
B.2 Size distribution of translation sets
We use qualification tests further improve the quality of crowdsourcing. Specifically, we set qual- ification test tasks to check whether workers can answer them correctly. Then we only distribute large-scale tasks to workers who are good at an- swering this kind of task.
We examined the size distribution of translation sets in the Ja-En and Zh-En En parts, separately. The result is shown in Figure 6.
B.3 Select ambiguous translation sets
We sent-BERT (Reimers and Gurevych, 2019) calculate the target subtitles similarity and use the cross-lingual version of sent-BERT to calculate the parallel subtitles similarity. Sent-BERT is one the best methods to evaluate semantic similarity. In each translation set, for parallel subtitles with similarities higher than threshold Tp. we select an ambiguous translation set and only the sim- ilarity between the two most different target sub- titles is lower than a threshold T,. Otherwise, we lower the Tp. and repeat the process above. Finally, we collect all the ambiguous translation sets under each Tp as result. Note that we select most ambiguous translation set containing two tar- get subtitles from each translation set. Therefore the remaining data of translation sets are retained the training set.
C Detailed Experimental Setup
The Transformer model consists of 1 encoder and decoder layers. The input/output layer dimension is 128, and the inner feed-forward layer dimension is 256. There are heads in the multi-head self- attention mechanism. We set the dropout as 0.3 and the label smoothing as 0.1.
We searched for the hyperparameters separately for the frame attention loss method, the ambiguity augmentation method, and the combination method (SAFA). For the SAFA model, set T 1, 2. For the frame attention and y For
7 0.5, and loss method only, we the ambiguity augmentation method only, we set w = 2.
Our implementation was based on Fairseq (Ott et al., 2019). For training, we used Adam Op- timizer (Kingma and Ba, 2015) with 31 0.9. B2 0.98 and 10 We adopted the same learning rate schedule (Vaswani et al., 2017). where the learning rate first increased linearly for warmup = 2, 000 steps from -T to -3 After the warmup, the learning rate decayed proportion- ally to the inverse square root of the current step. Each training batch contained 16, 000 tokens. We also adopted the early-stop training strategy (Zhang et al., 2020) avoid the overfitting issue.
We set T, and Tp separately. To set the thresh- old Tú, we do experiments 100 randomly se- lected translation sets. We generate ground truth by manually checking whether the source subtitle of each translation set is ambiguous and calculate the precision and recall under different T,. As we do crowdsourcing later to collect final results, re- call is more important than precision in this step. We set T, 0.3 with recall 0.56 and precision 0.38. Similarly, we do experiments with 200 ran- domly selected parallel subtitles to set Tp. When we only keep parallel subtitles with sentence simi- larity higher than 0.3, recall is 1.00 while precision is 0.90. We relax Tp from 0.8 to 0.3 with 0.1 inter- val in sequence.
We used Juman++ (Tolmachev et al., 2018). Stanford CoreNLP (Manning et al., 2014), and Moses (Koehn et al., 2007) to tokenize Japanese, Chinese, and English subtitles, respectively. On

---

Instruction:
Given a video (without sound) and two subéiles, there any subtidee strongly related to the video content (eg. object, gender of characters, action, obvieus emotion ..]? Please select most appropriate prropiate option.
helpful
find
videod rather than the characters' movement. if the video only shows some people talking with each other and nothing special, should not be regarded as strongly related to any subtitie.
Notes:
Up movaments match subtities are not considered relation. (Lika example) most questions, the subtites occur at the midcla of the videos (at around soconds). Each task contains questions, Please answer of them, then click "subrit"..
Example:
Question
For the following two subtities, is there any one strongly related content?
Wait
B. Please!
O None of them Only subtitle
Only subtitle B
Both of them
Figure 7: Crowdsourcing interface.
the EVA dataset, we mapped tokens appearing less than three times to unknown. As a result, in Ja- En translation, the Japanese vocabulary contains 38, 516 tokens while the English vocabulary con- tains 35, 540 tokens. In Zh-En translation, the Chi- nese vocabulary contains 34, 860 tokens while the English vocabulary contains 27, 028 tokens. On the VISA dataset, we used tokens to build the vocabulary. As result, the Japanese vocabulary contains 17, 676 tokens while the English vocabu- lary contains 17, 732 tokens. Therefore the number of model parameters in Tables 3 and 4 are different.
as the man says the sentence to a dog and he does not point to a special position, the source subtitle should be translated to "get away." In this example, the model pays and sixth frames containing the dog.
attention to the fourth, fifth,
The ambiguity of the first three examples is caused by omission, and the fourth is caused by emotion. The ambiguity of the last example is not caused by omission or emotion, and we approxi- mately classify it as an ambiguity caused by the polysemy "V > (go)."
D Frame Attention Analysis
We show some examples of frame attention. We show four examples from the case study section (Section 5.5) and one additional example. The examples are shown in Figures 8, 9, 10. 11, and 12. In the source sentence, each token has its attention on different frames. In the first two exam- ples, most frames can help disambiguate the source subtitles. The model pays more attention to the first half of the video clips. In the third example, the token " s X (close)" pay more attention to the fourth and fifth frames, which contain the door. Therefore, the model translates the source subtitle to Close the door" instead of closing other things. In the fourth example, both tokens pay much atten- tion the third and fifth frames. Especially the fifth frame shows the surprised expression of the woman. Therefore, the source subtitle is translated as an expression of surprise. the last example,

---

SRC: v (beautiful)
NMT: Beautiful
REF: You look beautiful
VMT: You look beautiful
オ (beautiful)
0.0935
0.2618
B.0670
0.0477
0.0911
0.0746
2.0988
0.0748
0.0555
0.0421
Figure 8: Frame attention of tokens the first case study example. The attention weight of each frame is top of the frame. The blue-to-red frame filter indicates low to high frame attention.

---

SRC: B (thanks)
NMT: Thank you
REF: Thank sir
VMT: Thank you, sir
おシズΚ (thanks)
10107
0.0962
0.0739
0.1583
0.0498
0.0589
0.0894
0.0637
ビ polite porm
0.0760
0.0647
0.1210
0.1201
0.0912
0.1470
0.1619
0.0434
0.0361
0.0463
0.0420
0.0303
Figure 9: Frame attention of words in the second case study example. The attention weight of each frame on top the frame. The blue-to-red frame filter indicates low high frame attention.

---

SRC: 3 (close)
NMT: Close it!
REF:
Close the door, man!
VMT: Close the door!
8 3 (close)
0.0519
0.0853
0.0659
0.1259
0,0341
0.0647
0.0933
0.0474
0.1165
0.1086
0.0555
0.1497
0.1058
0.1329
0.1339
0.0809
B.0346
0.0259
0.0637
0.0985
0,0704
Figure 10: Frame attention of words in the third case study example. The attention weight of each frame on top of the frame. The bluc-to-red frame filter indicates low high frame attention.

---

SRC: B2 (lie) &L (maybe)
NMT: You're lying
REF: Oh, no
VMT: Oh, shit
(iie)
0.1060
0.0639
0.1394
0. 0.0676
0.0969
0.0868
0.0771
0.0461
LL (maybe)
0
0.0828
0,1124
0.0744
0. 0.1292
0.0777
0.0679
0.0656
0.0931
0.0931
0.0732
Figure II: Frame attention of words in the fourth case study example. The attention weight of cach frame is on top the frame. The bluc-to-red frame filter indicates low high frame attention.

---

SRC:
おクラ (there) ~ (to) fīl+ (go)
NMT: Go over there.
REF:
Get away.
VMT: Go away.
おクら (there)
0.0743
0.0755
0.1149
0.0738
0.1223
0.1637
0.2068
0.0435
0.0380
0.0234
0.0284
0.0295
(to)
0.1344
0. 1441
0.0744
0.0322
Fīlカ (go)
0.0690
0.0421
0.0302
0.0408
0.0434
0.1058
0.0578
0.0199
Figure 12: Frame attention of words in the fifth example. The attention weight of each frame is on top of the frame. The blue-to-red frame filter indicates low to high frame attention.


Reviews:
[{"review_id": "I9thATXGoR", "reviewer": "Reviewer_3gan", "summary": "", "questions": "1. Have you considered other evaluation scores, such as Cider and Spice, as they are also popular metrics in video captioning?\n2. After collecting translation sets(Sec 3.1.2), what is the distribution of the size of the translation set? I think it is essential information about the ambiguity degree of the task.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and efforts. Please find the responses (**R**) to the concern points (**P**).\n\n**P1**: The definition of ambiguity\n\n**R1**: To clarify the definition, we will add \"that needs multimodal information for disambiguation\" at the end of the definition sentence.\n\n**P2**: Other evaluation scores\n\n**R2**: As shown in the following table, we calculated the CIDEr and SPICE scores for the main results. Since we focus on the subtitles translation task instead of the video caption generation task, we will add the results to the appendix as a reference. \n\n|                     |    Ja-En   |     Ja-En     |    Zh-En   |     Zh-En      |\n|:---------------------|:----------:|----------:|:----------:|----------:|\n| **Method**          | **CIDEr**  | **SPICE** |  **CIDEr** | **SPICE** |\n| Text-only           | 1.3167     |      6.56 |     2.0697 |     11.24 |\n| Text-only (context) | 1.2571     |      6.40 |     2.0805 | **11.50** |\n| VATEX               | 1.0919     |      5.64 |     1.8516 |      9.39 |\n| SAFA                | **1.3942** |      6.80 |     2.1113 |     11.34 |\n| - w/o Frame Attn    | 1.3726     |      6.80 | **2.1346** |     11.49 |\n| - w/o Ambi Aug      | 1.3758     |  **7.09** |     2.1143 |     11.12 |\n| - w/o Both          | 1.3380     |      6.64 |     2.1270 |     11.14 |\n\n\n**P3**: The distribution of the size of translation sets\n\n**R3**: Due to the rebuttal format, we regrettably can not show the distribution image here. Instead we show the table here. We did statistics on the distribution of the size of the translation sets. We will add the results to the appendix. \n\n| Size of Translation Sets |     2 |    3 |    4 |    5 |   6 |   7 |   8 |   9 |  10 |  11 |  12 |  13 |  14 |  15 | >15 |\n|--------------------------|------:|-----:|-----:|-----:|----:|----:|----:|----:|----:|----:|----:|----:|----:|----:|----:|\n| #Ja-En Translation Sets  | 20661 | 5742 | 2734 | 1466 | 895 | 637 | 474 | 349 | 277 | 242 | 189 | 142 | 115 | 105 | 995 |\n| #Zh-En Translation Sets  | 10459 | 2921 | 1324 |  746 | 499 | 313 | 245 | 166 | 145 | 116 |  79 |  70 |  41 |  58 | 460 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The ambiguity issue of multimodal translation is necessary and unique. The authors propose such a large-scale dataset, which is helpful to this area.\n2. The authors design two methods during training to alleviate such an issue.", "reasons_to_reject": "The paper's definition of ambiguity (L42-L45) seems too general and vague. I think the ambiguity here mainly refers to the source texts, which must be disambiguated by the video. So the type of ambiguity, such as polysemy ambiguity (20% according to L358), is not suitable for the ambiguity in question because they, in many cases, can be determined by the context (Like \"bank\" in \"river bank\").", "questions_to_authors": "1. Have you considered other evaluation scores, such as Cider and Spice, as they are also popular metrics in video captioning?\n2. After collecting translation sets(Sec 3.1.2), what is the distribution of the size of the translation set? I think it is essential information about the ambiguity degree of the task.", "paper_topic_and_main_contributions": "This paper introduces an extensive training and test set where the video can disambiguate translations. Then a model named SAFA is designed for the disambiguation with the aid of a video.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I don't think linguistic (L4) or language ambiguity (L72) is a suitable term to describe the kind of \"ambiguity\" in the paper. It tends to be confusing because linguistic ambiguity (such as structure ambiguity or polysemous ambiguity) has nothing to do with multimodal ambiguity."}}, {"review_id": "2BFCXQYDaJ", "reviewer": "Reviewer_Dww6", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and efforts. Please find the responses (**R**) to the concern points (**P**).\n\n**P1**: More illustrative Figure 1 and Figure 4\n\n**R1**: We have made improvements to Figures 1 and 4 to enhance their illustrative quality. Due to the rebuttal format, we regrettably cannot provide the revised figures here. Instead, we explain what we did to make the figures more illustrative. \n\nFor Figure 1, we added the following content as an explanation: Each pair of parallel subtitles belongs to a video clip. As a translation task, the video clip can help us translate the source subtitle. The first video clip shows two women escaping, suggesting a \"go\" translation, while the last video clip portrays a scene involving an army, suggesting a \"forward\" translation.\n\nFor Figure 4, we added the following content as an explanation: The frame attention loss uses Gaussian distribution to guide the model to pay more attention to the central frames, while the ambiguity augmentation makes the model put more weight on the data with possibly ambiguous source subtitles. Besides, we annotated \"query,” “key,” and “value” on the figure and used a rhombus instead of the rectangle for \"Possibly Ambiguous.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "[+] Dataset contribution\n[+] Writing is easy to follow\n[+] Concrete problem definition\nThis paper is prepared for publication", "reasons_to_reject": "[-] Provide more illustrative Figure 1 and Figure 4", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes multimodal machine translation dataset referred to as EVA (Extensive training set and Video-helpful evaluation set for Ambiguous subtitles translation) to mitigate the ambiguity in translation based on a video.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "53JhIiBSj1", "reviewer": "Reviewer_7em6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and efforts. Please find the responses (**R**) to the concern points (**P**).\n\n**P1**: What proportion of EVA exhibits polysemy vs omitted words?\n\n**R1**: As shown in Section 3.2, the proportions of omission and polysemy are approximately 30\\% and 20\\%, respectively. Sometimes, instances of ambiguity arise from a combination of various factors, thereby challenging precise classification. For example, \"放せ!\" can be translated as both \"Let me go!\" and \"Drop it!\". This ambiguity could arise due to the polysemy \"放せ!\" or it could stem from the omission of the object. We will add more explanation about the classification.\n\n**P2**: Can more examples of true polysemy be provided?\n\n**R2**: Yes. For Ja-En, \"取って\" may be translated into \"Go, fetch!\" or \"Take it.\"; \"すごい人ですね\" may be translated into \"There are many people\" or \"That's a great person\". For Zh-En, \"放!\" may be translated into \"Fire!\" or \"Down!\"; \"这里\" may be translated into \"This way\" or \"Over here\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well organized and easy to follow. \n\n2. The experiments are comprehensive and solid. \n\n    (a) I believe that the videos in the EVA dataset are somehow helpful, because according to Table 3&4, the improvement from text-only to SAFA on EVA is larger than that on the previous dataset VISA. \n\n    (b) The ablation study in Table 3 supports the effectiveness of the proposed methods, i.e., Frame attention loss and Ambiguity augmentation.", "reasons_to_reject": "The use of \"ambiguous\" and \"disambiguation\" is questionable, as the examples in Figure 5 seem to focus on omitted words (e.g., “Thank you, sir” vs “Thank you”, and “Close the door, man” vs “Close it”) rather than polysemy. The authors define ambiguity in Line 38, but more evidence is needed to justify this definition: \n1) What proportion of EVA exhibits polysemy vs omitted words? \n2) Can more examples of true polysemy be provided? \n\nWithout this justification, the framing around ambiguity and disambiguation seems inappropriate.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper argues that existing Video Machine Translation (VMT) datasets contain linguistic ambiguity, making visual information ineffective in disambiguation. To address this problem, the authors propose EVA, a large scale VMT dataset. EVA contains a video-helpful evaluation set in which subtitles are ambiguous, and videos are guaranteed helpful for disambiguation. \n\nFurthermore, the authors propose SAFA, a VMT model with Frame attention loss and Ambiguity augmentation. Experimental results show that SAFA outperforms existing VMT models on both EVA and the previous VISA datasets.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new dataset for training and evaluation of multimodal MT. The paper focuses on cases where the visual context can aid for a better translation, eg ambiguous cases or the cases where a word is dropped. The authors have addressed almost all critical points raised by the reviewers, although a better characterization of the cases where the visual modality provides help to improve the translation would be very helpful for the paper (beyond ambiguity). Furthermore, the paper proposes a multimodal MT system to leverage the visual modality and shows that it outperforms competitive baselines. Therefore, this paper seems a a good contribution to the conference and community.
Decision: Accept-Main