Do Videos Guide Translations?
Evaluation of a Video-Guided Machine Translation dataset
Tosho Hirasawa
Zhishen Yang
Tokyo Metropolitan
Tokyo Institute
Universityof Technology
hirasawa-tosho
zhishen.yang@nlp.c.titech.ac.jp @ed.tmu.ac.jp
Abstract
Video-guided machine translation (VMT) isa new multimodal machine translation task
aimed at using videos to guide translation. Visual information gleaned from videos is expected to provide context in the translation
progress. Results from the Video-guided Machine Translation Challenge 2020 suggest thatmultimodal models only have marginal performance improvements over their text-only
counterparts. We hypothesize that this iscaused by the simple and short video descriptions in VATEX, the dataset used in the challenge. In this study, we examine our hypothesis by conducting input-degradation, visual
sensitivity experiments, and human evaluation
of VATEX. The results indicate that textual descriptions of videos in VATEX are sufficientfor translation, which prevents the visual context from videos to guide the translation.

1

Introduction

Extending text-only machine translation, the multimodal machine translation task exploits information from other modalities to improve the translation quality. Video-guided machine translation(VMT) is a new multimodal machine translation
task that provides videos, as additional inputs, fora model to translate sentences from the source totarget languages. Compared to image-guided machine translation, videos provide visual and acoustic modalities with rich embedded information,such as actions, objects, and temporal transitions.
VMT refers to videos as additional information
to improve the translation quality. Therefore, anideal dataset for this task should provide videos
as complements rather than redundancies. The recently proposed VATEX dataset is a dataset for
VMT research and the shared task (VMT Challenge). From the results of the 2020 VMT challenges, all multimodal models only had marginal
performance gains compared to their text-only
counterparts. We hypothesize that this was caused

Mamoru Komachi
Tokyo Metropolitan
University
komachi@tmu.ac.jp

Naoaki Okazaki
Tokyo Instituteof Technology
okazaki@c.titech.ac.jp

by the design of the VATEX dataset: simple andshort video descriptions are sufficient for translations, making videos redundant information forthe models. To examine our hypothesis, similar
to (Caglayan et al., 2019), we conducted input
degradation experiments, visual sensitivity experiments, and human evaluations. The experimental
and human evaluation results showed that when
textual information is sufficient, visual informationfrom videos becomes redundant to a multimodal
model. The code used in this study is publicly
available.1

2

Related Work

VMT Challenge 2020 The top three teams inthe VMT Challenge 2020 presented recurrent neural network (RNN) and Transformer-based models. The winning team, Hirasawa et al. (2020),
used a doubly attentive RNN-based model (Calixto et al., 2017) with positional encoding for video
features. Two other teams proposed Transformerbased VMT models with modifications to incorporate video features.
Probing Auxiliary Modalities Probing the needfor auxiliary modalities is an important topic in
multimodal machine translation (MMT). In imageguided machine translation, Caglayan et al. (2019)conducted analytic experiments whereby source
sentences were degraded in three different ways
to simulate specific conditions in which images
should be beneficial. Hessel and Lee (2020) proposed a method to isolate cross-modal interactions
for multimodal classification tasks and showed thatcross-modal interactions have no or little contribution to the model performance. In this study, we
extend Caglayan et al. (2019) to probe the need forvideos as a visual modality in VMT.

1
https://github.com/ZhishenYang/eval_
on_vatex_dataset

3

Input Degradation

Inspired by Caglayan et al. (2019), we conductedfour source-side input-degradation experiments:
color, noun, verb, and progressive masking. Since
these input-degradation experiments are used tosimulate the scarce textual context condition, we
hypothesize that a multimodal model can rely on
the visual context obtained from videos and will
perform better than a monomodal model that only
relies on the textual context.
Color Deprivation We replaced English words
that represent color in the source sentences witha special token [c]. The masked tokens consist of
0.4% in both training set and validation set.
Noun Masking We replaced each noun in thesource English sentences with a special token, [n].
This masked 28.2% of the tokens in both the training and validation sets.
Verb Masking The authors of the VATEX
dataset used videos from the Kinetics-600 dataset,
which contains a broad range of actions. All verbs
in the source sentence were replaced with a special
token, [v]. This replaced 14.0% of the tokens inboth the training and validation sets.
Progressive Masking Progressive masking aims
to progressively replace the last N words in asource sentence with a special token, [p]. Unlike other masking experiments, progressive masking simulates a progressive low-resource scenario (Caglayan et al., 2019). We hypothesize thatwith the increasing number of masked tokens inthe source sentences, multimodal models with access to visual information will perform better thantext-only models.
We selected N = {2, 4, 6, 10, 20, 30} in the progressive masking experiment. When N = 30,
nearly 100% of all words were masked, and themultimodal model performed video captioning
with "expected length" as the only known information.

4

Visual Sensitivity

Visual Incongruence Test Inspired by (Elliott,2018) and (Caglayan et al., 2019), we implemented
visual incongruence tests to examine the visual sensitivity of multimodal models. In this test, we fed
multimodal models with incongruent visual features during the testing time. The hypothesis is thatthe performance of multimodal models will deteriorate when fed with visual features from irrelevant
videos.
Visual Features To test whether multimodal
models are sensitive to different visual features, we
extracted visual features using pre-trained models
for action or object classification. We conjecturethat visual features correlated to subjects/objects
and actions in videos will help predict nouns andverbs in video descriptions.

5

Experiments

5.1

DatasetWe used VATEX v1.1 (the latest version) 2 . The
translation direction was the same as that of the
VMT challenge 2020: from English to Chinese.
Because the public test set is on-hold, we used avalidation set to test the performance of the models. The statistics of the dataset are provided inAppendix A.
For the tokenization, we used spaCy 3 to tokenize the English sentences, and then employed
byte pair encoding (Sennrich et al., 2016) to split
the English tokens into subwords, where the number of merge operations was 8000. The Chinesetranslations were tokenized at the character level.
5.2

Visual Feature Extraction

As described in Section 4, two types of features
were correlated to the nouns and verbs that we usedin the experiments: ResNet-152 and I3D features.
We extracted the ResNet-152 features from the persecond frames of each video. The ResNet-152 feature is the averaged convolutional features from thelast convolutional layer of ResNet-152 (He et al.,2016) pretrained on ImageNet (Deng et al., 2009).
The I3D features were extracted from videos using
two-stream inflated 3D ConvNet (I3D) (Carreiraand Zisserman, 2017) and the I3D features provided in the VMT challenge 2020 2 .
5.3

ModelsFor the text-only baseline models, we used an attentive RNN model (Bahdanau et al., 2015) andtransformer model (Vaswani et al., 2017).
For the multimodal models, we employed twomodels from the VMT challenge 2020: hierarchical attentive RNN model with positional encod2
3

https://eric-xw.github.io/vatex-website/download.html
https://spacy.io/

Figure 1: Corpus-level BLEU scores of the validation set. The data labels show the BLEU scores of the text-onlybaselines.

ing (Hirasawa et al., 2020) and VGT-Shallow4 . See
Appendix D for more details.

6

Results

In this section, we discuss the results from the inputdegradation and visual sensitivity experiments in
Figure 1 and 2 5 . Without any input degradation, all transformer-based models had higher
BLEU scores than those of the RNN-based models. Among the multimodal models, VGT-Shallow(I3D) achieved the best BLEU scores.
Color Deprivation Since only a small fractionof tokens was masked, compared with models
trained on complete data, the differences between
the multimodal and monomodal models were
marginal. The text-only Transformer and attentive RNN model had slightly higher BLEU scoresthan those of the multimodal models.
Noun Masking All models had lower BLEU
scores compared to those of their complete databaselines. All multimodal models, except for the
VGT-shallow (I3D), had higher BLEU scores compared to those of the text-only models. Nounmasking has a larger masking scale; therefore, compared to the verb masking and color deprivation
experiments, the multimodal models can exploit
the visual context to infer missing information.
Verb Masking In verb masking, although allmodels had deteriorated performances, the differences between the multimodal and monomodal
models were minimal.
4

https://www.youtube.com/watch?v=
zHwXPmIQajA&t=517s
5
See Appendix B for more details.

Visual Sensitivity Test Even when feeding withincongruent visual features during testing, the multimodal models suffered slight performance deterioration as compared with training and testing on acomplete dataset. This indicates that visual features
have a minimal influence on multimodal models
when given complete text.
In our experiments, the multimodal models withdifferent architectures had different sensitivities tovisual features. The hierarchical attentive RNN
using ResNet-152 had higher BLEU scores thanthose obtained when using I3D features. However,for VGT-Shallow, this only occurred for the nounmasking experiment.
Progressive Masking Figure 2 shows the resultof progressive masking. For an increasing numberof masked tokens, the multimodal models started totake advantage of visual modalities, and therefore
outperformed the text-only models. Moreover, theHierarchical Attentive RNN had the best BLEU
score when nearly 100% of tokens were masked.

7

Human Evaluation

The authors of VATEX used a post-editing annotation strategy to collect parallel English-Chinesetranslation pairs, in which automatic translation
systems were employed. Based on the results of
the input-degradation experiments, we hypothesizethat if English-Chinese translations are sufficient,videos become redundant in post-editing. To test
our hypothesis, we conducted a human evaluation
task based on a post-editing annotation strategy.
We randomly selected 500 videos from VATEX’s
validation set to construct the human evaluation set,

EN:
VX:
ZH:
PE:

A person is showing three little kids making faces and being historically.
一个人正在展示三个小孩做鬼脸和历史。
(A person is showing three kids making funny faces and history.)
一个人正在展示历史上的三个做鬼脸的孩子。
(A person is showing three grimacing children in history.)
三个孩子在电脑前，一个男孩的椅子翻了。
(Three children were in front of the computer, and one boy’s chairturned over.)

Table 1: Example of human evaluation. “VX” shows the translation from the VATEX dataset; “ZH” is the humantranslation; and “PE” is the post-edited Chinese translation. The parenthesized sentences are obtained using Google
Translation on Chinese sentences.

Figure 2: Progressive masking: the multimodal models outperform the monomodal models with increasing
percentages of masked tokens.

and selected two English video descriptions fromeach video’s parallel translation pairs in random
order. Therefore, a total of 1000 instances exist
in the human evaluation set; each instance has an
English video description and a video URL.
Given an instance from the human evaluation
set, a human translator is first asked to perform an
English-Chinese translation, then watch the video,and post-edit the translation only if it does not properly describe the video. We also asked the translator
to provide reasons for the post-editing, as well as
any remarks. Four professional translators wererecruited, two for the translation and two for postediting, to guarantee the translation quality.
The average editing distance between original
Chinese translations from VATEX and human translations is 13.3, which indicates that the annotation
strategy of VATEX cannot generate as high-qualitytranslation as a human translator. The translator
post-edited 104 Chinese translations, 10.4% of thetotal number of instances, with an average editing distance of 5.1, compared with human translations, and 14.5 compared with original translations
from the VATEX dataset. We found that 98% of
post-edits are categorized as “Source English de-scription is inaccurate” which means the translators
actually used information gleaned from videos tocorrect the wrong description in English source
sentences.
Based on the above results, we found that in mostcases, source English sentences provide sufficient
information for the human translator to perform
translations; videos help to correct incorrect descriptions in the source sentences rather than todisambiguate translations. These findings also indicate that the short and simple sentences in the VATEX dataset are sufficient for translation purposes,
and their videos only provide rather redundant information, which is consistent with the automatic
metrics evaluations.
Table 1 shows an example of post-editing correction. The edit distance between the Chinese video
description in VATEX and our human post-editingis 15. In the source English descriptions, “being
historically” is an ambiguous and incorrect phrase,
and we also cannot align it with any parts in thevideo. Therefore, the translator changed most partsof the sentence to align with the video content.

8Conclusion

In this study, we probe textual modality dominanceand the contributions of visual modality in VMTtasks by analyzing a large-scale VMT dataset: VATEX. Results from input-degradation and visual
sensitivity experiments indicate that multimodal
models tend to ignore the visual modality when
textual modality has sufficient information to perform translation. We ascribe the experimental results to the simple and short video descriptions in
VATEX, providing sufficient information to accomplish translation, eventually preventing the visualcontext from videos to engage in the translation
process.

Acknowledgements
This work was supported by JSPS KAKENHI
Grant Number 19H01118. The research results
have been achieved by "Research and Developmentof Deep Learning Technology for Advanced Multilingual Speech Translation", the Commissioned
Research of National Institute of Information andCommunications Technology (NICT), Japan.

References
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly
learning to align and translate. In 3rd International
Conference on Learning Representations.
Ozan Caglayan, Pranava Madhyastha, Lucia Specia,
and Loïc Barrault. 2019. Probing the need for visualcontext in multimodal machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4159–4170,
Minneapolis, Minnesota. Association for Computational Linguistics.
Iacer Calixto, Qun Liu, and Nick Campbell. 2017.
Doubly-attentive decoder for multi-modal neuralmachine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 1913–
1924, Vancouver, Canada. Association for Computational Linguistics.
Joao Carreira and Andrew Zisserman. 2017. Quo vadis,
action recognition? a new model and the kinetics
dataset. In proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, pages
6299–6308.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li,
and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference
on Computer Vision and Pattern Recognition, pages
248–255. Ieee.
Desmond Elliott. 2018. Adversarial evaluation of multimodal machine translation. In Proceedings of the2018 Conference on Empirical Methods in Natural Language Processing, pages 2974–2978, Brussels, Belgium. Association for Computational Linguistics.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian
Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, pages
770–778.
Jack Hessel and Lillian Lee. 2020. Does my multimodal model learn cross-modal interactions? it’sharder to tell than you might think! In Proceedingsof the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP), pages 861–
877, Online. Association for Computational Linguistics.
Tosho Hirasawa, Zhishen Yang, Mamoru Komachi,
and Naoaki Okazaki. 2020. Keyframe segmentation and positional encoding for video-guided machine translation challenge 2020. arXiv preprint
arXiv:2006.12799.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016. Neural machine translation of rare words
with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 1715–
1725, Berlin, Germany. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz
Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural
Information Processing Systems 2017, pages 5998–
6008.

A

Dataset Statistics

Table 2 shows the statistics of dataset that we obtained. Due to some of the YouTube URLs areinvalid, we exclude these data fromB

Experiment Results

Table 3 shows the corpus-level BLEU scores ofeach model in Figure 1.

C

Hyper-parameters of baseline models

C.1

Attentive NMT

The attentive RNN model has a 2-layer bidirectional GRU encoder with 512 hidden dimensions, a2-layer GRU decoder with 512 hidden dimensions,and an embedding layer of 1024 dimensions. During training, we used the Adam optimizer with alearning rate of 0.0004, and batch size of 64 sentences.
C.2

Transformer

The Transformer model in our experiment is thesame as the Transformer base model in Vaswani
et al. (2017). We employed the same training strategy as Vaswani et al. (2017) to train our Transformer baseline.

D

Video-guided NMT

Video-guided Transformer (VGT) models were proposed at ALVR 2020. One of VGT models, VGTShallow, achieved the best performance as a single
model system. Although the author provided astrong baseline, the paper6 is not publicly available. However, the video about their models isavailable7 , and we implemented their model basedon the video.
The original work provides two variants of VGTmodels: VGT-Shallow and VGT-Deep. The vanilla
Transformer encodes an n-tokens source sentencex = (x1 , · · · , xn ), into the hidden state h =(h1 , · · · , hn ), before decoding the m-tokens target
sentence y = (yi , · · · , ym ) from h. In the VGTmodels, the model employs an auxiliary modality z = (z1 , · · · , zk ) of k-elements, which interact
with the language modality in their encoder. Notethat both VGT-Shallow and VGT-Deep have a standard Transformer decoder.
6“Depper Is Not Always Better: Strong Baselines forVideo-guided Neural Machine Translation”
7
https://www.youtube.com/watch?v=
zHwXPmIQajA&t=517s

Split

Language

Video

Sent.

Token

Train

EnglishChinese24K

121K

1,986K
2,891K

Valid

EnglishChinese

2.8K

15K

228K
331K

Table 2: The statistics of VATEX dataset that we usedin the experiments.

In the preliminary experiments, we found that
VGT-Shallow outperformed VGT-Deep, which was
also reported by the the original author. Therefore,
we only adopted VGT-Shallow in our experiments.
D.1

VGT-Shallow

The VGT-Shallow first encodes an input sentence
using a standard Transformer encoder to retrieve
the hidden state h. Subsequently, the model employs a single fusion layer that has one visual reconstruction module, one cross-modal multi-headattention module, and one element-wise weighted
sum module. Note that we exploit normalization
and residual connection between modules. Specifically, the visual reconstruction module uses multihead attention to reconstruct the auxiliary features.
0

hr = multiheadr (z, h, h)(1)where multiheadr is a multi-head attention module.
0
The obtained reconstructed feature hr is then fed
into the cross-modal attention module:000hx = multiheadx (h, hr , hr )(2)where multiheadx is a multi-head attention module.
Finally, the fusion layer computes the element0
wise sum over h using hx as the weight to obtain0
the final multimodal representation h :00h = hxh(3)0

The model decodes the target sentence using hinstead of h.

T

TIncongruence

TColor

TNoun

TVerb

Attentive RNN
Hierarchical Attentive RNN (I3D)
Hierarchical Attentive RNN (ResNet-152)

33.92
33.49*
33.68*

33.92
32.75*
33.37*

33.93
33.34*
33.71*

20.99
23.03*
23.18*

30.33
30.21
30.26

Transformer
VGT-Shallow (I3D)
VGT-Shallow (ResNet-152)

36.10
36.18
35.93*

36.10
36.17
35.48*

35.92
35.83
35.59*

23.01
22.80*
24.50*

32.01
31.84*
31.67*

ModelsTable 3: Corpus-level BLEU scores on validation set. * indicates that a model is significantly different from its
text-only counterpart with P-value <= 0.05. bold marks the model with the best BLEU score.