Introduction
The recently introduced How2 dataset BIBREF2 has stimulated research around multimodal language understanding through the availability of 300h instructional videos, English subtitles and their Portuguese translations. For example, BIBREF3 successfully demonstrates that semantically rich action-based visual features are helpful in the context of machine translation (MT), especially in the presence of input noise that manifests itself as missing source words. Therefore, we hypothesize that a speech-to-text translation (STT) system may also benefit from the visual context, especially in the traditional cascaded framework BIBREF4, BIBREF5 where noisy automatic transcripts are obtained from an automatic speech recognition system (ASR) and further translated into the target language using a machine translation (MT) component. The dataset enables the design of such multimodal STT systems, since we have access to a bilingual corpora as well as the corresponding audio-visual stream. Hence, in this paper, we propose a cascaded multimodal STT with two components: (i) an English ASR system trained on the How2 dataset and (ii) a transformer-based BIBREF0 visually grounded MMT system.
MMT is a relatively new research topic which is interested in leveraging auxiliary modalities such as audio or vision in order to improve translation performance BIBREF6. MMT has proved effective in scenarios such as for disambiguation BIBREF7 or when the source sentences are corrupted BIBREF8. So far, MMT has mostly focused on integrating visual features into neural MT (NMT) systems using visual attention through convolutional feature maps BIBREF9, BIBREF10 or visual conditioning of encoder/decoder blocks through fully-connected features BIBREF11, BIBREF12, BIBREF13, BIBREF14.
Inspired by previous research in MMT, we explore several multimodal integration schemes using action-level video features. Specifically, we experiment with visually conditioning the encoder output and adding visual attention to the decoder. We further extend the proposed schemes to the deliberation variant BIBREF1 of the canonical transformer in two ways: additive and cascade multimodal deliberation, which are distinct in their textual attention regimes. Overall, the results show that multimodality in general leads to performance degradation for the canonical transformer and the additive deliberation variant, but can result in substantial improvements for the cascade deliberation. Our incongruence analysis BIBREF15 reveals that the transformer and cascade deliberation are more sensitive to and therefore more reliant on visual features for translation, whereas the additive deliberation is much less impacted. We also observe that incongruence sensitivity and translation performance are not necessarily correlated.
Methods
In this section, we briefly describe the proposed multimodal speech translation system and its components.
Methods ::: Automatic Speech Recognition
The baseline ASR system that we use to obtain English transcripts is an attentive sequence-to-sequence architecture with a stacked encoder of 6 bidirectional LSTM layers BIBREF16. Each LSTM layer is followed by a tanh projection layer. The middle two LSTM layers apply a temporal subsampling BIBREF17 by skipping every other input, reducing the length of the sequence $\mathrm {X}$ from $T$ to $T/4$. All LSTM and projection layers have 320 hidden units. The forward-pass of the encoder produces the source encodings on top of which attention will be applied within the decoder. The hidden and cell states of all LSTM layers are initialized with 0. The decoder is a 2-layer stacked GRU BIBREF18, where the first GRU receives the previous hidden state of the second GRU in a transitional way. GRU layers, attention layer and embeddings have 320 hidden units. We share the input and output embeddings to reduce the number of parameters BIBREF19. At timestep $t\mathrm {=}0$, the hidden state of the first GRU is initialized with the average-pooled source encoding states.
Methods ::: Deliberation-based NMT
A human translator typically produces a translation draft first, and then refines it towards the final translation. The idea behind the deliberation networks BIBREF20 simulates this process by extending the conventional attentive encoder-decoder architecture BIBREF21 with a second pass refinement decoder. Specifically, the encoder first encodes a source sentence of length $N$ into a sequence of hidden states $\mathcal {H} = \lbrace h_1, h_2,\dots ,h_{N}\rbrace $ on top of which the first pass decoder applies the attention. The pre-softmax hidden states $\lbrace \hat{s}_1,\hat{s}_2,\dots ,\hat{s}_{M}\rbrace $ produced by the decoder leads to a first pass translation $\lbrace \hat{y}_1,\hat{y}_2,\dots , \hat{y}_{M}\rbrace $. The second pass decoder intervenes at this point and generates a second translation by attending separately to both $\mathcal {H}$ and the concatenated state vectors $\lbrace [\hat{s}_1;\hat{y}_1], [\hat{s}_2; \hat{y}_2],\dots ,[\hat{s}_{M}; \hat{y}_{M}]\rbrace $. Two context vectors are produced as a result, and they are joint inputs with $s_{t-1}$ (previous hidden state of ) and $y_{t-1}$ (previous output of ) to to yield $s_t$ and then $y_t$.
A transformer-based deliberation architecture is proposed by BIBREF1. It follows the same two-pass refinement process, with every second-pass decoder block attending to both the encoder output $\mathcal {H}$ and the first-pass pre-softmax hidden states $\mathcal {\hat{S}}$. However, it differs from BIBREF20 in that the actual first-pass translation $\hat{Y}$ is not used for the second-pass attention.
Methods ::: Multimodality ::: Visual Features
We experiment with three types of video features, namely average-pooled vector representations (), convolutional layer outputs (), and Ten-Hot action category embeddings (). The features are provided by the How2 dataset using the following approach: a video is segmented into smaller parts of 16 frames each, and the segments are fed to a 3D ResNeXt-101 CNN BIBREF22, trained to recognise 400 action classes BIBREF23. The 2048-D fully-connected features are then averaged across the segments to obtain a single feature vector for the overall video.
In order to obtain the features, 16 equi-distant frames are sampled from a video, and they are then used as input to an inflated 3D ResNet-50 CNN BIBREF24 fine-tuned on the Moments in Time action video dataset. The CNN hence takes in a video and classifies it into one of 339 categories. The features, taken at the CONV$_4$ layer of the network, has a $7 \times 7 \times 2048$ dimensionality.
Higher-level semantic information can be more helpful than convolutional features. We apply the same CNN to a video as we do for features, but this time the focus is on the softmax layer output: we process the embedding matrix to keep the 10 most probable category embeddings intact while zeroing out the remaining ones. We call this representation ten-hot action category embeddings ().
Methods ::: Multimodality ::: Integration Approaches
Encoder with Additive Visual Conditioning (-) In this approach, inspired by BIBREF7, we add a projection of the visual features to each output of the vanilla transformer encoder (-). This projection is strictly linear from the 2048-D features to the 1024-D space in which the self attention hidden states reside, and the projection matrix is learned jointly with the translation model.
Decoder with Visual Attention (-) In order to accommodate attention to visual features at the decoder side and inspired by BIBREF25, we insert one layer of visual cross attention at a decoder block immediately before the fully-connected layer. We name the transformer decoder with such an extra layer as –, where this layer is immediately after the textual attention to the encoder output. Specifically, we experiment with attention to , and features separately. The visual attention is distributed across the 49 video regions in , the 339 action category word embeddings in , or the 32 rows in where we reshape the 2048-D vector into a $32 \times 64$ matrix.
Methods ::: Multimodality ::: Multimodal Transformers
The vanilla text-only transformer (-) is used as a baseline, and we design two variants: with additive visual conditioning (-) and with attention to visual features (-). A -features a -and a vanilla transformer decoder (-), therefore utilising visual information only at the encoder side. In contrast, a -is configured with a -and a –, exploiting visual cues only at the decoder. Figure FIGREF7 summarises the two approaches.
Methods ::: Multimodality ::: Multimodal Deliberation
Our multimodal deliberation models differ from each other in two ways: whether to use additive () BIBREF7 or cascade () textual deliberation to integrate the textual attention to the original input and to the first pass, and whether to employ visual attention (-) or additive visual conditioning (-) to integrate the visual features into the textual MT model. Figures FIGREF9 and FIGREF10 show the configurations of our additive and cascade deliberation models, respectively, each also showing the connections necessary for -and -.
Additive () & Cascade () Textual Deliberation
In an additive-deliberation second-pass decoder (–) block, the first layer is still self-attention, whereas the second layer is the addition of two separate attention sub-layers. The first sub-layer attends to the encoder output in the same way -does, while the attention of the second sub-layer is distributed across the concatenated first pass outputs and hidden states. The input to both sub-layers is the output of the self-attention layer, and the outputs of the sub-layers are summed as the final output and then (with a residual connection) fed to the visual attention layer if the decoder is multimodal or to the fully connected layer otherwise.
For the cascade version, the only difference is that, instead of two sub-layers, we have two separate, successive layers with the same functionalities.
It is worth mentioning that we introduce the attention to the first pass only at the initial three decoder blocks out of the total six of the second pass decoder (-), following BIBREF7.
Additive Visual Conditioning (-) & Visual Attention (-)
-and -are simply applying -and -respectively to a deliberation model, therefore more details have been introduced in Section SECREF5.
For -, similar to in -, we add a projection of the visual features to the output of -, and use -as the first pass decoder and either additive or cascade deliberation as the -.
For -, in a similar vein as -, the encoder in this setting is simply -and the first pass decoder is just -, but this time -is responsible for attending to the first pass output as well as the visual features. For both additive and cascade deliberation, a visual attention layer is inserted immediately before the fully-connected layer, so that the penultimate layer of a decoder block now attends to visual information.
Experiments ::: Dataset
We stick to the default training/validation/test splits and the pre-extracted speech features for the How2 dataset, as provided by the organizers. As for the pre-processing, we lowercase the sentences and then tokenise them using Moses BIBREF26. We then apply subword segmentation BIBREF27 by learning separate English and Portuguese models with 20,000 merge operations each. The English corpus used when training the subword model consists of both the ground-truth video subtitles and the noisy transcripts produced by the underlying ASR system. We do not share vocabularies between the source and target domains. Finally for the post-processing step, we merge the subword tokens, apply recasing and detokenisation. The recasing model is a standard Moses baseline trained again on the parallel How2 corpus.
The baseline ASR system is trained on the How2 dataset as well. This system is then used to obtain noisy transcripts for the whole dataset, using beam-search with beam size of 10. The pre-processing pipeline for the ASR is different from the MT pipeline in the sense that the punctuations are removed and the subword segmentation is performed using SentencePiece BIBREF28 with a vocabulary size of 5,000. The test-set performance of this ASR is around 19% WER.
Experiments ::: Training
We train our transformer and deliberation models until convergence largely with transformer_big hyperparameters: 16 attention heads, 1024-D hidden states and a dropout of 0.1. During inference, we apply beam-search with beam size of 10. For deliberation, we first train the underlying transformer model until convergence, and use its weights to initialise the encoder and the first pass decoder. After freezing those weights, we train -until convergence. The reason for the partial freezing is that our preliminary experiments showed that it enabled better performance compared to updating the whole model. Following BIBREF20, we obtain 10-best samples from the first pass with beam-search for source augmentation during the training of -.
We train all the models on an Nvidia RTX 2080Ti with a batch size of 1024, a base learning rate of 0.02 with 8,000 warm-up steps for the Adam BIBREF29 optimiser, and a patience of 10 epochs for early stopping based on approx-BLEU () for the transformers and 3 epochs for the deliberation models. After the training finishes, we evaluate all the checkpoints on the validation set and compute the real BIBREF30 scores, based on which we select the best model for inference on the test set. The transformer and the deliberation models are based upon the library BIBREF31 (v1.3.0 RC1) as well as the vanilla transformer-based deliberation BIBREF20 and their multimodal variants BIBREF7.
Results & Analysis ::: Quantitative Results
We report tokenised results obtained using the multeval toolkit BIBREF32. We focus on single system performance and thus, do not perform any ensembling or checkpoint averaging.
The scores of the models are shown in Table TABREF17. Evident from the table is that the best models overall are -and –with a score of 39.8, and the other multimodal transformers have slightly worse performance, showing score drops around 0.1. Also, none of the multimodal transformer systems are significantly different from the baseline, which is a sign of the limited extent to which visual features affect the output.
For additive deliberation (-), the performance variation is considerably larger: -and take the lead with 37.6 , but the next best system (-) plunges to 37.2. The other two (-& -) also have noticeably worse results (36.0 and 37.0). Overall, however, -is still similar to the transformers in that the baseline generally yields higher-quality translations.
Cascade deliberation, on the other hand, is different in that its text-only baseline is outperformed by most of its multimodal counterparts. Multimodality enables boosts as large as around 1 point in the cases of -and -, both of which achieve about 37.4 and are significantly different from the baseline.
Another observation is that the deliberation models as a whole lead to worse performance than the canonical transformers, with deterioration ranging from 2.3 (across -variants) to 3.5 (across -systems), which defies the findings of BIBREF7. We leave this to future investigations.
Results & Analysis ::: Incongruence Analysis
To further probe the effect of multimodality, we follow the incongruent decoding approach BIBREF15, where our multimodal models are fed with mismatched visual features. The general assumption is that a model will have learned to exploit visual information to help with its translation, if it shows substantial performance degradation when given wrong visual features. The results are reported in Table TABREF19.
Overall, there are considerable parallels between the transformers and the cascade deliberation models in terms of the incongruence effect, such as universal performance deterioration (ranging from 0.1 to 0.6 ) and more noticeable score changes ($\downarrow $ 0.5 for –and $\downarrow $ 0.6 for —) in the -setting compared to the other scenarios. Additive deliberation, however, manifests a drastically different pattern, showing almost no incongruence effect for -, only a 0.2 decrease for -, and even a 0.1 boost for -and -.
Therefore, the determination can be made that and -models are considerably more sensitive to incorrect visual information than -, which means the former better utilise visual clues during translation.
Interestingly, the extent of performance degradation caused by incongruence is not necessarily correlated with the congruent scores. For example, –is on par with –in congruent decoding (differing by around 0.1 ), but the former suffers only a 0.1-loss with incongruence whereas the figure for the latter is 0.4, in addition to the fact that the latter becomes significantly different after incongruent decoding. This means that some multimodal models that are sensitive to incongruence likely complement visual attention with textual attention but without getting higher-quality translation as a result.
The differences between the multimodal behaviour of additive and cascade deliberation also warrant more investigation, since the two types of deliberation are identical in their utilisation of visual features and only vary in their handling of the textual attention to the outputs of the encoder and the first pass decoder.
Conclusions
We explored a series of transformers and deliberation based models to approach cascaded multimodal speech translation as our participation in the How2-based speech translation task of IWSLT 2019. We submitted the –system, which is a canonical transformer with visual attention over the convolutional features, as our primary system with the remaining ones marked as contrastive ones. The primary system obtained a of 39.63 on the public IWSLT19 test set, whereas -, the top contrastive system on the same set, achieved 39.85. Our main conclusions are as follows: (i) the visual modality causes varying levels of translation quality damage to the transformers and additive deliberation, but boosts cascade deliberation; (ii) the multimodal transformers and cascade deliberation show performance degradation due to incongruence, but additive deliberation is not as affected; (iii) there is no strict correlation between incongruence sensitivity and translation performance.
Acknowledgements
This work was supported by the MultiMT (H2020 ERC Starting Grant No. 678017) and MMVC (Newton Fund Institutional Links Grant, ID 352343575) projects.

Question: Was evaluation metrics and criteria were used to evaluate the output of the cascaded multimodal speech translation?
Answer: BLEU scores

Question: What dataset was used in this work?
Answer: How2