Soul-Mix: Enhancing Multimodal Machine Translation with Manifold Mixup (2024)

Abstract: Multimodal machine translation (MMT) aims to improve the performance of machine translation with the help of visual information, which has received widespread attention recently. It has been verified that visual information brings greater performance gains when the textual information is limited. However, most previous works ignore to take advantage of the complete textual inputs and the limited textual inputs at the same time, which limits the overall performance. To solve this issue, we propose a mixup method termed Soul-Mix to enhance MMT by using visual information more effectively. We mix the predicted translations of complete textual input and the limited textual inputs. Experimental results on the Multi30K dataset of three translation directions show that our Soul-Mix significantly outperforms existing approaches and achieves new state-of-the-art performance with fewer parameters than some previous models. Besides, the strength of Soul-Mix is more obvious on more challenging MSCOCO dataset which includes more out-of-domain instances with lots of ambiguous verbs.

heading: 1 Introduction
text: Multimodal machine translation (MMT) endeavors to enhance the translation systems via integrating different multimodal datasets, specifically from the visual inputs, into conventional text-only neural machine translation (NMT) systems (Caglayan et al., 2019; Yao and Wan, 2020; Yang et al., 2020; Fei et al., 2023; Tayir et al., 2024; Tayir and Li, 2024). When visual context is incorporated into the translation process, the accuracy of the translation is expected to improve significantly. This is because visual context can clarify words that have multiple meanings, which ensures that the intended meaning is conveyed more precisely and accurately.
* Corresponding author.
Most current MMT frameworks aim to address the semantic gap between images and texts by designing fusion frameworks. Caglayan et al. (2021) applies the pre-training technique and proposes a cross-lingual method to learn the visually-grounded representations for MMT. Li et al. (2022a) applies a selective attention framework to correlate images to words. Li et al. (2022b) enhances MMT by introducing visual hallucination during inference time, as opposed to other MMT approaches based on the sentence-image pairs. Huang et al. (2023) contrasts multi-modal input with adversarial samples, in this case, the model learns to identify the most informative sample which is combined with the consistent image and several visual objects extracted from it.
Despite the promising progress achieved by existing MMT methods, most of them do not take full advantage of both complete and limited textual inputs. While recent research has shown that limited textual context can lead to better translations based on visual inputs, most attempts to incorporate limited textual inputs ignore the potential benefits of complete textual inputs (Caglayan et al., 2019). Intuitively, predicted translations based on complete textual inputs tend to rely more on textual information, ensuring semantic accuracy, while predicted translations based on the limited textual inputs tend to incorporate more information from visual inputs, clarifying words with multiple meanings. However, the lack of complete textual inputs could limit the performance when utilizing limited textual inputs.
To tackle this problem, we propose a framework termed Soul-Mix based on manifold mixup (Zhang et al., 2018; Verma et al., 2019) to enhance MMT by leveraging visual information more effectively. Specifically speaking, we mix the predicted translations of the complete textual input and the limited textual inputs. The mixed translation and the translation of the complete textual input are both utilized to compute the cross-entropy loss. In addition, we utilize Jensen-Shannon Divergence (JSD) to regu-
11283
larize their predictions to further boost the performance. Experiment results on the Multi30K (Elliott et al., 2016) dataset of English-to-German, Englishto-French, and English-to-Czech translation directions demonstrate that our framework outperforms previous MMT methods, particularly on the more challenging MSCOCO dataset (Elliott et al., 2017), which includes more out-of-domain instances and ambiguous verbs. Further model analysis also verifies the advantage of our approach.
To sum up, our contributions are three-fold: • We propose a manifold mixup method to use
visual information more effectively. • Experiment results show that our framework
achieves the new state-of-the-art performance with only a small increase in parameters.
• Further model analysis demonstrates that our framework is more robust to noise from irrelevant visual input information.

heading: 2 Related Work
text: Related work primarily includes the studies on multimodal machine translation and manifold mixup.

heading: 2.1 Multimodal Machine Translation
text: Machine translation aims to convert text or speech from one language to another (Chen et al., 2022; Cheng et al., 2023b; Chen et al., 2024; Gui et al., 2024). As a language shared by people worldwide, the visual modality helps machines to have a more comprehensive perception of the real world. The superiority of the multimodal context in MMT has been illustrated in recent works. Therefore, there has been a growing interest in MMT tasks (Elliott et al., 2016), which attracts more and more attention. As pre-trained models have great potential in other tasks (Cao et al., 2022; Li et al., 2023a,b; Jin et al., 2023; Dong et al., 2023a; Cai et al., 2023; Zhu et al., 2023b; Dong et al., 2023b; Cai et al., 2024; Huang et al., 2024), pre-trained models are also widely leveraged in MMT, such as ResNet (He et al., 2016). To prevent the encoding of irrelevant information in images, Yao and Wan (2020) proposes multimodal self-attention in the Transformer, which allows the model to effectively capture the relevant information in these modalities. Nishihara et al. (2020) introduces the supervised visual attention mechanism that leverages manual alignments between words in an utterance and their corresponding regions in the given image. This approach could generate supervisions for visual attention and trains
the visual attention component of the encoder utilizing these supervisions, boosting the model’s ability to align textual and visual information accurately. Huang et al. (2020) focuses on the huge challenge of unsupervised MMT by incorporating pseudo visual pivoting and the visual inputs. Wu et al. (2021) proposes two innovative approaches for MMT. The first method, gated fusion, is designed for conventional MMT, which fuses visual and textual inputs. The second method, dense-retrieval augmentation, leverages the retrieval-based approach, incorporating dense retrieval mechanisms to boost translation accuracy via retrieving the relevant visual and textual data. Li et al. (2022a) develops a new selective attention mechanism that correlates image patches with words, which has been widely utilized in recent research, as it allows for more precise alignment between visual and textual elements, thereby improving the overall performance of multi-modal translation systems. Ye and Guo (2022) leverages a mixup strategy to extract useful visual features, enhancing the machine translation process. This technique improves the model’s ability to use the visual information effectively, leading to better translation outcomes. Ye et al. (2022) designs the novel robust multi-modal interactive fusion method, including the cross-modal relation-aware mask mechanism. It enhances the model’s robustness to noise and improves the interaction between different modalities, leading to more accurate and more reliable translations. Cheng et al. (2023d) leverages asymmetric contrastive learning at these two levels to leverage the knowledge from image captioning and object detection. In this paper, we propose to utilize manifold mixup (Zhang et al., 2018; Verma et al., 2019) to transfer knowledge from visual information.

heading: 2.2 Mixup
text: How to improve the model’s robustness is a topic worth discussing (Xin et al., 2022; Cheng et al., 2023a; Yin et al., 2023; Cheng et al., 2023c; Xin et al., 2023b; Xin and Zou, 2023; Xin et al., 2023a; Zhu et al., 2024; Zhuang et al., 2024). Zhang et al. (2018) first proposes mixup to boost the model’s robustness. Beyond this, Verma et al. (2019) builds manifold mixup via applying this technique to the hidden representations, extending the surface-level mixup method. Some recent works have introduced mixup on machine translation (Zhang et al., 2019; Li et al., 2021), sentence classification (Chen et al., 2020; Jindal et al., 2020; Sun et al., 2020), multilingual understanding (Yang et al., 2022), speech
SRC: a man rows his boat through the water with yellow oars
Color a man rows his boat through the water with [MASK_C] oars
Character a [MASK_P] rows his boat through the water with yellow oars
Noun a man rows his boat through the [MASK_N] with yellow oars
recognition (Medennikov et al., 2018; Meng et al., 2021), and sentiment analysis (Zhu et al., 2023a). Note that Ye and Guo (2022) also applies mixup in MMT. However, we focus on mixing the representations of the complete textual inputs and limited textual inputs to enhance MMT.

heading: 3 Method
text: In this section, we first introduce the problem defination (§3.1) and the degradation schemes (§3.2). Then, we introduce our proposed Soul-Mix, including the source sentence encoder (§3.3), the aggregation module (§3.4), the target sentence decoder (§3.5), and the manifold mixup module (§3.6). At last, we introduce the final training objective (§3.7). The overview of our approach is shown in Figure 2.

heading: 3.1 Problem Defination
text: The corpus of MMT usually contains triples, which could be denoted as D = {(x, y, z)}, where z denotes the image and x, y denote the corresponding description of the image in the source language and target language. Given {(x, z)}, the cross-entropy loss is formulated as follows:
LCE(x, y, z) = − |y|∑
i=1
log p(yi|y<i, x, z) (1)

heading: 3.2 Degradation Schemes
text: As shown in Figure 1, when dealing with textual inputs, we adopt the degradation schemes (Caglayan et al., 2019; Li et al., 2022a), which involves using three degradation schemes: color masking, character masking, and noun masking.
Color masking refers to the process of substituting all the words that denote a specific color with a special token [Mask_C], which aims to boost the ability to capture color information from vision.
Character masking means that we replace character words by a special token [Mask_P]. Following Li et al. (2022a), we regard “man", “woman", “people", “men", “girl", and “boy" as the character words, which aim to enhance the ability to capture gender information from vision. This is because, in some languages, the masculine and feminine forms of the same noun are different. For example, “Eine Baseballspielerin” in German means a female baseball player, while “Ein Baseballspieler” in German means a male baseball player.
Noun masking means that all nouns contained in the utterance might be replaced. However, since noun masking is a more complex scenario than the character masking, we only mask one of the nouns using a special token [Mask_N].
It is worth noting that while the majority of sentences in the Multi30K dataset contain a color word, a character word, and a noun, there are still some utterances that do not include any of these words. In such cases, we directly use the original sentence as the output after applying the degradation process.

heading: 3.3 Source Sentence Encoder
text: As illustrated in the left part of Figure 2, our source sentence encoder follows the same structure as the transformer encoder (Vaswani et al., 2017). Given the source sentence x = {x1, · · · , xn}, the output of the source sentence encoder is as follows:
Elx = FFN(Multihead(E l−1 x ,E l−1 x ,E l−1 x )) (2)
where E0x denotes the embedding with the position embedding, n denotes the length of x, and l denotes the numbers of Encoder layers.

heading: 3.4 Aggregation Module
text: As illustrated in the central part of Figure 2, following Zhang et al. (2020); Yin et al. (2020); Cheng et al. (2023d) we introduce an aggregation module
to fuse these two modalities. For the input image z, a pre-trained ResNet-50 CNN (He et al., 2016) is utilized to obtain the average-pooled visual representation, which is projected to the same dimension as Ex. The projected visual representation Ez is:
Ez = Wz ResNetpool(z). (3)
where Wz is the parameter matrix. A gating matrix Λ is introduced to control the fusion of Ex and Ez:
Λ = sigmoid(W1ΛEx +W 2 ΛEz) (4)
where W1Λ and W 2 Λ are two parameter matrices.
After fusion, we obtain the output E as follows:
E = Ex + ΛEz (5)
E is fed into the target sentence decoder.

heading: 3.5 Target Sentence Decoder
text: As illustrated in the right part of Figure 2, our target sentence decoder follows the same structure as the one proposed in Vaswani et al. (2017). For the output of the aggregation module E and the embedding of the input target sentence D0, the output of each layer Dl in the target sentence decoder is:
Hl1 = Multihead(D l−1,Dl−1,Dl−1) (6) Hl2 = Multihead(H l 1,E,E) (7) Dl = FFN(Hl2) (8)
where Hl1 is the output of the l-th self-attention of the decoder, Hl2 is the output of the cross attention layer in the l-th layer, and Dl is the hidden states in the l-th layer of the decoder. Then a softmax layer is applied to generate the probability distribution, which takes the hidden states D in the top layer of the target sentence decoder as the input.

heading: 3.6 Manifold Mixup Module
text: Previous studies have demonstrated that when the textual input is limited, MMT models could leverage visual information more effectively (Caglayan et al., 2019). In this work, we extend this method by utilizing a manifold mixup technique to simultaneously leverage the predicted translations of the limited textual inputs and the predicted translation of the complete textual input. Specifically, we apply several degradation schemes to textual inputs as Section 3.2. For the complete input sentence x0, we utilize color masking, character masking, and noun masking, to obtain the processed sentences x1, x2, and x3, respectively.
We utilize {(xi, z0)}3i=0 as the input of the encoder and the pre-trained ResNet-50 CNN, respectively, and apply manifold mixup technique to mix the predicted translations of each pair of inputs. The mixed predicted translation ymix is as follows:
ymix = 3∑
i=1
λF(xi, z0) + (1− 3λ)F(x0, z0) (9)
where F(·) denotes the corresponding output of the target sentence decoder and λ is a hyper-parameter of the manifold mixup technique.

heading: 3.7 Training Objective
text: The training objective of our framework includes two cross-entropy losses, where the first is calculated between the predicted translation ycom of complete textual input and the ground truth y, and the second is calculated between the mixed predicted translation ymix and the ground truth y, which can
be formulated as follows:
L1CE = − |y|∑
i=1
log p(yi|y<i, x0, z0) (10)
L2CE = − |y|∑
i=1
log p(yi|y<i, x0, x1, x2, x3, z0)
(11)
Besides the cross-entropy losses, we also regularize the above two output predictions ycom, ymix by minimizing JSD between the two output distributions, which is formulated as follows:
LJSD = |y|∑
i=1
JSD{p(yi|y<i, x0, z0)∥
p(yi|y<i, x0, x1, x2, x3, z0)} (12)
The final training objective is as follows:
L = L1CE + L2CE + γLJSD (13)
where γ is the coefficient weight to control LJSD.

heading: 4 Experiments

heading: 4.1 Datasets and Metrics
text: All experiments are conducted on the Multi30K1 (Elliott et al., 2016) dataset, which is an extension of Flickr30k (Young et al., 2014) widely utilized for MMT. Each image in Multi30K is accompanied by a sentence in English (En) and the manual translations in German (De), French (Fr), and Czech (Cs). The training set contains 29,000 text-image pairs and the validation set contains 1014 pairs. Four test
1https://github.com/multi30k/dataset
sets are used to evaluate the MMT models, including Test2016 (Elliott et al., 2016), Test2017 (Elliott et al., 2017), MSCOCO (Elliott et al., 2017), and Test2018 (Barrault et al., 2018). The details of the dataset can be found in Appendix. A.
To evaluate the performance of translation, we report 4-gram BLEU (Papineni et al., 2002) and METEOR (Denkowski and Lavie, 2014). 4-gram BLEU measures the quality of translations in accuracy and fluency and METEOR takes into account both precision and recall. Higher BLEU and METEOR mean higher performance.

heading: 4.2 Implementation Details
text: Following the previous works, the byte pair encoding (BPE) algorithm (Sennrich et al., 2016) with 6,000 merging operations is used to segment words into subwords, which could generate a vocabulary of 5876 tokens for En-De, 5684 tokens for En-Fr, and 5972 tokens for En-Cs. A pre-trained ResNet50 CNN (He et al., 2016) is utilized to extract the image features. Both the encoder and decoder contain 4 layers. Sign test (Collins et al., 2005) is used as the standard statistical-significance test.
During the training process, the value of label smoothing is set to 0.1, and dropout is set to 0.1. The weight λ is set to 0.2 and the weight γ is set to 2. During decoding, the beam size is set to 5. For all the experiments, we select the model that works the best on the dev set and then evaluate it on the test set. To avoid overfitting, the training will early-stop if the loss on dev set does not decrease for 3 epochs as in (Zhang et al., 2020). We apply Adam optimizer (Kingma and Ba, 2015) with β1 = 0.9, β2 = 0.98, and 4k warm-up updates to optimize all the parameters, where we linearly
increase the learning rate from 10e-7 to 5e-5. During the inference process, we follow the previous work (Wu et al., 2019) to average the checkpoints of the last 10 epochs for evaluation. Multibleu.perl8 script2 is used to compute case-sensitive 4-gram BLEU scores for all test sets. The entire training process takes several hours. All the experiments are conducted on a single Nvidia V100 GPU with fp16 and based on fairseq3 (Ott et al., 2019).

heading: 4.3 Baselines
text: We first compare our method with a text-only baseline, which trains a transformer model without any visual information. In addition, we report the performance of several MMT models, including Imagination, Multimodal Self-attn, WRA-guided, Selective Attn, DLMulMix, PLUVR, VALHALLA(M), CAT-MMT bi, E2H-MNMT, and RG-MMT-EDC. For a fair comparison, we utilize the same configuration for all the baselines as our framework.

heading: 4.4 Results
text: Table 1 demonstrates the performance comparison between Soul-Mix and the baselines in English-toGerman, and English-to-French translation directions. We also report the result in English-to-Czech translation direction in Table 2. Based on these results, we have the following observations:
(1) Soul-Mix achieves consistent improvements on all the test sets, including the Test2016 dataset, the Test2017 dataset, the Test2018 dataset, and the
2https://github.com/moses-smt/mosesde coder/blob/master/scripts/generic/mult i-bleu.perl
3https://github.com/facebookresearch/ fairseq
MSCOCO dataset. For English-to-German translation direction, our approach achieves 44.24, 37.14, and 34.26 BLEU on these test sets, outperforming the previous results. Similar situations can also be found in the results of METEOR and other translation directions, which demonstrate the effectiveness of our framework. We believe that it is because our method applies the manifold mixup technique to use the visual information more effectively. In addition, JSD plays a role in regularization, which can further improve the robustness of the model.
(2) The number of parameters in our proposed approach is slightly more than that of Transformer. More encouragingly, it is significantly lower than that of some other MMT frameworks, such as Imagination (Elliott and Kádár, 2017), which can further verify the superiority of our framework.

heading: 5 Analysis

heading: 5.1 Effect of Different Degradation Schemes
text: In our method, three different degradation schemes are introduced to generate these incomplete textual inputs, including x1, x2, and x3, respectively. To verify the effectivenss of the degradation schemes, we sequentially remove these degradation schemes and accordingly modify the Eq. 13, denoting them as "w/o COM", "w/o CHM", and "w/o NM", respectively. These experimental results are demonstrated in Table 3. We could observe varying degrees of decline, indicating that all three degradation schemes
play a positive role. Besides, we also attempt to replace these three degradation schemes with random masking, denoted as “+ RM". It can be observed that the proposed masking strategies are more effective. We believe that the reason is that via utilizing these mask strategies, the masked words could have a stronger correlation with the image, thereby further enhancing the ability to use visual information.

heading: 5.2 Ablation Study
text: To verify the advantages of Soul-Mix from different perspectives, we conduct ablation experiments in English-to-German translation direction. Experiment results on Test2016 dataset, Test2017 dataset, and MSCOCO dataset are shown in Table 4.
(1) Effectiveness of Manifold Mixup. One of the core contributions of Soul-Mix is to apply the Manifold Mixup technique to mix up the predicted translations of complete textual input and limited textual inputs. To verify the effectiveness of manifold mixup, we remove L2CE and LJSD in Eq.13 and refer it to w/o mixup in Table 4. We could observe the performance degradation. These results demonstrate that manifold mixup can indeed improve the performance of the framework through leveraging the visual information more effectively.
(2) Effectiveness of JSD. To validate the effectiveness of JSD, we remove LJSD in Eq.13 and refer it to w/o JSD in Table 4. We could find that BLEU drops by 0.37, 0.12, and 0.48 on these three datasets, respectively. In addition, METEOR also
drops by 0.22, 0.33, and 0.49, respectively. These results show that JSD could indeed further enhance the performance of MMT. We believe that it is because JSD could act as a regularization technique to prevent overfitting during the training process.

heading: 5.3 Incongruent Decoding
text: Incongruent decoding involves replacing the original image with an incongruent one during the test process. The MMT framework that better leverages visual information typically experiences more performance degradation with incongruent decoding.
As shown in Figure 3, we employ incongruent decoding on the English-to-German translation direction of the Test2016 test set to test whether our method can effectively integrate the visual modality. We could clearly observe that Soul-Mix shows greater performance degradation compared to other frameworks, further demonstrating its superior ability to effectively leverage visual information.

heading: 5.4 Case Study
text: Finally, we perform a case study to further demonstrate the advantages of our framework. We select a
text and its corresponding image from the Test2016 dataset for the English-to-German translation direction. We choose two MMT frameworks for comparison, including VALHALLA(M) (Li et al., 2022b) and E2E-MNMT (Ye et al., 2023).
As illustrated in Table 5, we could obviously observe that VALHALLA(M) fails to incorrectly predict “Frisbee” as “plate”, while both E2E-MNMT and Soul-Mix correctly predict it. We believe the reason is that E2E-MNMT and Soul-Mix successfully leverage visual information to generate more accurate translations, especially for the words with multiple meanings. Besides, we can also find that VALHALLA(M) and E2E-MNMT incorrectly predict “brown” as “grey”, while Soul-Mix correctly predicts it. A possible reason is that the two frameworks are misled by the grey tail of the dog in the picture, and they mistakenly regard the color of the dog as “brown”. These results further validate that Soul-Mix could avoid being misled by visual information more effectively than previous frameworks.

heading: 6 Conclusion
text: In this paper, we propose Soul-Mix to leverage visual information more effectively via the manifold mixup technique for MMT. Experiment results on the Multi30K dataset of three translation directions demonstrate the effectiveness and the superiority of our proposed framework, which can surpass all the previous works with a relatively small increase in the number of parameters. Further analysis shows that our framework is more robust to the noise from irrelevant visual information. In the future, we will explore how to apply the manifold mixup technique to visual inputs for MMT more effectively.
Limitations
While our Soul-Mix model has achieved the encouraging performance, it is important to recognize that our current method is predicated on the utilization
of image data during inference. This indicates that the efficacy of our method is dependent on the presence of visual data. Moving forward, our research agenda will focus on overcoming this constraint by investigating new techniques to boost our model’s performance when only the textual data is accessible during the inference phase.
Ethics Statement
All the data utilized in this study is gathered from the public resources. While our method has demonstrated the promising performance, it is important to note that the results they produce are not guaranteed to be flawless. Therefore, users should exercise caution and not solely depend on these results.

heading: Acknowledgements
text: This paper was partially supported by NSFC (No: 62176008). We thank all the anonymous reviewers for their insightful comments.

heading: A Dataset
text: We employ four test sets to evaluate MMT models:
• The Test2016 (Elliott et al., 2016) test set with 1,000 text-image pairs in the initial Multi30K dataset. The languages in the Test2016 test set include English, German, French, and Czech.
• The Test2017 (Elliott et al., 2017) test set with 1,000 text-image pairs from WMT2017, with more difficult source sentences. Languages in the Test2017 test set include English, German, and French, respectively.
• The MSCOCO (Elliott et al., 2017) test set with 461 text-image pairs. Note that it is more challenging because there are more ambiguous verbs and instances. The languages in this dataset include English, German, and French.
• The Test2018 (Barrault et al., 2018) test set with 1,071 text-image pairs, including more entity words and low-frequency words. Languages in Test2018 include English, German, French, and Czech, respectively.

heading: Future Work
text: In the future, we will explore how to apply the manifold mixup technique to visual inputs for MMT more effectively. Limitations
While our Soul-Mix model has achieved the encouraging performance, it is important to recognize that our current method is predicated on the utilization
of image data during inference. This indicates that the efficacy of our method is dependent on the presence of visual data. Moving forward, our research agenda will focus on overcoming this constraint by investigating new techniques to boost our model’s performance when only the textual data is accessible during the inference phase