Virtual Visual-Guided Domain-Shadow Fusion via Modal Exchanging for Domain-Specific Multi-Modal Neural Machine Translation

Authors: zhenyu  hou
junjun  guo
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 13:32:55

Abstract: Incorporating domain-specific visual information into text poses one of the critical challenges for domain-specific multi-modal neural machine translation (DMNMT). While most existing DMNMT methods often borrow multi-modal fusion frameworks from multi-modal neural machine translation (MNMT) in the general domain, they overlook the domain gaps between general and specific domains. Visual-to-textual interaction in a specific domain frequently exhibits multi-focus characteristics, making it difficult to consistently focus on domain-specific multi-visual details using traditional multi-modal fusion frameworks. This challenge can lead to a decrease in machine translation performance for domain-specific terms. To tackle this problem, this paper presents a virtual visual scene-guided domain-shadow multi-modal fusion mechanism to simultaneously integrate multi-grained domain visual details and text with the guidance of modality-agnostic virtual visual scene, thereby enhancing machine translation performance for DMNMT, especially for domain terms. Specifically, we first adopt a modality-mixing selection-voting strategy to generate modality-mixed domain-shadow representations through layer-by-layer intra-modality selection and inter-modality exchanging. Then, we gradually aggregate modality-mixed domain representations and text across modality boundaries with the guidance of a modality-agnostic virtual visual scene to enhance the collaboration between domain characteristics and textual semantics. The experimental results on three benchmark datasets demonstrate that our proposed approach outperforms the state-of-the-art (SOTA) methods in all machine translation tasks. The in-depth analysis further highlights the robustness and generalizability of our approach across various scenarios. Our code is available on https://github.com/HZY2023/VVDF.

Main paper: 
 introduction: 
 Domain-specific Multimodal Neural Machine Translation (DMNMT) aims to translate sentences within specific domains from a source to a target language by incorporating images as additional modality inputs. Recently, DMNMT tasks have attracted increasing attention, particularly due to their remarkable application scenarios in domains like cross-border e-commerce shopping, cross-border financial transactions, and cross-border cultural exchange.Due to the similarity to multimodal neural machine translation (MNMT) [10, 42, 44] , previous DMNMT works typically follow MNMT frameworks in general domain and concentrate on integrating visual and textual modalities using various cross-modal fusion strategies, such as cross-modal gating [15, 22, 42] , cross-modal attention [12, 36, 45] , and adaptive feature selection [9, 43] . Despite achieving impressive performance, there are significant domain gaps in general and specific domains. Domain-specific sentences contain a series of domain-related expressions, such as domain entities, domain terms, and domain idioms, which are significantly different from terms in general domain. Representation and translation of these domain-specific expressions pose critical challenges for DMNMT. Numerous studies have shown that visual information is often essential for translating domain-specific terms [33, 46] , whereas it is usually optional for general-domain text [39] . Therefore, integrating domain-specific visual details into text represents an effective strategy to address this challenge.Unfortunately, domain-related visual information tends to be fine-grained and dispersed. Figure 1 illustrates two examples of multi-modal machine translation in the general and specific domains. In contrast to multi-modal information in the general domain, the domain-specific image contains more focal areas related to domain terms in the text, such as the colored dashed regions illustrated in Figure 1 (b) . Therefore, visual-to-textual interaction in a specific domain often exhibits multi-focus characteristics.Several attempts have been made to explore domain-specific multi-modal fusion problems in DMNMT through techniques such as domain multi-modal data augmentation [46] , domain-specific pretraining [33] and domain-specific multimodal feature interaction [14] . However, most existing multi-modal fusion approaches primarily follow the steps of traditional multi-modal fusion in general domain, making it challenging to consistently focus on domainspecific multi-visual details. This limitation may impact their adaptability and effectiveness for multi-modal domain representations.Modality exchanging [37, 38] provides a potential cross-modal selection mechanism within and across modalities to aggregate these dispersed domain visual details simultaneously from domain image-text data pairs. The domain-specific multi-modal information can be aggregated from visual to textual direction through intra-modality selection and inter-modality exchanging. These modality-mixed domain details can then serve as multi-modal domainshadow representations to augment textual domain representations, thereby enhancing domain translation performance.However, these modality-mixed domain-shadow representations are generated through cross-modal feature direct exchanging, which may lack smoothness in multi-modal space. Therefore, there is a significant representation gap between the representations of modality-mixed domain-shadow details and raw text. The modalityagnostic virtual visual scene [25] , created through visual distillation [25, 30] with textual inputs, serves as multi-modal domain guidance to enhance the generation of smoother multimodal representations for domain image-text data pairs. Inspired by this, this paper presents a virtual visual scene-guided progressive domainshadow multi-modal fusion mechanism to gradually facilitate the integration of textual semantics with domain-specific visual details through layer-by-layer cross-modal exchanging, thereby enhancing the collaboration between domain characteristics and textual semantics. Compared to existing works, the major contributions of our paper are three-fold.• To tackle the multi-focus challenges of visual-to-textual interaction in DMNMT, we present a novel virtual visual sceneguided progressive domain-shadow fusion mechanism aimed at integrating textual semantics with domain-specific visual details progressively to improve the model's ability to perceive fine-grained domain visual details in DMNMT. Our proposed approach is designed to concurrently capture dispersed domain visual details through visual-textual modality mixing, and gradually aggregate modality-mixed domain representations and text with the shadow guidance of a modality-agnostic virtual visual scene. Additionally, the virtual visual scene benefits from cross-modal adaptive distillation. • We employ a modality-mixing selection-voting strategy to aggregate domain-specific multi-modal representations through intra-modality selection and inter-modality exchanging. Initially, we introduce a fine-grained domain voting strategy to select domain visual details and domain loosely-related textual tokens in their respective modality-specific spaces. Subsequently, we facilitate the exchange of selected information across modality boundaries to generate modalitymixed domain representations by incorporating visual details into text. • The extensive experiments on three datasets demonstrate that our proposed approach achieves state-of-the-art (SOTA) scores on two domain-specific and one in-general machine translation tasks. The in-depth analysis showcases that our proposed approach also achieves significant robustness and generalizability across various scenarios, such as noisy imagetext or even text-only scenarios. Moreover, our proposed virtual visual scene generation module still exhibits strong model compression capability, underscoring its potential for practical applications. 
related work: 
 Multimodal Neural Machine Translation. Recently, MNMT has drawn much attention in the field of Natural Language Processing (NLP). Existing MNMT works mainly focus on how to better integrate visual information into text to enhance the performance of machine translation. There are two types of visual information integration strategies for MNMT, including 1) image-must method: traditional MNMT works [16, 21, 36] often preferred to employ the aligned images to enhance machine translation performance through image-must multi-modal fusion strategies. Yao et al. [40] proposed a Transformer-based multimodal self-attention mechanism to address the noisy-robust multi-modal fusion problem in MNMT. Similarly, Ye et al. [41] developed a mask-guided cross-attention framework to tackle the issue of visual-textual semantic alignment in MNMT. Furthermore, Li et al. [24] devised a semi-supervised multimodal attention to fuse textual and visual modalities through cross-modal alignment. 2) image-free MNMT methods: recently, image-free MNMT methods [9, 25, 27, 30] have achieved wide attention to enhance machine translation performance through knowledge distillation. These approaches aimed to alleviate the constraints imposed by triplet data for MNMT. Long et al. [27] proposed a visual imagination method to synthesize continuous image features for machine translation. Li et al. [30] utilized a visual-hallucination method to generate discrete visual representations of text, enhancing machine translation performance. Guo et al. [16] aimed to enhance machine translation performance with the aid of synthetic representations by minimizing the semantic gaps between ground-truth and synthetic images. However, the aforementioned studies primarily focused on in-general cross-modal fusion, ignoring the specific challenges posed by DM-NMT. How to leverage visual information to improve domain-specific machine translation performance is still an open problem. Knowledge Distillation. Knowledge distillation (KD) [3, 17, 18, 30] aims to transfer knowledge from a teacher to student models, which has achieved wide attention in the fields of both computer vision and NLP. This concept was initially introduced by [3] and subsequently improved by [20] . Subsequent works have further improved the logits-based KD through structural information [13] , model ensemble [28] and adversarial learning [26] . Then some approaches [1, 4] explored feature-based knowledge distillation by utilizing the intermediate representations as hint knowledge. Furthermore, cross-modal knowledge distillation [11, 23, 31] has garnered significant attention recently. Sarkar et al. [31] devised a selfsupervised framework to perform effective information sharing between audio and video streams to obtain more generalized representations through cross-modal KD. IKD-MMT [30] introduced an inverse knowledge distillation framework to generate multimodal representations according to source text. In addition, some works have applied KD to Large Language Models. For example, Li et al. [23] proposed a CLIP-based knowledge distillation hashing approach to capture the semantic relevance and coexistent information for multimodal data. However, most KD approaches often overlook the fact that textual-visual KD approaches contain valuable multimodal information during the distillation process, which could effectively guide the fusion of multimodal information. 
methodology: 
 One of the critical challenges of DMNMT is to incorporate domain visual details into text to enhance machine translation performance. However, in contrast to multi-modal fusion in the general domain, domain-specific visual-to-textual fusion suffers from multi-focus challenges. To tackle this challenge, this paper presents a virtual visual scene-guided progressive domain-shadow multimodal fusion approach to capture domain-specific multi-modal representations, thereby enhancing machine translation performance. The overall framework of our paper is illustrated in Figure 2 , which consists of the following three subsections: 1) Modality-specific embeddings; 2) Domain-shadow aggregation with modality-mixing selection-voting strategy; and 3) Virtual visual scene-guided progressive domain-shadow fusion.Denote by {𝑥 𝑘 , 𝑣 𝑘 , 𝑦 𝑘 } as the 𝑘-th domain data pair, where 𝑥 𝑘 = {𝑥 1 , . . . , 𝑥 𝑘 𝑛 } and 𝑦 𝑘 = {𝑦 1 , . . . , 𝑦 𝑘 𝑚 } denote the domain-specific source and target sentences, respectively. 𝑣 𝑘 represents their corresponding image, 𝑛 and 𝑚 are the lengths of 𝑥 𝑘 and 𝑦 𝑘 .We first leverage the textual embedding module to extract initial textual representation, as shown as follows:EQUATIONwhere Emb s (•) denotes the traditional textual embedding layer with positional embedding; the textual embedding representation 𝐸 𝑥 ∈ R 𝑛×𝑑 , and 𝑑 denotes the dimension of the textual embedding vector. 1We then leverage the pretrained Resnet-101 model [19] to extract initial visual representation, as shown as follows:EQUATIONwhere Emb 𝑣 (•) denotes the visual embedding layer with the pretrained Resnet-101 model followed by a single-layer Multi-layer Perception; the visual embedding representation 𝐸 𝑣 𝑘 ∈ R 7×7×𝑑 .Incorporating the fine-grained and dispersed visual domain details into text has been proven effective in enhancing machine translation performance for DMNMT. To achieve this, we first attempt to aggregate domain-specific multi-modal information from the visual-to-textual direction to generate domain-shadow information through modality-mixing selection-voting strategy. Specifically, we first employ a fine-grained domain voting strategy to select domain closely-related visual details and domain loosely-related textual tokens in visual and textual modality spaces, respectively. Then we exchange the candidate fine-grained information across modality boundaries from visual to textual to generate modality-mixing domain representations.We first employ two modalityspecific 𝐿-stacked Transformer layers to extract the textual and visual representations, respectively. The textual representation can be encoded as follows,EQUATIONwhere TransEnc 𝑙 𝑠 (•) denotes the 𝑙-stacked Transformer layers, each Transformer layer consists of the multi-head attention (MHA) and feed forward networks (FFN); the layer index 𝑙 = 1, . . . , 𝐿, and when 𝑙 = 1, we set 𝐶 0 𝑥 = 𝐸 𝑥 ; the textual representation 𝐶 𝑙 𝑥 ∈ R 𝑛×𝑑 . Similarly, the visual representation is extracted as follows,EQUATIONwhere TransEnc 𝑙 𝑣 (•) denotes the 𝑙-stacked Transformer layers, each Transformer layer consists of MHA and FFN; the layer index 𝑙 = 1, . . . , 𝐿, and when 𝑙 = 1, we set 𝐶 0 𝑣 = 𝐸 𝑣 ; the visual representation𝐶 𝑙 𝑣 ∈ R 49×𝑑 .To better capture domainspecific information from image-text data pair, we employ a domain feature selection mechanism within each modality space to adaptively select domain-related visual details and domain looselyrelated textual details through intra-modality probability sampling. We design two domain-aware selective gating mechanisms in each stacked Transformer layer, as depicted as follows:EQUATIONEQUATIONwhere 𝜎 ( and visual selected gating in the 𝑙-th Transformer layer, and𝐺 𝑙 𝑥 , 𝐺 𝑙 𝑣 ∈ R 𝑛×𝑑 , R 49×𝑑 .Then we employ a probability-based fine-grained 𝑡-sampling mechanism 2 to select domain-related visual details and domain loosely-related textual details dynamically, as shown as follows:EQUATIONEQUATIONWhere MulSam(•) denotes a probability-based fine-grained feature sample mechanism and 𝑡 is the number of sampling iterations; 𝐺 𝑙 𝑥 represents the inverse sample probability, and𝐺 𝑙 𝑥 = 1 -𝐺 𝑙 𝑥 . 𝑆 𝑙 𝑥 , 𝑆 𝑙 𝑣 ∈ R 𝑡 ×𝑑 ,R 𝑡 ×𝑑 denote the selected representations in the 𝑙-th Transformer layer.Voting. We randomly exchange the selected representations across modality boundaries 𝜏 times to generate 𝜏 candidate modality-mixing domain representations by incorporating visual details into text. Then we select the smoothest exchange sample from all 𝜏 candidate exchange sets to obtain the modality-mixing representations.Inter-modality Exchanging. We adopt a point-to-point exchange mechanism from visual to textual direction to exchange domain-related visual representations and domain loosely-related textual representations 𝜏 times to generate the modality-mixing domain representation set 𝜙 𝑙 x , as depicted as follows,EQUATION2 We adopt 𝑡 times independent repeated sampling with replacement.where Exch(•) denotes the point-to-point exchange operation; < 𝑆 𝑙 𝑣,𝛼 → 𝑆 𝑙 𝑥,𝛽 > denotes an operation by replacing the 𝛼-th vector of 𝑆 𝑙 𝑣 with the 𝛽-th vector of 𝑆 𝑙 𝑥 . 𝛼, 𝛽 = randint(1, 𝑡) represents a random sample from 1 to 𝑡; C𝑙 𝑥 (𝑟 ) ∈ R 𝑛×𝑑 denotes a candidate exchange sample.The candidate modality-mixing representation set can be obtained as follows,EQUATION)where 𝑟 is the index of candidate modality-mixing representation.Inter-modality Voting. We only attempt to select a smoothest exchange sample 𝐷 𝑙 𝑥 from all 𝜏 candidate exchange set 𝜙 𝑙 𝑥 = { C𝑙 𝑥 (𝑟 ) |𝑟 = 1, . . . , 𝜏 } to obtain the modality-mixing representation via the minimal change in KL divergence score, as described as follows,EQUATIONwhere 𝐷 𝑙 𝑥 ∈ R 𝑛×𝑑 is the selected modality-mixing representation; 𝑢 is the selected index from 1 to 𝜏, and 𝑢 can be defined as follows,EQUATIONwhere arg min(•) denotes the operation of calculating the minimum KL scores among all candidates; KL C𝑙 𝑥 (𝑟 ) 𝐶 𝑙 𝑥 denotes the KL divergence score between the 𝑟 -th candidate modality-mixing representation C𝑙 𝑥 (𝑟 ) and the textual representation 𝐶 𝑙 𝑥 in the 𝑙-th layer.This modality-mixing domain representation 𝐷 𝑙 𝑥 will be used to generate domain-specific multi-modal representation.To promote collaboration between modality-mixing domain representation and textual representation, we present a virtual visual scene-guided domain-shadow fusion strategy in each Transformer layer.We employ a 𝑃-stacked Transformer encoder to generate virtual visual representation by taking the initial textual embedding as input, and we have that,EQUATIONwhere StackTransEnc ℎ (•) denotes the 𝑃-stacked Transformer encoder layers, each Transformer layer consists of MHA and FFN modules; 𝐶 ℎ ∈ R 𝑛×𝑑 denotes the virtual visual scene representation generated from the source text.Then we adopt a virtual visual scene-guided domain information aggregation strategy to integrate modality-mixed domain representation and virtual visual representation through cross-modal gating fusion, as described as follows,EQUATION)where 𝐹 𝑙 ∈ R 𝑛×𝑑 denotes the fused multi-modal domain representation with the guidance of virtual visual representation at the 𝑙-layer; and the gating 𝛿 𝑙 is calculated as follows:EQUATIONwhere 𝑊 𝑙 ℎ ∈ R 𝑑 ×𝑑 and 𝑊 𝑙 𝑓 ∈ R 𝑑 ×𝑑 are learnable parameters, sigmoid(•) represents the element-wise sigmoid transformation.To comprehensively enhance domain-shadow fusion, we adopt an adaptive cross-layer domain-shadow fusion strategy to progressively integrate modality-mixed domain-shadow information layer-by-layer with the guidance of virtual visual representation, as demonstrated in Figure 2 .The virtual visual scene plays a critical role in domain-specific multi-modal representation. To further enhance domain-shadow fusion, we employ a multi-layer visual distillation mechanism aimed at capturing multi-grained domain visual details at each Transformer layer. During the training stage, the model's parameters are optimized using a joint loss function, which is detailed as follows:EQUATION)where 𝑙𝑜𝑠𝑠 𝐶𝐸 represents traditional machine translation loss and 𝜃 𝑛 is the layer-wise loss hyper-parameter; the visual distillation loss 𝑙𝑜𝑠𝑠 𝑛 ℎ is defined as follows:EQUATIONwhere KL(•) denotes the operation of calculating KL scores; (•||•) represents visual-centric distillation by randomly up-sampling 49 times in the length dimension to transform 𝐶 ℎ into the length dimension of 𝐶 𝑛 𝑣 .Furthermore, in the inference stage, our model can utilize virtual visual information instead of the ground truth visual representation as visual input to provide multi-grained visual representation for machine translation. Therefore, our model can adapt to text-only scenarios. 
experiments 4.1 experimental settings: 
 Datasets. We conduct experiments on three benchmark MNMT datasets, including two domain-specific datasets, Fashion-MMT [33] and EMMT [46] , and one general-domain dataset, Multi-30k [7] . Specifically, 1) Fashion-MMT is a MNMT dataset containing two sub-datasets: Fashion-MMT(clean) and Fashion-MMT(large) in the E-commerce domain. The Fashion-MMT(clean) dataset is composed of 40,000 image-text pairs. Each data pair includes an English description, one or more images, and a manually edited Chinese translation. The Fashion-MMT(large) dataset contains 114,257 imagetext pairs, where each data pair contains one or more images, an English description, and a noisy Chinese sentence translated by a text-only SOTA model. 2) EMMT: The EMMT dataset is a realworld e-commercial dataset collected from TikTok Shop and Shopee. It comprises 22,500 annotated triplets, where each triplet consists of an English product description, a manually annotated Chinese translation, and a corresponding image. The test set is selected by professional annotators, comprising examples that are challenging to translate without corresponding images. 3) Multi-30k: Multi-30k is the widely used benchmark for MNMT tasks, covering a variety of general-domain scenarios. The Multi-30k dataset contains 29k text-image data pairs for training and 1014 data pairs for validation. And we follow standard evaluation setup to report the results on three test splits, Test2016, Test2017 and MSCOCO.Evaluation Metrics. We utilize three types of metrics to evaluate the performance of machine translation, including BLEU [29] , METEOR [5] , and BLEURT. BLEURT is a robust noise evaluation metric proposed by [32] , demonstrating a strong correlation with human evaluation. We also report the average scores and Student's t-test (T.TEST) scores by running each model three times.Implementation Details. We employ byte pair encoding (BPE) segmentation with 8k, 10k, and 6k merge operations for the Fashion-MMT, EMMT and Multi-30k datasets, respectively. The vocabulary sizes are 8880-2936 tokens for the Fashion-MMT dataset, 10407-9799 tokens for the EMMT dataset, 5644-5876 tokens for the Multi-30k (En-De) translation task, and 5644-5972 tokens for the Multi-30k (En-Fr) translation task. We utilize the pre-trained CLIP model [32] to represent textual and visual features into a shared multimodal space, thereby obtaining the most semantic-related image features corresponding to its text for Fashion-MMT dataset. Furthermore, our model consists of 4 stacked encoders and 4 stacked decoders based on the Transformer-based seq2seq framework for all datasets. 1 . "Our model(G)" and "Our model(H)" denote our proposed model that utilizes ground-truth images and virtual visual information, respectively. We can see that 1) Our model achieves a significant improvement over all the other SOTA DM-NMT approaches across three types of evaluation metrics on two datasets.2) The proposed approach significantly outperforms textonly transformer approach, confirming the effectiveness of visual information for machine translation. 3) Both Our model(G) and Our model(H) achieve comparable machine translation scores on all test sets. It demonstrates the effectiveness of our proposed approach for DMNMT.To further confirm the robustness of our proposed method, we conduct additional experiments on the Multi30k dataset. The results of English-to-German and English-to-French translation tasks are presented in Table 2 . The findings are as follows: 1) In comparison to existing MNMT models, our method achieves SOTA scores in the BLEU and METEOR metrics on the test2016, test2017, and MSCOCO test sets. 2) Compared to our reproduced NMT and MNMT models, our approach demonstrates significant improvements under the same parameter and environment settings. Furthermore, we also conduct significance tests between our reproduced models and our approach. The significance test results indicate that our model achieves a statistically significant improvement over these models (p-value < 0.01). 3) Our model(G) and Our model(H) also achieve comparable results on these three test sets. This confirms the effectiveness of our virtual scene generation module. Furthermore, it is noteworthy that the MSCOCO test split, which includes sentences with ambiguous verbs and out-of-domain samples from the COCO Captions dataset, is often challenging for MNMT models. However, our model performs exceptionally well on this test set, which suggests it can effectively employ visual information to handle ambiguity through our proposed modality exchanging mechanism.Modules. We first conduct ablation studies on different modules to demonstrate the effectiveness of the modules in our proposed approach. The experiment results are shown in Table 3 . The conclusions could be drawn as follows: 1) Replacing the cross-modal exchanging module with original textual representation causes a significant performance drop on Fashion-MMT(small) and Fashion-MMT(large) datasets. 2) Replacing the cross-modal fusion module with a pooling-addition operation also causes a remarkable machine translation performance decline on two tasks. 3) Removing virtual visual scene generation module causes huger BLEU and BLEURT scores drop than replacing cross-modal fusion module, which confirms that the visual scene generation module plays a significant role in guiding the process of multimodal fusion. The results confirm the validity of the proposed modules in domain-specific translation tasks, especially for cross-modal exchanging module.We then conduct experimental analysis on the impact of parameter counts on the virtual visual scene generation module to investigate its model compression capability, as shown in Figure 3 . Where the rectangular boxes represent the BLEU scores, the dashed and dotted lines represent the model's parameter counts. We evaluate the model compression capability for several types of modules with comparable parameter counts, including our proposed 𝑃-stacked Transformer encoder, the traditional selective attention [22] , the Gated Fusion [39] , and Imagination [8] modules.It can be included as follows: 1) Our proposed virtual visual scene generation module still exhibits strong machine translation performance even when the number of stacked Transformer layers is set to 1. 2) To further demonstrate the robustness and generalization of our virtual visual generation module, we also conduct experiments in the widely-used Multi30k(En-De) Test2016 task. The experimental results showcase that our 𝑃-stacked Transformer encoder achieves the highest BLEU scores when the layer number is 1, with the fewest number of parameters. These findings confirm the effectiveness of model compression, highlighting its potential for virtual visual representation.To investigate the robustness of our model for visual information, we further examine the validity of images in machine translation by adversarial evaluation [6] , as demonstrated in Table 4 . Specifically, we replace ground-truth images by the following three types of images, including blank image (BlkImg), randomly selected image (RSImg), noisy image (NsImg). Furthermore, we evaluate machine translation performance with visual adversarial evaluation by considering the following three additional scenarios: 1) Text-BlkImg data pair scenario; 2) Text-RSImg data pair scenario, and 3) Text-NsImg data pair scenario. The experimental results demonstrate that our approach exhibits significant robustness across several visual noisy scenarios. Furthermore, our model(H) achieves slightly higher BLEU and METEOR scores in all three noisy multi-modal scenarios, as it does not rely on noisy visual images. This suggests that noisy images indeed have a negative impact on machine translation. Figure 4 depicts the translation of two domain-specific cases of Fashion-MMT dataset. Colors highlight improvement. In these examples, our proposed approach can translate domain-specific terms correctly, such as "sketched blooms, " "cap sleeve top, " "dusty pink ground, " etc. The text-only model fails to generate the domain term "cap sleeve top" without the aid of visual information. This confirms the importance of visual information for DMNMT. Moreover, our model(H) still can translate domain-specific terms correctly, such as " 手绘花朵, "" 飘逸的粉色布料, " and " 精致的褶边. " It demonstrates the effectiveness of virtual visual scene generation for domain-specific term translation.To explore the robustness of our proposed model and its ability to use image information for improving domain-specific translations when key text details are missing, we replaced certain domain terms with [MASK] during testing. As illustrated in Figure 5 , by visualizing the attention weights of the Vision Transformer, we masked words like "checks", "diamond" and "studs". These words, which vary greatly between specific and general domains, pose a translation challenge without corresponding image cues. In Case 1, despite masking the word "checks, " which differs significantly from its usual expression, our model achieved accurate translation by focusing on related image areas. Intriguingly, in Case 2, accurately translating masked domain-related words "diamond" and "studs" is significantly challenging without corresponding images. The focused image information by our model indicates it can precisely identify relevant areas for these domain-specific terms, enhancing translation performance. 
conclusion: 
 This paper has addressed the multi-focus challenges associated with visual-to-textual interaction in DMNMT, introducing a virtual visual scene-guided progressive domain-shadow fusion approach to enhance the model's capability in perceiving fine-grained, domainspecific visual details. Extensive experiments conducted on three benchmark datasets demonstrate that our proposed approach surpasses SOTA MNMT models, achieving significant improvements across all machine translation tasks. Moreover, our in-depth ablation studies and adversarial evaluation underscore the robustness and generalizability of our approach across both general and domain-specific contexts, even under noisy or text-only conditions. Additionally, the virtual visual scene generation module has shown a remarkable potential for model compression, indicating its viability for real-world applications. 
 
 Appendix:  


Decision: Two reviewers are in favor of accepting (weak) the paper while one tends towards rejecting (borderline).
I suggest accepting this paper because the authors adequately addressed most of the reviewers' concerns in their rebuttal.
Regarding the reviewer reporting a final rating of borderline reject, his final rating justification says that the "author partially addressed my concern" while keeping his original score.

Reviews: review: paper_summary: paper_summary: This paper is about improving domain-specific translation by exploiting multimodal information, i.e., images associated to text.
A complex transformer-based model is proposed to fuse visual and textual representations that enable focusing on domain-specific aspects of the multimodal content and the level of interaction between the different modalities.
strength_weakness: strength_weakness: The proposed model is an original and sound approach to the problem. Its ideas can find application in other tasks, e.g., multi-modal multi-language classification.
The technical parts of the paper are ok (see limitations for a more general comment on the paper content).
Experimental setup is clearly presented and solid.
Multiple dataset are used to test the methods.
To the best of my knowledge the methods used as baselines are a sound choice for the task.
Results show a statistically significant improvement, supporting the claims and the proposed methods.
Ablation study is interesting.
Code to replicate the experiments is available.
limitations: limitations: This is a technical paper, with a "cold" way to present its content. This could limit its reach in the MM community.
Title is almost sequence of keywords and it is complex, somewhat cryptic to the non-expert, and does not clearly conveys the task and contribution of the paper.
A title should be simpler to read, and convey a general idea of the work, that is then clearly presented in the abstract and in the paper with increasing detail and precision.
The whole paper is a bit hard to read, as it is very repetitive in the use of multiword terms that are not properly explained until the very central part of the paper, leaving the reader with partial information for the most part of the reading. For example the expression "virtual visual scene-guided domain-shadow multi-modal fusion" appears more than 10 times, and it can be understood only from the technical section of the paper. Using a bit of the first part of the paper to break it in its various parts and giving an intuitive explanation of the concepts involved would help a lot the readability.
A computational cost comparison is missing.
paper_summary: paper_summary: This paper focuses on domain-specific multi-modal neural machine translation, which faces the challenges of visual-to-textual interaction in a specific domain, frequently exhibiting multi-focus characteristics. To address the challenge, this paper proposes a  a virtual visual scene-guided progressive domain-shadow fusion approach, which includes domain-shadow aggregation with modality-mixing selection-voting strategy and  virtual visual scene-guided progressive domain-shadow fusion. Extensive experimental results on public datasets demonstrate the effectiveness of proposed method in the face of missing modal data.
strength_weakness: strength_weakness: 1.The method significantly enhances the model’s capability in perceiving fine-grained, domain-specific visual details, addressing the multi-focus challenges of visual-to-textual interaction in DMNMT.

2.Comprehensive experimental analysis has shown that this method exhibits significant robustness and generalizability across various scenarios.
limitations: limitations: 1.The expression in the paper is too casual, and the structure is not reasonable. For instance, Lines 129 and 307 even extend beyond the line, causing disorder. Incorrect use of double quotes and single quotes in lines 895 and 901. “... presented in Tables 2” should be “... presented in Table 2”.

2.The proposed method lacks novelty. Several modules proposed by the authors are very common. Where is your advantage? 

3.In line 447, why are textual embeddings used as input to generate virtual visual representations?

4.In line 455, the modality-mixed domain representation already includes visual and textual features. Why is it necessary to combine it with the virtual visual representation obtained in the previous step?

5.The methods [1,2] are not included in the comparative experiments. Provide reasons for their exclusion.

[1]Progressive modality-complement aggregative multitransformer for domain multi-modal neural machine translation
[2]Dual transfer learning for neural machine translation with marginal distribution regularization
paper_summary: paper_summary: In order to realize domain-specific multi-modal neural machine translation, this paper proposes a novel modality-mixing selection-voting strategy to integrate visual details into text representations through intra-modality selection and inter-modality exchanging. After that, a virtual visual scene-guided progressive domain-shadow fusion is developed to generate domain-specific multi-model representation, so that the machine translation performance can be improved.
strength_weakness: strength_weakness: 1.  The paper is well written and easy to follow. The figures are clear and easy to understand.
2.  Utilizing modality-mixing selection-voting strategy to incorporate visual details into text is very interesting. And the related operations are technical reasonable.
3.  Extensive experiments and ablation studies are conducted, and the results demonstrate the effectiveness of the proposed method.
limitations: limitations: Just two small questions:

1.	I can understand “visual-to-textual interaction in a specific domain often exhibits multi-focus characteristics”. But I am little confused that why dose this bring challenges or negative effects to the multi-modal machine translation? Could the authors explain it in more details?

2.	In my own opinion, the Virtual Visual Scene-guided Progressive Domain-shadow Fusion module is just a gated-fusion process, could the authors explain its underlying mechanism for tackling the multi-focus challenges (as claimed in 249th-250th rows)?

Overall, I vote for borderline accept for this paper.