Towards Multimodal-augmented Pre-trained Language Models via Self-balanced Expectation-Maximization Iteration

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 19:39:03

Abstract: Pre-trained language models (PLMs) that rely solely on textual corpus may present limitations in multimodal semantics comprehension. Existing studies attempt to alleviate this issue by incorporating additional modal information through image retrieval or generation. However, these methods: (1) inevitably encounter modality gaps and noise; (2) treat all modalities indiscriminately; and (3) ignore visual or acoustic semantics of key entities. To tackle these challenges, we propose a novel principled iterative framework for multimodal-augmented PLMs termed MASE, which achieves efficient and balanced injection of multimodal semantics under the proposed Expectation Maximization (EM) based iterative algorithm. Initially, MASE utilizes multimodal proxies instead of explicit data to enhance PLMs, which avoids noise and modality gaps. In E-step, MASE adopts a novel information-driven self-balanced strategy to estimate allocation weights. Furthermore, MASE employs heterogeneous graph attention to capture entity-level fine-grained semantics on the proposed multimodal-semantic scene graph. In M-step, MASE injects global multimodal knowledge into PLMs through a cross-modal contrastive loss. Experimental results show that MASE consistently outperforms competitive baselines on multiple tasks across various architectures. More impressively, MASE is compatible with existing efficient parameter fine-tuning methods, such as prompt learning.

Main paper: 
 introduction: 
 Leveraging self-supervised techniques, pre-trained language models (PLMs) such as BERT [9] , T5 [30] , GPT [3] , have achieved strong progress towards comprehending human language. The paradigm of fine-tuning PLMs has achieved tremendous success in various downstream NLP tasks [3, 26, 28] . However, most existing PLMs rely predominantly on contextual learning which only takes the textual context as self-supervision [25, 38] . These language learners Most existing methods treat various modal information indiscriminately and inevitably encounter modality gaps and noise. Our MASE can achieve a uniform representation space while achieving self-balancing weighting of various modal information.overlook the potential of learning from multimodal corpora, e.g., image, audio and video [2] and cannot well capture multimodalgrounded semantics, e.g., visual commonsense [17] . Moreover, prior studies have proved that these PLMs exhibit human biases [12] and lack general multimodal knowledge [50] , e.g., color and shape. To overcome the above challenges, existing methods mainly enhance the visual comprehension ability of PLMs by retrieving or synthesizing real images. These methods employ images to enhance PLMs during the pre-training [39, 43] or the fine-tuning stage [15, 25] , or leverage images to bolster the zero-shot capabilities of PLMs [48] . Despite the notable progress achieved by these studies, we identify several key issues: (1) Modality Gap and Noise: As shown in Figure 1 , they necessitate labor-intensive image retrieval or generation and are prone to introducing irrelevant noise, which can degrade the performance. Furthermore, it is widely recognized that a large semantic gap [21] exists between different modalities, which will further hinder modal aggregation.(2) Modality Imbalance: They focus exclusively on image information while overlooking audio and video modalities, and handle all modal information indiscriminately. Recent work [7, 15] reveals that the relevance of various modal types varies across specific NLP tasks, and not all information from additional modalities proves beneficial. Consequently, the indiscriminate treatment of all modal information is suboptimal. (3) Ignoring Entity-level Multimodal Semantics: As shown in Figure 2 , from the perspective of the scene graph, we discover that key entities are highly correlated with additional modal information, such as 'dog' corresponding to visual features, and 'called' corresponding to audio features. Previous work overlooked fine-grained visual or acoustic semantics correlated with these key objects, which are crucial for improving the multimodal understanding of entities in PLMs. In this paper, we propose a novel principled EM-based iteration framework for self-balancing multimodal-augmented PLMs (MASE). As illustrated in Figure 1 , MASE tackles the above challenges through proposed strategies in the following three aspects:(1) MASE utilizes multimodal proxies as a bridge for information transmission and employs an EM-based iteration for optimization. MASE initially leverages textual instances to obtain implicit modal proxies through multimodal contrastive pretraining models (MC-PTMs) (e.g., CLIP [29] , AudioCLIP [16] and CLIP-ViP [47] , etc.). These proxies are intrinsically aligned with real-world data and enable the circumvention of noise and modality gaps. Subsequently, MASE conceptualizes the balance and efficient injection of multimodal information from proxies as a probabilistic problem, and adopts the EM algorithm [8] for optimization.(2) MASE utilizes a novel information-driven strategy for allocation estimation. Specifically, MASE modals information allocation weights as latent variables under our iteration framework, adopts the proposed mutual information (MI) based method to estimate the contribution of various modalities to the targets in the E-step. By our MI-based estimation approach, we can calculate a specified posterior probability of assigned weights to explore the intrinsic structure of multimodal information, achieve multimodal balance in the optimization process.(3) MASE achieves entity-level multimodal information enhancement via the proposed multimodal-semantic scene graph (MSSG). The multimodal representation of entities (including objects and attributes) and relations can reveal a large amount of world-commonsense information and rich visual-spatial semantics, which helps to enhance the multimodal understanding ability of PLMs. To fully exploit fine-grained cross-modal features at the entity level, We innovatively concatenate multimodal features with entity nodes in the vanilla scene graph to construct MSSG, and propose employing heterogeneous graph attention to efficiently extract enhanced representation upon MSSG in E-step.Following the balance and enhancement in the E-step, we utilize cross-modal contrastive learning to achieve global optimization of MASE in the M-step. Extensive experiments conducted on thirteen datasets across five tasks demonstrate the effectiveness and universality of MASE. In addition, MASE can be combined with existing efficient parameter fine-tuning strategies, such as prompt learning. In general, our contributions are three-fold:(Theory) We propose a principled iterative framework termed MASE that utilizes the EM-based algorithm to inject multimodal semantics into PLMs balance and efficiently. To our knowledge, it is the first theoretical framework for multimodal-augmented PLMs.(Methodology) We innovatively propose an MI-based information allocation estimation method and an entity-level multimodal aggregation method via the proposed MSSG to achieve modal semantic balance and enhancement, respectively.(Experiments) Extensive experiments on five tasks (including NLP, visual reasoning, and visual question answering) demonstrate the effectiveness and generality of MASE. More impressively, our method is compatible with different architectures and existing efficient parameter fine-tuning methods. 
related work: 
 Pre-trained Language Models. Recent advancements in largescale PLMs have been achieved through self-supervised learning techniques applied to extensive text corpora [3, 9, 30] . These PLMs exhibit robust generalization capabilities, and fine-tuning them significantly enhances the performance of various downstream tasks, including NLU, question answering, and text generation. It has been observed that language models trained exclusively on textual data may not adequately grasp multimodal contexts and world knowledge, such as visual commonsense [12, 48, 50] . Moreover, recent studies indicate that merely expanding the textual corpus does not overcome these limitations [14, 50] . In this article, we introduce a universal probability framework designed to enrich the multimodal semantic integration within PLMs.Multimodal Contrastive Pre-trained Models. MC-PTMs are developed through training on a substantial corpus of modalitypairing samples, enabling the mapping of various modalities into a unified representation space. CLIP [29] undergoes pre-training with a vast collection of image-text pairs, effectively achieving semantic alignment between images and text. AudioCLIP [16] and CLIP-ViP [47] expand CLIP-based approach to the audio modality and video domain, respectively. These works inspire us to obtain semantics aligned with real modal data through textual instances to enhance multimodal understanding of PLMs.Multimodal-Augmented Language Models. Most studies focus on incorporating visual modality information into PLMs through retrieval or image generation methods. Tan and Bansal [39] , Wang et al. [43] incorporate visual knowledge in the pretraining phase of PLMs. Guo et al. [15] , Lu et al. [25] enhance PLMs by integrating visual information during the fine-tuning process. Furthermore, Yang et al. [48] employs image generation or retrieval methods to encode visual representations, aiming to improve zeroshot NLU. As a comparison, MASE is the first theoretical framework for MA-PLMs, which can simultaneously achieve multimodal semantic injection and modal information balance. 
methods: 
 In this section, we first introduce the task setting and the multimodal proxy based baseline models for MA-PTMs in Section 3.1, and then describe our proposed probabilistic model in Section 3.2 for infusing multimodal knowledge into PLMs during fine-tuning. Task Settings. In general NLP scenarios, PLMs are adapted through fine-tuning on a specific dataset D = {𝑥 𝑖 , 𝑦 𝑖 } 𝑛 𝑖=1 , aimed at executing tasks like classification or regression, where, 𝑥 𝑖 and 𝑦 𝑖 is the 𝑖-th textual instance and its label, respectively, with 𝑛 denoting the total number of training instances. Our study focuses on devising an efficient framework for integrating multi-modal information into PLMs throughout the fine-tuning process. The set of modalities engaged in this paper is denoted as M = {T, I, A, V}, representing text, image, audio, and video modalities, correspondingly. Proxy-based Baseline Models for MA-PLMs. As the multimodal proxy based multimodal-augmented PLM approach has not been studied in NLP tasks, we extend the vision-enhanced PLM approach, namely VAWI [15] , to a multimodal setting and combine with MC-PTMs to achieve multimodal proxy based multimodal information extraction. MC-PTMs perform contrastive learning across a large number of cross-modal data pairs, and their text features are highly aligned with other modal features. Thus, the text semantics of MC-PTMs contain modal-specific representation encodings and can serve as implicit proxies. In practice, we employ the CLIPbased text encoder of MC-PTMs to facilitate implicit multimodal information as ℎ 𝑚 𝑖 = MC-PTMs 𝑚 (𝑥 𝑖 ), where, 𝑚 denotes the represents a specific modality, i.e., 𝑚 ∈ M\T. In this work, we utilize CLIP [29] , AudioCLIP [16] and CLIP-ViP [47] to obtain hidden state encodings of image, audio, and video modality proxies, respectively. Subsequently, we concatenate different modal semantics (i.e., ℎ 𝑚 𝑖 , ∀𝑚 ∈ M) and feed them into a classifier or regression to obtain predictions. We further provide theoretical analysis in supplements about our multimodal proxy is an excellent multimodal information transmission bridge.Our MASE, as shown in Figure 3 , adaptively assigns soft weights to different modal semantics ℎ 𝑚 𝑖 , where the lower weights resort to weakening the negative impact of irrelevant modalities. Specifically, MASE prioritizes the weight of information assignment as a latent variable and then constructs a general probabilistic model for various PLMs as follows: Latent Variables for Information Assignment. We consider the weight of information assignment as a latent variable 𝑧 𝑖 which represents the degree of contribution of different modalities to specific predictions. The latent variable 𝑧 𝑖 = {𝑧 𝑚 𝑖 } as a measure of information allocation will directly act on the multimodal semantics ℎ 𝑚 𝑖 , ∀𝑚 ∈ M to obtain the optimal representation ℎ * 𝑖 :EQUATIONProbabilistic Model with Latent Variables. Based on the multimodal representations h 𝑖 = {ℎ 𝑚 𝑖 } and latent variables 𝑧 𝑖 , we can optimize the model by maximizing the following log-likelihood: EQUATIONEQUATIONwhere Θ denotes the learnable parameters of the model. Based on this, we can use Jensen's inequality [19] to derive the lower bound F (Θ, {𝑃 𝑖 } 𝑛 𝑖=1 ) of the log-likelihood of Eq. 2 as:EQUATIONwhere we abbreviate 𝑃 𝑖 (𝑧 𝑖 ) as 𝑃 𝑖 .Eq. 4 inspires us to optimize the latent variables 𝑧 𝑖 and the model parameters Θ by optimizing the lower bound F Θ, {𝑃 𝑖 } 𝑛 𝑖=1 . This approach addresses the challenges associated with optimizing the likelihood function ℓ (Θ, 𝑧). In this work, we propose a strategy based on the Expectation-Maximization (EM) algorithm to optimize this lower bound.MASE iteratively optimizes Θ and {𝑃 𝑖 } 𝑛 𝑖=1 through E-step and Mstep to achieve the balance of information allocation and effective injection of multimodal information, respectively. E-step for Self-balanced Information Assignment. In E-step, we keep the current parameters Θ (𝑡 -1) fixed and and maximize the lower bound F to estimate the allocation weights of different modal information as follows:EQUATIONWe will specifically discuss our information-driven allocation method for estimating 𝑝 𝑧 𝑖 | h 𝑖 , 𝑦 𝑖 ; Θ (𝑡 -1) in Section 3.4. E-step for Entity-level Information Injection. After obtaining the balanced modal features ℎ 𝑚 𝑖 , ∀𝑚 ∈ M, we employ the proposed MSSG and heterogeneous graph attention networks to obtain enhanced multimodal features to ĥ𝑚 𝑖 , as detailed in Section 3.5. M-step for Multimodal Information Injection. In this step, we hold 𝑃 𝑖 (𝑧 𝑖 ) (𝑡 ) and update Θ to maximize the lower bound for injecting multimodal information as:EQUATIONIn practice, MASE achieves the optimization goal of M-step in Eq. 6 by performing cross-entropy loss and cross-modal contrastive loss on the aggregated multimodal features ĥ * 𝑖 via Eq. 1. The crossentropy loss is used to represent prediction error as:EQUATIONwhere CE(•) denotes the cross-entropy function. And the crossmodal contrastive loss is used to optimize the representation between modalities, which can be represented as:EQUATION)where 𝜏 is the temperature coefficient. Specifically, we minimize the following total loss function in M-step to inject balanced multimodal knowledge into PLMs:EQUATIONwhere 𝛼 denotes a trade-off coefficient.The optimization loss L 𝑀 incorporates strategies for allocating multimodal information via 𝑧 𝑖 to achieve an adaptive balance and inject knowledge into PLMs effectively during the M-step phase.During training, we initialize 𝑧 𝑖 to a uniform distribution and then iterate on E-step and M-step until convergence.In Section 3.3, it remains a crucial question of how to estimate the distribution of multimodal information allocation latent variables 𝑝 (𝑧 𝑖 | h 𝑖 , 𝑦 𝑖 ; Θ) in the E-step. We propose a novel MI-driven multimodal knowledge allocation strategy for dynamically estimating the distribution of latent variables. Specifically, we calculate the mutual information between different modal semantics and task labels in E-step as a measure of modalities' importance. This strategy is based on intuition: the greater the contribution of modal knowledge to task targets, the greater the mutual information. We mathematically define the mutual information between specific modal knowledge ℎ 𝑚 and labels 𝑦 as:EQUATIONwhere H Θ (•) denotes information entropy under the model Θ. In practice, MASE adopts a Monte Carlo estimator to estimate the mutual information of random variables in a batch. The mutual information I (ℎ 𝑚 , 𝑦) has a lower bound and an upper bound, i.e., 0 ≤ I (ℎ 𝑚 , 𝑦) ≤ H Θ (𝑦). H Θ (𝑦) is a fixed value representing the information entropy of the task dataset labels. Based on the above analysis, we can utilize normalized mutual information to establish an information allocation distribution model 𝑝 𝑧 𝑖 | h 𝑖 , 𝑦 𝑖 ; Θ (𝑡 ) forEQUATIONwhere 𝑚 ∈ M denotes different modalities. We achieve the dynamic balance of multimodal information in the iterative optimization by using our MI-driven allocation strategy (Eq. 11) in E-step.To fully exploit fine-grained inter-modality features at the entity level, we propose the construction of a multimodal-semantic scene graph. This MSSG utilizes balanced multimodal features to enable entity-level fine-grained interactions, as illustrated in Figures 2 and 3 . Our MSSG is an extension of the vanilla semantic scene graph that represents the intrinsic relationships between entities under textual semantics. To incorporate collaborative modality effects into representation learning, we concatenate additional modal features (such as video and audio features) into entities within the traditional scene graph to obtain a structured multimodal context. Specifically, we employ a scene graph parser [46] 1 to initially generate the textual scene graphs G T = {E T , R T }, where E T represents entities and R T denotes relation within textual descriptions. Subsequently, we augment these textual entities E T by aggregating multimodalentity features to construct the MSSG G M = {E M , R M }, where,• E M = {[ M\{T} || 𝑚 ℎ 𝑚 𝑖 , 𝑒 T 𝑖 ]}, ∀𝑒 T𝑖 ∈ E T , represents the entity set aggregated with multimodal semantics, where, || is a concatenation operation.• R M = R T denotes the set of the edges connecting nodes in the scene graph.MSSG provides key insights for achieving fine-grained, entitylevel multimodal interactions. However, the heterogeneous nature of MSSG, featuring diverse node types (such as 'dogs' and 'clothes') and relational edges (such as 'above' and 'below'), presents challenges for traditional graph processing techniques, such as graph convolution networks (GCN) [18, 51] or graph attention networks (GAT) [41] . These standard approaches often struggle to capture the diversity of nodes and edges in heterogeneous graphs [35] To achieve fine-grained aggregation in our heterogeneous MSSG G M , we propose utilizing heterogeneous graph attention networks (HGAT) to distinguish specific information from different types of nodes along different relation edges. Specifically, we feed node features 𝐻 𝑖 ∈ E M into HGAT with 𝐾 attention heads to obtain attention scores of edge 𝑟 on the 𝑘-th head as:EQUATIONEQUATIONwhere, || is a concatenation operation, 𝑟 denotes the type of edge from node 𝑗 to node 𝑖; 𝜎 represents the activation function; the set N 𝑖 is the first-order neighbors of node 𝑖 on graph G M ; and a [𝑘,𝑟 ] and 𝑊 [𝑘,𝑟 ] are trainable matrix of 𝑟 on the 𝑘-th head. The core difference between HGAT in Eq. 12 and GAT is that we assign a learnable weight 𝑊 [𝑘,𝑟 ] to each edge, rather than sharing a set of parameters. We can further obtain the enhanced multimodal representation as follows:EQUATION)Then we decouple the enhanced features Ĥ = { Ĥ𝑖 } into modalityspecific representations ℎ T , ℎ I , ℎ A and ℎ V for M-step in Section 3.3.We summarize the whole training process of the EM framework in Algorithm 1. It is worth noting that our method proposes a framework wherein probability estimation can be flexibly adapted to suit a wide range of scenarios through various approaches. Obtain ĥ𝑚 , ∀𝑚 ∈ M via Eq. 14; \\ E-step 7:Update Θ via Eq. 9. \\ M-step 8:end while 9: end for 10: return The optimal parameters Θ * . 
experiments 4.1 experimental setup: 
 Datasets. We conduct main experiments across four types of tasks:(1) Natural Language Understanding (NLU). We evaluate our MASE on six GLUE benchmark [42] , i.e., SST-2 [34] , QNLI [32] , QQP [6] , MNLI [44] , MRPC [10] , and STS-B [5] . (2) Question Answering. In the realm of question answer task, we select CSQA [37] and SQuADv2.0 [31] to evaluate MASE. (3) Text Generation. We utilize the CommonGen [22] as the dataset to validate MASE. (4) Visual Reasoning. We evaluate our MASE on color reasoning (using the MemoryColor [27] and ColorTerm [4] datasets) and shape reasoning (using ObjectShape [50] dataset) tasks. (5) Visual Question Answering. We further evaluate our MASE on cross-modal question-answering tasks on VQA 2.0 benchmark [13] .Baselines. We compare MASE with the pre-trained language models (PLMs), the multimodal proxy based baselines (MPB), multimodal contrastive pre-trained models (MC-PTMs), and visuallyaugmented pre-trained language models (VA-PLMs). ( 1) PLMs: We utilize BERT [9] , RoBERTa [23] , XLNet [49] , and T5 [30] as the backbones, and directly fine-tune them as baselines. (2) MPB: We use the methods described in Section 3.1 as the baseline of multimodal proxy based MA-PLMs. (3) MC-PTMs: CLIP [29] , AudioCLIP [16] and CLIP-ViP [47] are selected as baselines. (4) VA-PLMs: We select VOKEN [38] , iACE [25] , and VAWI [14] as baselines. Implementation Details. We implement all methods2 based on Huggingface Transformers [45] . The number of iterations for the EM algorithm is set to 10. We set the learning rate as 1e-4 on GLUE benchmark, 3e-5 on CSQA and SQuADv2.0 datasets, and 2e-5 on CommonGen. We utilize Adam as the optimizer and train all models for 3 epochs. {𝑃 𝑖 } 𝑛 𝑖=1 is initialized as a uniform distribution, i.e., the initial weight of each modality is 0.25 for four modalities. The experimental details on VQA2.0 are consistent with ConceptBert [11] . More details and experiments can be found in supplements.Evaluation on NLU Tasks. Experimental results on GLUE are shown in Table 1 , from which we have several observations:(1) The integration of additional multimodal information significantly bolsters the predictive capabilities of PLMs in NLU tasks.Table 1 : Comparison of accuracy and learnable parameter size on the GLUE benchmark, with the best results highlighted in bold. "T, I, A, V" indicates four distinct modalities: text, image, audio, and video, respectively. We gradually add different modal semantics to MASE. "+None" denotes we directly fine-tune the base model. The results of VOKEN, iACE and VAWI on GLUE are reported from Lu et al. [25] and Guo et al. [14] . More results on other base models are provided in the supplements. MPB, VA-PLMs and our MASE achieve substantial performance gains consistently across all backbone networks. This indicates that introducing general object knowledge (e.g., color and shape) through additional modalities can markedly improve PLMs.(2) MASE demonstrates superior performance compared to MPB and all VA-PLM baselines when the same number of additional modalities are incorporated. We attribute this to (i) the efficient injection of semantics and the balance of modal information through our MASE framework; (ii) the strategy of incorporating multimodal information extracted through multimodal proxy based methods to avoid the inclusion of noise and modality gaps.(3) Furthermore, MASE surpasses all baselines in terms of PLM performance gains across all backbone networks, which demonstrates the effectiveness and generality of our MASE. Evaluation on Text Generation Tasks. We conduct experiments on the CommonGen dataset as results shown in Table 2 . We use T5-3B as the backbone network and replace the optimization objective of M-step with the text generation objective. We can observe that MASE performs better in all metrics than the base model and retrieved-based methods. This suggests that our method can aggregate multimodal information into PLMs for better text generation.Evaluation on Question Answering Tasks. We present our experimental results on CSQA and SQuAD v2 datasets in Table 3 . For comparison with the retrieval-based methods, we adopt Bing Image Search3 for image retrieval and utilize the CLIP image encoder to extract visual features. We also evaluate different approaches in low-resource settings. Analysis of Table 3 yields several insights:(1) MASE significantly improves performance at low-resource settings (i.e., with only 5% of data available). This demonstrates that MASE can integrate balanced additional information into PLM, effectively countering the adverse effects of data scarcity.(2) We can observe that the performance gain obtained by retrievalbased methods is lower than MASE. Employing our multimodal enhancement proves to be more efficient and low-cost compared to retrieval-based strategies.(3) In the realm of QA, MASE can achieve significant performance gains across a variety of datasets and multiple foundational models, including BERT, RoBERTa, and XLNet. This further validates that MASE offers a twofold advantage: it can introduce additional modal information in a balanced and efficient manner and it is compatible with different architectures and various scenarios. Evaluation on Visual Reasoning Tasks. We utilize color and shape reasoning datasets, i.e., the MemoryColor [27] , ColorTerm [4] , and ObjectShape [50] datasets, for evaluating visual knowledge transfer of our MASE, as results shown in Table 4 . Based on these results, we observe that MASE positively contributes to the understanding of object colors and shapes for PLMs, demonstrating the effectiveness of our approach in enhancing the multimodal comprehension abilities of PLMs. Evaluation on Visual Question Answering Tasks. To verify the effectiveness of our MASE on cross-modal QA, we conduct experiments on the VQA 2.0 benchmark and compare with baselines, as shown in Figure 5 . We can observe significant VQA performance gains brought about by our MASE, e.g., +1.6% and +2.3% overall accuracy using ViLBERT and ConceptBert as base methods. This validates that our MASE can inject multimodal semantics into PLMs and improve their multimodal understanding ability, which significantly improves the performance in the cross-modal visual question-answering task.The Effect of the Cross-modal Contrastive Loss in M-step. We conduct experiments to study the cross-modal contrastive loss of MASE in M-step as illustrated in Table 6 . It can be seen that the performance of MASE significantly decreases without the crossmodal contrastive loss to improve the information aggregation. This verifies that (1) the optimization algorithm of MASE in M-step is capable of injecting additional modality information sufficiently and effectively. (2) the interaction of different modalities plays a significant role in enhancing the expressive capabilities of PLMs.The Effect of the Information-driven Self-balancing in Estep. We employ BERT-base and RoBERTa-base as the backbones and conduct experiments to study the impact of our informationdriven self-balancing strategy in E-step. Observations indicate that the absence of our information-driven self-balancing strategy significantly diminishes performance. This suggests that our informationdriven self-balancing strategy can dynamically estimate the contribution of different modalities and is effective and balanced in transferring multimodal knowledge to PLMs.Combined with Prompt Learning. We combined MASE with prompt learning [36] to evaluate the effectiveness of MASE under the parameter-efficient fine-tuning approach. As shown in Table 6 , we can observe significant performance gains from MASE compared to the MPB baseline using the prompt learning strategy. This shows that MASE can be effectively compatible with existing efficient fine-tuning methods to improve the performance of PLMs. This experiment further demonstrates the great potential of MASE in practical scenarios.To study the effect of our EM-based iteration, we visualize the changes in the latent variable during each E-step of the training process on CSQA and SQuAD v2 datasets. We use RoBERT-base as the base model. Specifically, we report the latent variables averaged over four different modalities as shown in Figure 4 , and observe several important phenomena:(1) The contribution of each modality to the task varies. We can observe that the weights of different modalities are different on CSQA and SST-2 datasets. This indicates that optimizing the balance of modal information is crucial for improving the multimodal understanding ability of PLMs.(2) The weight of information allocation will undergo dynamic changes during the iteration stage. In the early stage of EM iteration, there is a significant change in the allocation of weights, which is due to insufficient optimization of model parameters in the early stage. After the model parameters are fully optimized, the rate of change of the latent variables becomes gentle, which proves that the modal allocation has reached equilibrium.  
conclusion: 
 In this paper, we present a novel principled self-balancing probabilistic framework, MASE, designed for injecting multimodal semantics into PLMs. MASE assigns weights to modal information as latent variables and adopts an EM-based iterative algorithm to iteratively optimize the two objectives of multimodal information injection and balance. Furthermore, we propose a novel MI-driven allocation estimation method and achieve entity-level multimodal interactions via MSSG. The experimental results show that MASE can effectively improve the performance of PLMs on multiple tasks, and is compatible with efficient fine-tuning strategies. 
 
 Appendix: All the source code and models will be released after review.https://learn.microsoft.com/en-us/azure/cognitive-services/bing-imagesearch/overview 


Decision: This work proposes the MASE framework, an iterative-based approach for multimodal-augmented pre-trained language models, leveraging Expectation-Maximization to balance the introduction of multimodal semantics, in a principled manner. 

Overall, after the rebuttal, the paper got three positive reviews, having received two Weak Accepts and one Borderline Accept.

Reviewers unanimously agree on the novelty of the MASE framework and on the extensive experimental evaluation carried out by the authors. 

Among its weaknesses, reviewers mentioned the brief related work section (uQua), followed by some ablation studies that are missing. Namely, whether certain modalities could have a bigger benefit than others (CzTF), and variants of the MSSG graph (zjcx). 

In summary, while the paper could be improved with further ablation studies, reviewers recognize that the conducted experiments validate the proposed approach. Moreover, reviewers recognize the principled nature of the proposed Expectation-Maximization approach, in jointly addressing inter-modality gaps, noise, and entity-level semantics. Therefore, I suggest this work to be accepted as Poster.

Reviews: review: paper_summary: paper_summary: This paper presents MASE, a novel framework for enhancing pre-trained language models (PLMs) with multimodal information. MASE addresses issues like modality gaps, noise, and entity-level semantics by using an Expectation Maximization (EM) algorithm and multimodal proxies. It integrates a mutual information-based allocation strategy and constructs a multimodal-semantic scene graph (MSSG) for better entity-level understanding. Experiments show MASE's effectiveness across multiple tasks and its compatibility with existing fine-tuning methods.
strength_weakness: strength_weakness: 1. The introduction of MASE presents a significant advancement in integrating multimodal information into PLMs, addressing key issues like modality gaps and noise with a principled EM-based iterative framework.

2. The use of multimodal proxies, mutual information-based allocation estimation, and multimodal-semantic scene graphs are innovative methods that enhance the understanding and performance of PLMs.

3. Extensive experiments on thirteen datasets across five tasks validate the effectiveness and versatility of MASE, showing its consistent outperformance against competitive baselines.
limitations: limitations: 1. Could the implementation of the EM-based iterative framework and multimodal-semantic scene graphs be computationally intensive? How does this affect the scalability of the method to very large datasets or real-time applications?

2. While the framework aims for balanced multimodal integration, could certain modalities benefit PLMs more significantly than others? Have the authors analyzed the impact of individual modalities on performance? What are the results for T+A and T+V combinations in Tables 1 and 3?
paper_summary: paper_summary: This paper presents MASE, an innovative iterative framework for multimodal-augmented pre-trained language models (PLMs) to overcome the limitations of existing methods that face modality gaps, noise, and neglect of key visual or acoustic semantics. Experimental results show MASE significantly outperforms baselines across various tasks and architectures and is compatible with efficient parameter fine-tuning methods like prompt learning.
strength_weakness: strength_weakness: - MASE introduces a novel iterative framework utilizing an EM-based algorithm to balance and efficiently inject multimodal semantics into PLMs, which is the first theoretical framework specifically designed for multimodal-augmented PLMs.
- The paper proposes a mutual information-based information allocation estimation method and an entity-level multimodal aggregation method via the multimodal-semantic scene graph (MSSG) to enhance modal semantics. The method is introduced in great detail, providing clear explanations of each component and process within the MASE framework.
- Extensive experiments on five tasks, including NLP, visual reasoning, and visual question answering, demonstrate MASE's effectiveness and generality. The paper includes a wide range of experiments, thoroughly validating the proposed method's performance and robustness across multiple tasks and scenarios.
limitations: limitations: - The related work section of the paper is relatively brief, potentially lacking in comprehensive coverage of existing methods and approaches in the field.
- The paper does not explicitly address the performance of the proposed method, MASE, on widely used settings such as BERT-large and RoBERTa-large. This omission leaves a gap in understanding how well MASE scales and adapts to these larger, more commonly used models.
- Typo:  "balance and efficiently" in 177 line, "Figure 5" -> "Table 5" in 749 line, "+1.6%" -> "+0.6%" in 749 line
paper_summary: paper_summary: This paper introduces a principled self-balancing probabilistic framework called MASE, designed to infuse multimodal semantics into PLMs. Utilizing latent variables, MASE assigns weights to modal information and employs an EM-based iterative algorithm to optimize the two objectives of multimodal information infusion and balancing. Additionally, this paper suggests an MI-driven allocation estimation method to accomplish entity-level multimodal interactions through MSSG. Ultimately, the authors conduct numerous experiments, thereby validating the effectiveness of the proposed framework.
strength_weakness: strength_weakness: 1.The MASE framework proposed in this paper is novel.
2.The authors conduct extensive experiments to demonstrate that MASE can effectively improve the performance of PLMs on multiple tasks.
3.The logic of this paper is relatively smooth.
limitations: limitations: 1. The authors concatenate multimodal features with entity nodes in the vanilla scene graph to construct MSSG. However, in the ablation study, experiments involving the removal of entity nodes are lacking.
2. It is suggested that the font format in Figure 3 be unified.