Modality-Balanced Learning for Multimedia Recommendation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-29 13:20:43

Abstract: Multimedia content is of predominance in the modern Web era. Many recommender models have been proposed to investigate how to incorporate multimodal content information into traditional collaborative filtering framework effectively. The use of multimodal information is expected to provide more comprehensive information and lead to superior performance. However, the integration of multiple modalities often encounters the modal imbalance problem: since the information in different modalities is unbalanced, optimizing the same objective across all modalities leads to the under-optimization problem of the weak modalities with a slower convergence rate or lower performance. Even worse, we find that in multimodal recommendation models, all modalities suffer from the problem of insufficient optimization. 

To address these issues, we propose a Counterfactual Knowledge Distillation (CKD) method which could solve the imbalance problem and make the best use of all modalities. Through modality-specific knowledge distillation, CKD could guide the multimodal model to learn modality-specific knowledge from uni-modal teachers. We also design a novel generic-and-specific distillation loss to guide the multimodal student to learn wider-and-deeper knowledge from teachers.  Additionally, to adaptively recalibrate the focus of the multimodal model towards weaker modalities during training, we estimate the causal effect of each modality on the training objective using counterfactual inference techniques, through which we could determine the weak modalities, quantify the imbalance degree and re-weight the distillation loss accordingly. 

Our method could serve as a plug-and-play module for both late-fusion and early-fusion backbones. Extensive experiments on six backbones show that our proposed method can improve the performance by a large margin.

Main paper: 
 preliminaries: 
 In this section, we first introduce the notations and formulate the multimedia recommendation task. Then, to motivate our model design, we conduct simple and intuitive experiments to show that multimodal models suffer from the imbalance problem. Finally, we analyze the imbalance problem from the optimization perspective.We use the notation U and I to represent the sets of users and items, respectively. The number of items is denoted as 𝑛 = |I|. Each user 𝑢 ∈ U has interacted with some items I 𝑢 , indicating that the preference score 𝑦 𝑢𝑖 = 1 for 𝑖 ∈ I 𝑢 . We represent the input ID embedding of user 𝑢 and item 𝑖 as 𝒙 𝑢 and 𝒙 𝑖 , respectively, where 𝒙 𝑢 , 𝒙 𝑖 ∈ R 𝑑 and 𝑑 is the embedding dimension.In addition to user-item interactions, each item is associated with multimodal content information. We represent the feature vector of item 𝑖 and the preference vector of user 𝑢 for modality 𝑚 as 𝒆 𝑚 𝑖 ∈ R 𝑑 𝑚 and 𝒑 𝑚 𝑢 ∈ R 𝑑 , where 𝑑 𝑚 represents the dimension of the features and 𝑚 ∈ M represents the modality. For example, the visual and textual modalities are represented by 𝑚 = 𝑣 and 𝑚 = 𝑡, respectively. It should be noted that our method is not limited to two modalities and can accommodate multiple modalities. The goal of multimedia recommendation approach is to predict users' preferences ŷ𝑢𝑖 accurately by considering both user-item interactions and item multimodal content information.In this subsection, we introduce the general framework of multimedia recommender models. Our proposed method can serve as a flexible plug-and-play module for any multimedia recommender backbones. For multimedia recommender models, the calculation of the preference score of user 𝑢 on item 𝑖 can be generalized as:EQUATIONwhere 𝑓 Θ (•) represents different methods to model user-item interactions and item multimodal content information. Θ are the trainable parameters of the models. Multimedia recommender models adopt Bayesian Personalized Ranking (BPR) loss [27] to conduct the pair-wise ranking, which encourages the prediction of an observed entry to be higher than its unobserved counterparts: EQUATIONWe conduct a pilot study on Amazon-Clothing datasets [21] to show that both late-fusion method VBPR [11] and early-fusion method EgoGCN [1] suffer from the modality imbalance problem and are not fully optimized. We show the experimental performances of multimodal model and uni-modal model (image-only and text-only models) with the same architecture. During the training phase, for multimodal models, we trained it using both textual and visual input.For uni-modal models, we ablate the input of the other modality with the average feature vector (this keeps the other modality indistribution, but renders them uninformative). During the inference phase, for multimodal models, we report both multimodal performances and uni-modal performances (dashed lines in Figure 1 ) that are obtained by ablating the input of other modality. The experimental results are shown in Figure 1 (performance comparison of all backbones and datasets is shown in Table 3 ), and note that all methods utilize early stopping which results in the different length of curves. There are three phenomena regarding the modality imbalance problem. 1) The performance of visual and textual modality within them is worse than that of their corresponding uni-modal models, which suggests that both modalities are under-optimized and far from reaching the upper bound of their capabilities. 2) Even worse, the text-only VBPR outperforms the multi-modal VBPR. It shows that in the multi-modal joint training process, there is a strong mutual inhibition phenomenon between the modalities, resulting in 1 + 1 < 2. 3) Weak modals with poor performance (visual modality in the example) suffer from more serious under-optimization problems.In this subsection, we introduce the analysis of the modality imbalance problem from the optimization perspective. We show that in the multimodal joint training, the dominant modality which converges faster and has better performance would reduce the updating step and thus suppress the optimization of other modalities.Taking VBPR [11] as an example, it simply concatenates different modalities and we can formulate the representations of users and items:EQUATIONand we can re-formulate the score function as:EQUATIONand we can also re-write the BPR loss in equation 2. For convenience, we only consider the BPR loss of one training triple (𝑢, 𝑖, 𝑗): EQUATIONEQUATIONwhere 𝜂 is the learning rate. According to equation 5, the gradient𝜕 L 𝑢𝑖 𝑗 𝜕𝑆 𝑚 𝑢𝑖 𝑗can be written as:EQUATIONWe can find that for any two modalities 𝑚 1 and 𝑚 2 , we have𝜕 L 𝑢𝑖 𝑗 𝜕𝑆 𝑚 1 𝑢𝑖 𝑗 = 𝜕 L 𝑢𝑖 𝑗 𝜕𝑆 𝑚 2 𝑢𝑖 𝑗and thus this term forms a "brigde" between the optimization of different modalities. We can infer that if one modality 𝑚 1 works particularly well with superior performance, it will result in a large 𝑆 𝑚 1 𝑢𝑖 𝑗 , that is, a large denominator in Equation 7, thereby resulting in a smaller𝜕 L 𝑢𝑖 𝑗 𝜕𝑆 𝑚 𝑢𝑖 𝑗. According to equation 6, the other modalities could obtain a smaller updating step. As a result, when the training of the multimodal model is about to converge, the other modality could still suffer from under-optimized representation and need further training. 
the proposed method: 
 Motivated by the above analysis, in this section, we present our proposed method CKD to alleviate the imbalance problem and make each modality fully optimized. The overall framework is shown in Figure 2 . Firstly, we introduce our modality-specific knowledge distillation framework to minimize the gap between uni-modal teachers and multi-modal students. We design a generic-to-specific distillation loss to help the student model learn deeper-and-wider knowledge from teachers. Finally, inspired by the counterfactual inference techniques, we propose to quantify the degree of imbalance by estimating the causal effect of each modality, through which we can adaptively re-weight the distillation losses and thus guide the student model to focus more on the weak modalities with the under-optimization problem.CKD proposes to utilize uni-modal models as teachers to guide the multimodal student to avoid the imbalance problem during training. Due to the different modalities inputted, the uni-modal and multimodal models may significantly differ in their outputs. To ease the gap between teachers and students, we propose modality-specific knowledge distillation by dividing the multimodal model into unimodal channels and only transferring modality-specific knowledge from the corresponding teacher model.We first introduce how we train unimodal teacher models. A direct solution is constructing specific uni-modal models. For example, we can use the most powerful uni-modal models as teachers which will undoubtedly lead to a very powerful multimodal student model. However, it is unfair for the performance comparison. In this paper, we aim to solve the imbalance problem, so we only consider training uni-modal teachers with the same model architecture as the multimodal student models, without introducing additional powerful models.To make CKD model-agnostic which could serve as a play-andplug module for any existing multimedia recommendation backbones, we treat backbone models as black-boxes and train unimodal teachers by only ablating the input. Specifically, when training a teacher of modality 𝑚 1 , we ablate the input of other modalities M -{𝑚 1 } by replacing it with its average feature vector, which could keep these modalities in-distribution, but renders them uninformative. Assuming that the backbone multimodal is represented by 𝑓 (•), we can formulate the prediction of uni-modal teacher of modality 𝑚 1 according to equation 1 as:EQUATIONwhere we ablate the input of other modalities by with the average feature vector: p𝑚 = uni-modal teachers: minEQUATIONSimilarly, we can obtain the uni-modal teacher Θ 𝑚 for each modality.Student. Since we aim to only transfer modalityspecific knowledge from uni-modal teachers to the multi-modal student, we also ablate the input of the multi-modal model to obtain the modality-specific prediction:EQUATIONWe then employ a prediction-level distillation paradigm which provides a more compact representation of the teacher model's knowledge, allowing for efficient transfer without requiring the entire feature space. We design two distillation losses: (1) specific distillation loss to transfer the specific knowledge conveyed by the triples in the training set, and (2) generic distillation loss to transfer the deeper dark knowledge conveyed by the universal triples and achieve better generalization. 𝑢𝑖 -t𝑚 𝑢 𝑗 and Δ𝑠 𝑚 𝑢𝑖 𝑗 = ŝ𝑚 𝑢𝑖 -ŝ𝑚 𝑢 𝑗 , which could capture the relative user preferences. Furthermore, we hope that the student model not only imitates the teacher model but could surpass the teacher model by modeling the interaction between modalities. To this end, instead of utilizing the common-used KL-divergence or Mean Squared Error loss function, we employ hinge loss to guide the student to make more informed predictions on the training triples. Hinge loss encourages the student model to perform better than the uni-model teacher on the training triple (𝑢, 𝑖, 𝑗), i.e, Δ𝑠 𝑚 𝑢𝑖 𝑗 > Δ𝑡 𝑚 𝑢𝑖 𝑗 , instead of just imitating it, i.e, Δ𝑠 𝑚 𝑢𝑖 𝑗 = Δ𝑡 𝑚 𝑢𝑖 𝑗 . Specifically, for modality 𝑚 1 , the specific distillation loss can be formulated as:EQUATIONwhere D = {(𝑢, 𝑖, 𝑗)|𝑖 ∈ I 𝑢 , 𝑗 ∉ I 𝑢 } denotes the triples in the training set. (𝑢, 𝑖, 𝑗) denotes the pairwise training triples where 𝑖 ∈ I 𝑢 is the positive item and 𝑗 ∉ I 𝑢 is the negative item sampled from unobserved interactions.3.1.4 Generic Distillation Loss. In addition to the explicit knowledge implied by training triples, CKD also proposes generic distillation loss to learn the wider knowledge from teacher models and achieve better generalization. Unlike specific distillation which pairs each user with one positive item and one negative item, generic distillation aims to transfer knowledge about (𝑢, 𝑗, 𝑘) where 𝑣 𝑗 and 𝑣 𝑘 are uniformly sampled from the item set and not paired with the user 𝑢. Since 𝑣 𝑗 and 𝑣 𝑘 are not fixed to be positive and negative, (𝑢, 𝑗, 𝑘) does not convey supervision signals and thus we cannot hope student model to perform better than teachers following equation 11. Therefore, following [40] , we optimize the cross entropy-based distillation loss to align them:EQUATIONwhere Δ t𝑚 𝑢 𝑗𝑘 and Δs 𝑚 𝑢 𝑗𝑘 are ranking results processed by sigmoid function 𝜎 (•) with temperature 𝜏:EQUATIONthe overall distillation loss of modality 𝑚 can be formulated:EQUATIONwhere 𝜆 𝑔 is the hyper-parameter that balances the two distillation losses. Through specific-to-generic distillation, the multi-modal student can learn both deeper-and-wider knowledge from uni-modal teachers.In this subsection, we introduce counterfactual conditional learning speed to estimate the causal effect of each modality on the jointtraining objective, through which we could quantify the imbalance degree and adaptively re-weight the distillation loss of different modalities and thus guide the student model to focus more on the weak modalities with the under-optimization problem.Since the multimodal model is trained jointly and thus the parameters are learned based on multimodal inputs, we cannot directly use the trained parameters to infer the model's casual behavior under a specific modality, especially for early-fusion methods. The counterfactual inference [28] is to answer the counterfactual question based on factual observations. For example, when we estimate the contribution of visual modality, we want to answer the question "How much improvement can the introduction of visual modality bring when other modalities remain unchanged?"From a causal perspective, we can define the outcome, denoted as 𝑍 , as the prediction of pairwise rankings Δ𝑠 𝑢𝑖 𝑗 . The treatment, labeled as 𝑇 , pertains to the incorporation of visual modality input. Specifically, we assign 𝑇 = 0 to signify the absence of visual modality input and 𝑇 = 1 to denote its inclusion. 𝑍 (𝑇 = 1) represents the hypothetical outcome that would be observed if all modalities were incorporated. Conversely, 𝑍 (𝑇 = 0) represents the hypothetical outcome if all other modalities were held constant while excluding the visual modality input. By contrasting the potential outcomes 𝑍 (𝑇 = 1) and 𝑍 (𝑇 = 0), we can assess the causal impact of integrating visual modality input on the learning objective. This comparative analysis enables us to comprehend the treatment's influence and ascertain whether the inclusion of visual modality input yields any notable changes in the predicted outcome. The individual treatment effect (ITE) 𝛿 𝑚 (𝑢,𝑖,𝑗 ) of triple (𝑢, 𝑖, 𝑗) can be formulated:EQUATIONwhere Δ𝑠 m (𝑢,𝑖,𝑗 ) denotes the ranking prediction where the input modality 𝑚 is masked with its average feature vector. We can obtain the Average Treatment Effect (ATE) as the casual contribution by taking an average over the ITEs: EQUATIONEQUATION)where a lower 𝜌 𝑚 indicates that the modality 𝑚 is under-optimized and suffers more from the imbalance problem. We propose to reweight the distillation loss of different modalities according to:EQUATIONwhere we emphasize the distillation loss of modalities with lower 𝜌. In this way, the overall optimization objective is presented:EQUATION)where 𝜆 𝑘𝑑 is the hyper-parameter that balances the learning rate between BPR loss and distillation losses. 
experiments: 
 In this section, we conduct experiments on four widely used realworld datasets. We first describe the experimental settings, including datasets, baselines, evaluation, and implementation details. Then, we report and discuss the experimental results to answer the following research questions:• RQ1: How does CKD perform when plugged into different multimedia recommendation methods? • RQ2: Does CKD achieve balanced multimodal learning? • RQ3: How do different components impact CKD 's performance Some other experiments such as hyper-parameter sensitive analysis are shown in supplementary material. R@20 NDCG@20 P@20 R@20 NDCG@20 P@20 R@20 NDCG@20 P@20 R@20 NDCG@20 P@20 1 . Amazon dataset includes visual modality and textual modality. The 4,096-dimensional visual features have been extracted and published 1 . We also extract sentence embeddings as textual features by concatenating the title, descriptions, categories, and brand of each item and then utilize sentence-transformers [26] to obtain 1,024-dimensional sentence embeddings.To evaluate the effectiveness of our proposed model, we plug it into several state-of-the-art recommendation models, including late-fusion methods VBPR [11] , DeepStyle [18] , MMGCN [37] , GRCN [36] and early-fusion methods EgoGCN [1] and MGCN [42] .4.1.3 Baseline Models. We also conduct experiments on two stateof-the-art baseline methods that aim to solve the imbalance problem:1 http://jmcauley.ucsd.edu/data/amazon/links.html• GB [32] optimizes both uni-modal and multimodal losses simultaneously, and reweights the losses according to the overfitting-to-generalization-ratio. • OGM [24] alleviates the optimization imbalance problem with on-the-fly gradient modulation to adaptively control the optimization of each modality.Both the baselines require an explicit distinction between the parameters of the different modalities and thus are only applied to late-fusion models. For early-fusion method EgoGCN and MGCN, we implement them by roughly dividing the parameters into two parts.For each dataset, we select 80% of the historical interactions of each user to constitute the training set, 10% for validation, and the remaining 10% for the test set. We adopt three widely used metrics to evaluate the performance of preference ranking: Recall@𝑘, NDCG@𝑘, and Precision@𝑘. By default, we set 𝑘 = 20 and report the averaged metrics for all users in the test set. We implemented our method in PyTorch and the embedding dimension 𝑑 is fixed to 64 for all models to ensure fair comparison. The optimal hyper-parameters were determined via grid search on the validation set: the learning rate is tuned amongst {0.0001, 0.0005, 0.001, 0.005}, the coefficient of 𝐿 2 normalization is searched in {1e-5, 1e-4, 1e-3, 1e-2}, the dropout ratio in {0.0, 0.1, • • • , 0.8}, the 𝜆 𝑔 in {0.1, 0.5, 1, 5} and 𝜆 𝑘𝑑 in {1e-3, 1e-2, 1e-1, 5e-1}. We set the temperature 𝜏 = 0.1 in Equation 13. Besides, we stop training if recall@20 on the validation set does not increase for 10 successive epochs.We start by conducting experiments to evaluate the overall performance when baselines and our method are plugged into different backbone models. The results are reported in Table 2 , from which we have the following observations:• Our method CKD significantly outperforms baselines when plugged into different backbone models, verifying the effectiveness of our methods. Specifically, our method improves over the backbone multimedia recommendation models by 18.6%, 20.1%, 20.9%, and 15.7% in terms of Recall@20 on average on Baby, Sports, Clothing and Beauty, respectively. This indicates that our proposed method successfully mitigates the modality imbalance problem, leading to enhanced optimization of parameters for each modality in the multi-modal backbone model. Consequently, the overall recommendation performance experiences a substantial boost. • When applied to the early-fusion backbone model EgoGCN, both GB and OGM do not yield significant improvements. In contrast, our approach not only achieves performance gains in simple late-fusion models but also demonstrates improvements in complex early-fusion models. By treating the backbone model as a black box and only applying ablating at the input end and distillation at the output end, our method is model-agnostic, offering greater flexibility.4.3.1 Uni-modal Performance Comparison. In this subsection, we show the experimental results of each individual modality to verify the effectiveness of our method in alleviating the modality imbalance problem. We report the uni-modal performances within each multimodal backbone model, the performances of uni-modal teachers, and the uni-modal performances within baselines and our proposed method. The experimental results are shown in Table 3 , from which we have the following observations:• As the phenomenon in Figure 1 , the uni-modal channels within multimodal backbones exhibit sub-optimal performance in comparison to their respective uni-modal teachers. This observation suggests that despite the intended capability of multimodal models to capture complementary information across modalities, they fall short of fully harnessing the potential inherent in each modality. Consequently, both the overall performance and the performance of individual uni-modal channels fail to attain their maximum potential. • While baseline methods have shown some enhancement in uni-modal performance, they still fall significantly short of reaching their maximum potential. This is evident from the substantial performance disparity between baselines and uni-modal teacher models. • The uni-modal channels within CKD significantly outperform those within the vanilla backbone models, especially for the modalities that are optimized insufficiently in joint multimodal training. For example, when plugged into VBPR, our method improves 28.7% and 64.3% for textual modality and visual modality, respectively. Through knowledge distillation, we can make up for the under-optimization of the visual modality in joint multimodal training, thereby greatly improving the information utilization rate of the visual modality and the overall recommendation performance. • Our proposed CKD also outperforms the uni-modal teachers in most cases. Through generic and specific distillation losses, our knowledge distillation framework does not simply imitate uni-modal teachers, but improves the information utilization rate of each modality under their guidance, captures complementary information between modalities and thus can obtain recommendation performance beyond them.Training. We additionally illustrate the performance curves during the training process, as depicted in Figure 3 , akin to the representation in Figure 1 . It is discernible that both the overall multimodal performance and unimodal performances of our proposed methods surpass that of the backbone models. This observation serves to substantiate the efficacy of CKD.To further analyze CKD, in this subsection, we conduct component analyses by repeatedly assessing and comparing the models after removing each component. Specifically, we design the following variants of CKD:• w/o Gen. removes the generic distillation loss and only retains the specific distillation loss. • w/o re-weight removes the counterfactual conditional learning speed loss re-weight. • repl. KL-divergence/MSE Loss replaces the hinge loss with KL-diverge or MSE loss in the specific distillation module.Due to limited space, we only present the results of VBPR [11] , MMGCN [37] and EgoGCN [1] in Table 4 and leave others in supplementary material. We can observe that each of these key components contributes substantially. The generic distillation considers more general triplets in addition to traditional triplets with positive and negative items, which greatly facilitates the deeper dark knowledge distillation. Additionally, the counterfactual conditional learning speed can monitor the discrepancy of each unimodal's contribution to the overall learning objective, which could reflect the degree of imbalance in the training process. By dynamically increasing the distillation weight of weak modalities, the optimization process of each modality is controlled and the imbalance problem can be alleviated. Finally, compared to KL-divergence and MSE losses, our proposed hinge loss could encourage the student to make more informed predictions. 
related work 5.1 multimedia recommendation: 
 Recommendation systems have achieved significant success through the implementation of Collaborative Filtering (CF) methods. However, they face challenges in handling sparse data characterized by limited user-item interactions, as their predictive models traditionally depend on substantial interactions to generate accurate recommendations based on user preferences. To overcome this obstacle, recent researchers have incorporated diverse types of auxiliary information, such as visual images, textual descriptions, and [5, 6, 38] . This integration has given rise to multimedia recommendation systems, leveraging extensive multimedia content details associated with items [2-4, 8-11, 13-20, 22, 23, 25, 29, 33-37, 41, 43-51, 53] . Traditional research in multimedia recommendation [2, 3, 11, 13, 18, 47] extends the basic CF framework by incorporating multimodal content alongside item representations. More recently, there is a growing interest in GNN-based multimedia recommendation systems [15, 19, 20, 36, 37, 43] and integrating contrastive learning [10, 25, 33, 35, 41, 44, 45] .Multimodal models, though expected to outperform single-modal ones by incorporating more information, have shown counterintuitive results in previous experiments, sometimes performing even worse [24, 30, 32] . The heterogeneity of modalities poses a challenge for unified training strategies to fully exploit their potential. Recent approaches aim to address this by balancing optimization across modalities [7, 24, 30, 32, 39] . Wang  
conclusion: 
 In this paper, we have proposed CKD which aims to solve the modal imbalance problem and make the best use of all modalities, which could easily serve as a plug-and-play module for any existing multimedia recommendation models. Extensive experiments on four real-world datasets and six state-of-the-art multimedia recommendation backbones have been conducted to demonstrate that CKD achieves superior performance.  
: 
 represented in multiple modalities, including images, texts, videos, etc. Multimedia recommendation has garnered increasing attention from researchers in recent years. Many early researches [11, 18, 31, 37, 43] follow late-fusion paradigm, which models user preferences towards each modality individually and then fuse them through summation or concatenation. Recent researchers [1, 12, 44, 52] try to adopt early-fusion which could model fine-grained inter-modal interactions and achieve better performance. For example, EgoGCN [1] performs edge-wise modulation fusion, adaptively distilling the most informative inter-modal messages to spread while preserving intra-modal processing. The use of multimodal information is expected to provide a more comprehensive understanding of user preferences and subsequently yielding superior recommendation performance. However, recent researches [24, 32] find that optimizing the same objective for different modalities leads to the under-optimization problem of the weak modalities with slower converge rate or lower performance, which is named as modal imbalance problem. In multimedia recommendation, this phenomenon still exists and is even more pronounced since the information contained in different modalities is highly unbalanced. Taking the e-commerce scenario of the Amazon dataset [21] as an example, since the textual modality contains more detailed information such as the titles, categories, and descriptions of items, it converges faster with better performance compared with the visual modality [44] . This gap leads to significant underoptimization issues in multimodal recommendation models. We conduct a pilot study to validate this. Based on Figure 1 (please refer to Section 2.3 for more experiment details and findings), we could find that both visual and textual performance within the multimodal model is worse than that in the uni-modal models, i.e., visual-only and textual-only model, respectively.To this end, recent researches [24, 32] try to modulate the learning speed of different modalities. However, these methods require an explicit distinction between the parameters of the different models and thus are only applied to late-fusion models. In this work, we propose Counterfactual Knowledge Distillation framework, CKD for brevity, to solve the imbalance problem and make the best use of all modalities for both late-fusion and early-fusion models. The overall framework of our proposed CKD is shown in Figure 2 . (1) We first utilize uni-modal models as teachers to guide the multimodal student through modality-specific knowledge distillation.(2) Secondly, we design generic-and-specific distillation losses to guide the multimodal student models to learn wider-and-deeper knowledge about both universal and training triples from teachers.(3) Finally, to adaptively focus more on weaker modalities with slow converge rate, we employ counterfactual inference techniques to estimate the causal effect of each modality on the training objective, through which we could determine the weak modalities, quantify the imbalance degree and re-weight the distillation loss accordingly. Since the above operations only involve the input and output of the backbone models, CKD treats them as black-boxes, which is model-agnostic and could be served as a plug-and-play module for any existing multimedia recommendation backbones.We conduct extensive experiments to verify the effectiveness of our proposed method on four public real-world recommendation datasets. Experimental results demonstrate that our proposed CKD gains significant improvements when plugged into six state-of-theart models.To summarize, the main contribution of this work is threefold:• We argue that the existing multimodal recommendation models suffer severely from the modality imbalance problem and all modalities are under-optimized and far from reaching the upper bound of their capabilities. • We propose a novel Counterfactual Knowledge Distillation framework which could serve as a plug-and-play module for any existing multimedia recommendation backbones to solve the imbalance problem and make the best use of all modalities. • We perform extensive experiments on four public datasets when plugged into six backbones. The empirical results validate the effectiveness of our proposed model. 
 
 Appendix:  


Decision: This paper proposed a novel Counterfactual Knowledge Distillation (CKD) method to address the modal imbalance problem. Based on the reviewers’ comments, the motivation is clear, and the paper is well-organized. However, there are still areas that need improvement. I recommend that the authors revise their paper according to the reviewers’ comments. At this time, I lean towards acceptance.

Reviews: review: paper_summary: paper_summary: This paper introduces the Counterfactual Knowledge Distillation(CKD) method, addressing insufficient optimization of multimodal representation in multimedia recommendation. It incorporates modality-specific knowledge distillation and causal inference into multimodal representation learning. Extensive experiments across four datasets showcase the effectiveness of CKD in multimedia recommendation.
strength_weakness: strength_weakness: 1. This paper employs modality-specific knowledge distillation to enhance multimodal representation. Moreover, it introduces counterfactual inference to adaptively re-weight the distillation loss across different modalities.
2. The paper is well-organized.
3. The evaluation utilizes various datasets and models.
limitations: limitations: 1. As shown in section 2.4, does parallel multimodal representation learning also lead to this problem?

2. More baselines need to be included such as BM3[1], SLMRec[2], and so on.

> [1] Xin Zhou, Hongyu Zhou, Yong Liu, Zhiwei Zeng, Chunyan Miao, Pengwei Wang, Yuan You, and Feijun Jiang. 2023. Bootstrap latent representations for multi-modal recommendation. In Proceedings of the ACM Web Conference 2023. 845–854  
[2] Zhulin Tao, Xiaohao Liu, Yewei Xia, Xiang Wang, Lifang Yang, Xianglin Huang, and Tat-SengChua. 2023. Self-Supervised Learning for Multimedia Recommendation.  IEEE Trans. Multim. 25(2023), 5107–5116.

3. More detailed analysis in Section 4.2 is needed. For instance, why does the performance improvement vary across different models?

4. The conclusions presented in this paper are insufficient to provide comprehensive and effective information.

5. The authors should clarify why only VBPR and EgoGCN are utilized in Section 4.3.2. Are the performances of other baselines comparable?
paper_summary: paper_summary: The paper introduces a novel framework named Counterfactual Knowledge Distillation (CKD) that addresses the modal imbalance problem in multimedia recommendation systems. CKD leverages uni-modal models as teachers to guide a multimodal student model through a knowledge distillation process, enhancing the optimization of underperforming modalities. By employing counterfactual inference techniques, CKD also estimates the causal effect of each modality, enabling an adaptive focus on weaker modalities and significantly improving recommendation performance across various datasets and backbone models. The proposed method serves as an effective plug-and-play module to balance multimodal learning and boost the overall recommendation quality.
strength_weakness: strength_weakness: 1. The motivation is clear and the idea is easy to follow;
2. This paper is written clearly, and the purpose of each section is evident;
3. This paper conducted a rich set of experiments and provided the main baselines. Moreover, the method provided have shown significant improvement compared to the baselines；
limitations: limitations: 1. Figure 1 in the supplementary materials demonstrates that the choice of hyperparameters significantly impacts the results, and the optimal sets of hyperparameters vary across different methods, which to some extent limits the practicality of the approaches;
2. I would suggest providing a comparison of hyperparameter tuning for a same backbone (e.g., EgoGCN) across different datasets;
paper_summary: paper_summary: The paper proposes Counterfactual Knowledge Distillation framework, which could serve as a plug-and-play module for existing multimedia recommendation backbones and consists of three main components: (1) utilizes uni-modal models as teachers to guide the multi-modal student through modality-specific knowledge distillation. (2) designs generic-and-specific distillation losses to guide the multi-modal student models to learn wider-and-deeper knowledge about both universal and training triples from teachers. (3) employs counterfactual inference techniques to estimate the causal effect of each modality on the training objective.
strength_weakness: strength_weakness: (1)The paper provides a thorough analysis of the existing modality imbalance problem. 
(2)The paper introduces a plug-and-play module for existing multimedia recommendation backbones to solve the imbalance problem.
(3)The paper conducts rigorous experiments to prove the validity of the proposed framework.
limitations: limitations: I think the paper is well-considered.
paper_summary: paper_summary: This paper proposes a counterfactual knowledge distillation method, termed CKD, to address the issue of imbalanced modalities in multimedia recommendation. Specifically, this method distills uni-modal knowledge from teachers to the student and leverages counterfactual inference to identify weak modalities for distillation loss re-weighting.
strength_weakness: strength_weakness: 1\. The problem of imbalanced modalities is well elucidated, supported by a pilot study.

2\. The method is well-structured, facilitating ease of reading.

3\. The overall performance improvement is significant.
limitations: limitations: 1\. The pilot analysis in Section 2.4 is insufficient and misleading. Despite the same gradient for $S^m_{uij}$, the gradient for the trainable parameters differs due to the different values of $e^m_i - e^m_j$. Moreover, only the VBPR method is considered.

2\. The different loss formats for generic distillation loss are unclear. The core difference between specific and generic distillation loss is the sample methods of triples. It is unclear why a different cross-entropy loss is used for the generic loss while a hinge loss is used for the specific one. Can the generic distillation loss also use the hinge loss format? This inconsistency raises concerns.

3\. The motivation for transforming $\gamma^m$ to $\rho^m$ is unclear.

4\. The performance report in Table 3 is incorrect. OGM achieves 0.0531 in the visual Baby category, while "Ours" achieves 0.0529 but is indicated as the best performance.

5\. The suggestion to replace the generic distillation loss with a hinge loss format should be addressed, along with an analysis explaining why the generic distillation loss aims for imitation rather than outdoing, unlike the specific distillation loss.

6\. Some important works are missing, such as CR[1], EliMRec[2], and Modality Balancing[3].

[1]Wang, Wenjie, et al. "Clicks can be cheating: Counterfactual recommendation for mitigating clickbait issue." *Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval*. 2021.

[2]Liu, Xiaohao, et al. "Elimrec: Eliminating single-modal bias in multimedia recommendation." *Proceedings of the 30th ACM International Conference on Multimedia*. 2022.

[3]Shang, Yu, et al. "Enhancing Adversarial Robustness of Multi-modal Recommendation via Modality Balancing." *Proceedings of the 31st ACM International Conference on Multimedia*. 2023.