Title: Adaptive Gradient Masking for Balancing ID and MLLM-based Representations in Recommendation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal recommendation

MLLM
Abstract: In large-scale recommendation systems, multimodal (MM) content is increasingly introduced to enhance the generalization of ID features.
The rise of Multimodal Large Language Models (MLLMs) enables the construction of unified user and item representations.
However, the semantic distribution gap between MM and ID representations leads to \textit{convergence inconsistency} during joint training: the ID branch converges quickly, while the MM branch requires more epochs, thus limiting overall performance.
To address this, we propose a two-stage framework including MM representation learning and joint training optimization.
First, we fine-tune the MLLM to generate unified user and item representations, and introduce collaborative signals by post-aligning user ID representations to alleviate semantic differences.
Then, we propose an Adaptive Gradient Masking (AGM) training strategy to dynamically regulate parameter updates between ID and MLLM branches.
AGM estimates the contribution of each representation with mutual information, and applies non-uniform gradient masking at the sub-network level to balance optimization.
We provide theoretical analysis of AGM's effectiveness and further introduce an unbiased variant, AGM*, to enhance training stability.
Experiments on offline and online A/B tests validate the effectiveness of our approach in mitigating convergence inconsistency and improving performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe semantic distribution gap between ID features and multimodal representations leads to convergence inconsistency during joint training: the ID branch converges quickly, whereas the multimodal branch requires more training rounds, which limits overall performance. To tackle this issue, the paper proposes a two-stage framework consisting of multimodal representation learning and joint training optimization. Specifically, it first fine-tunes a multimodal large language model (MLLM) to generate unified user and item representations, introducing collaborative signals via post-alignment of user ID embeddings to mitigate semantic discrepancies. Then, it introduces an Adaptive Gradient Masking (AGM) training strategy that dynamically adjusts parameter updates between the ID and MLLM branches. AGM estimates the contribution of each representation via mutual information and applies non-uniform gradient masks at the sub-network level to balance optimization.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n* The proposed method effectively addresses the convergence inconsistency between ID and MLLM representations in large-scale recommender systems, leading to improved recommendation performance.\n* The paper introduces Adaptive Gradient Masking (AGM) and its theoretically motivated unbiased variant, AGM\\*.\n* Extensive offline experiments and online A/B testing validate the effectiveness of the proposed approach.\n\n## Weaknesses\n\n* In Section 3.2.1, the authors only describe aligning text-image embeddings within the multimodal item embedding space. Why is there no alignment between the item multimodal embedding and the item ID embedding?\n* Additionally, the backbone function $g()$ operates on $e_u$ and $e_i$ for downstream tasks. How does the UEG module ensure that the learned $e_u^{mm}$ and $e_v^{mm}$ reside in the same space? The training tasks for UEG in Section 3.2.2 rely on aligning user multimodal embeddings and ID embeddings, but this does not guarantee that the final $e_u^{mm}$ and $e_v^{mm}$ are in the same space.\n* Why does Equation (3) use the L2 norm as the alignment loss? Have the authors considered alternative distance metrics?\n* The description in Lines 213–219 is rather vague. For example, “non-uniform adaptive sampling” is not properly introduced. The phrase “This sampling method primarily directs our focus toward parameters carrying richer information” is unclear — is the goal to update these parameters or to preserve them? Finally, what does “probabilistic sampling extends coverage to a broader range of parameters” specifically refer to? Why does this provide broader coverage compared to uniform sampling?\n* Could the authors clarify how the range of $\\eta$ in Line 661 is determined and why $\\eta$ is assigned such a value in Line 667?\n* The notation system in Section 3 is confusing:\n  * In the problem definition in Section 3.1, the dataset $\\mathcal{D}$ uses $u$ and $v$ to indicate clicks $y$, but also uses subscripts $(u_i, v_i, y_i)$, which can lead to inconsistencies if the same user/item appears with different subscripts.\n  * In subsequent descriptions, the notation mixes indexed and non-indexed forms for users/items. For instance, in the formula in Line 185, $\\hat{y}_i$ is used together with unindexed $u$ and $v$.\n  * The variable $x_i$ in Equation (8) is not clearly introduced in context.\n  * …\n* There are minor writing issues throughout the paper:\n  * For example, the abbreviation “MM” in Line 4 should be defined as “multimodal (MM)” upon first use.\n  * There are inconsistent capitalization issues, such as in Line 50.\n  * …\n\nQUESTIONS:\nPlease see the Weaknesses section for detailed questions and suggestions.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper proposes a two-stage framework for recommendation systems that first fine-tunes multimodal large language models (MLLMs) to generate unified user and item embeddings and then introduces Adaptive Gradient Masking (AGM) to dynamically balance gradient updates between ID-based and multimodal representations during joint training. The approach mitigates convergence inconsistency, leading to improved recommendation performance, as demonstrated by theoretical analysis and both offline and online experiments.

STRENGTHS AND WEAKNESSES:
Pros:
1. The author's motivation is clear and reasonable.
2. The proposed method corresponds intuitively to the motivation.
3. A theoretical analysis of the method is provided.

Cons：
1. Lack of time complexity analysis.
2. In the experiment, the author used AUC as the evaluation metric. Adding more recommendation-related evaluation metrics will help to keep consistent with the comparison method.
3. Regarding Section 4.4 Industrial Application, a more descriptive discussion would help to enhance the persuasiveness.

QUESTIONS:
1. AGM seems to need to calculate the Fisher Information Matrix and dynamic sampling mask. Will it cause unacceptable training/inference delays?
2. How does AGM perform when one type of information is of poor quality?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo address the convergence inconsistency problem in the joint training of ID-based and MLLM-based representations in recommendation systems, the authors propose a two-stage framework: (1) fine-tuning MLLMs to generate unified user/item representations while aligning them with ID features, and (2) introducing an Adaptive Gradient Masking (AGM) strategy to dynamically balance parameter updates between ID and multimodal branches during joint training. The method is evaluated through extensive offline experiments and online A/B tests, demonstrating effectiveness in mitigating convergence issues and improving recommendation performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1) The proposed AGM mechanism is innovative, using mutual information to estimate representation contributions and applying non-uniform gradient masking at the sub-network level. The theoretical analysis of AGM's convergence properties adds rigor to the approach.\n2) Comprehensive experimental validation across multiple datasets and backbone architectures demonstrates the method's robustness. The online A/B test results showing significant improvements are particularly convincing for real-world applicability.\n3) The paper provides thorough ablation studies (Table 3) analyzing each component's contribution, and includes a zero-shot evaluation (Table 6) to assess multimodal representation quality, both of which strengthen the methodological claims.\n\nWeaknesses\n1) The computational cost of fine-tuning MLLMs for recommendation tasks isn't adequately discussed. Given the large parameter size of MLLMs, this could be a practical limitation for some applications.\n2) While multiple datasets are used, they're all from similar domains (e-commerce and short videos). Testing on more diverse recommendation scenarios (e.g., news, music) would better demonstrate generalizability.\n3) The comparison with baseline methods (Table 2) could be enhanced by including more recent MLLM-based recommendation approaches beyond those cited.\n\nQUESTIONS:\n1) Could you provide more details about the computational resources required for MLLM fine-tuning? This would help assess practical deployment feasibility.\n2) Have you considered testing your method on recommendation domains beyond e-commerce/short videos (e.g., music or news recommendation) to demonstrate broader applicability?\n3) The paper mentions AGM* improves upon AGM's bias - could you quantify how much performance gain comes specifically from this unbiased variant?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the convergence inconsistency in large-scale recommendation systems by proposing a two-stage framework involving multimodal representation learning and joint training optimizations. The authors also provide theoretical analysis on the convergence properties of the proposed method.  Both offline and online experiments validate the proposed AGM strategy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The motivation of the proposed framework is convincing.\n\n2. The proposed AGM strategy is innovative. It uses mutual information to estimate representation contributions and apply non-uniform gradient masking to implement modality-specific gradient updates.\n\n3. The theoretical analysis of AGM adds rigor to the work.\n\n4. The experiments demonstrate significant performance across different categories of datasets. The inclusion of online A/B testing results also demonstrates the practical effectiveness of the proposed method in real-world industrial applications.\n\nWeaknesses\n\n1. Although the authors provide implementation code in the supplementary material, it appears that the implementation of the proposed multimodal representation learning method is missing. Including this part would enhance the reproducibility and credibility of the work.\n\n2. In Lines 23-24, the paper mentions that “they suffer from limited generalization, making them inadequate for addressing long-tail items, data sparsity, and cold-start scenarios.” However, it appears that the experimental evaluation does not include these scenarios. Including such evaluations would strengthen the empirical validation of the proposed method.\n\nQUESTIONS:\n1. Table 2 presents the overall recommendation performance, where the proposed method shows an advantage over GUME, though the margin is relatively small. This raises the question of how the proposed method compares to long-tail-optimized methods like GUME specifically on long-tail items.\n\n2. Table 6 shows the evaluation of multimodal representations, which demonstrates the proposed method outperforms the \"MLLM\" method with a large margin. But I am concerned about the fairness of the evaluation. If I understand correctly, AGM fine-tunes MLLM with user behavior, which could be a major contributing factor to its performance under the Intermediate Evaluation Protocols of AlignRec.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-04"}