Adaptive Gradient Masking for Balancing ID and MLLM-based Representations in Recommendation
Authors: Yidong Wu
Siyuan Chen
Binrui Wu
Fan Li
Jiechao Gao
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal recommendation
MLLM
Primary area: applications

Abstract: In large-scale recommendation systems, multimodal (MM) content is increasingly introduced to enhance the generalization of ID features.
The rise of Multimodal Large Language Models (MLLMs) enables the construction of unified user and item representations.
However, the semantic distribution gap between MM and ID representations leads to \textit{convergence inconsistency} during joint training: the ID branch converges quickly, while the MM branch requires more epochs, thus limiting overall performance.
To address this, we propose a two-stage framework including MM representation learning and joint training optimization.
First, we fine-tune the MLLM to generate unified user and item representations, and introduce collaborative signals by post-aligning user ID representations to alleviate semantic differences.
Then, we propose an Adaptive Gradient Masking (AGM) training strategy to dynamically regulate parameter updates between ID and MLLM branches.
AGM estimates the contribution of each representation with mutual information, and applies non-uniform gradient masking at the sub-network level to balance optimization.
We provide theoretical analysis of AGM's effectiveness and further introduce an unbiased variant, AGM*, to enhance training stability.
Experiments on offline and online A/B tests validate the effectiveness of our approach in mitigating convergence inconsistency and improving performance.

Adaptive Gradient Masking for Balancing ID and MLLM-based Representations in Recommendation
Yidong Wu *
Siyuan Chen University of Bristol Bristol, UK
Binrui Wu Fudan University Shanghai, China
Imperial College London London, UK
Fan Li
Jiechao Gao Stanford University Stanford, USA
KuaiShou Technology Beijing, China
Abstract
large-scale recommendation systems, multimodal (MM) content increasingly introduced to enhance the generalization of ID features. The rise of Multimodal Large Language Models (MLLMs) enables the construction unified user and item representations. However, the semantic distribution gap between MM and ID representations leads to convergence inconsistency during joint training: the ID branch converges quickly, while the MM branch requires more epochs, thus limiting overall performance. To address this, we propose a two-stage framework including MM representation learning and joint training optimization. First, we fine-tune the MLLM to generate unified user and item representations, and intro- duce collaborative signals by post-aligning user ID representations alleviate semantic differences. Then, we propose an Adaptive Gradient Masking (AGM) training strategy dynamically regulate parameter updates between ID and MLLM branches. AGM estimates the contribution of each representation with mutual in- formation, and applies non-uniform gradient masking the sub-network level balance optimization. We provide theoretical analysis of AGM's effectiveness and further introduce an unbiased variant, AGM*, to enhance training stability. Experi- ments on offline and online A/B tests validate the effectiveness of our approach in mitigating convergence inconsistency and improving performance.
Introduction
Large-scale industrial recommendation systems have traditionally relied on ID-based features, such ID features (e.g.. FM [1. DCN (2), list-wise ID features (e.g.. DIN [] TWIN (4), to model user-item interactions. These features offer strong memorization capabilities and are effective capturing co-occurrence patterns, However, they suffer from limited generalization, making them inadequate for addressing long-tai tail items, data sparsity, and cold-start scenarios. To mitigate these limitations, recent research [5] 6 has incorporated multimodal (MM) content (e.g., images and textual descriptions) to enrich the semantic representations of users and items.
The emergence of Multimodal Large Language Models (MLLMs) facilitates the generation of unified, high-level semantic embeddings from diverse modalities, offering promising avenues for enhancing recommendation performance. As demonstrated in many recent studies, integrating categorical features (e.g.. ID and category) with MLLM-based representations (e.g.. images and texts) can
"Equal Contribution
Corresponding Author: jiechao@stanford.com edu
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

0.70
10:M
Dov 12W
0.70
0.75
Dong IIIIN
Gradient
Mask gradient
0.6
0.65
3
3
4
S
0.55
062
0.50
Y 226
-/16-1/6
10
15
20
10
15
20
ID Network Back-forward MM Network
Epoch
(a) ID
(b) MM
(c) Gradient Masking.
Figure 1: (a) Validation-set AUC comparison between the ID-only model (ID) and the combined model (ID-MM), where "ID in ID-MM" denotes the contribution of the ID component within the combined model; (b) Validation-set AUC comparison between the multimodal-only model (MM) and the combined model (ID-MM), where "MM in ID-MM" denotes the contribution of the multimodal component within the combined model. (c) The illustration of Adaptive Gradient Masking.
Works like FREEDOM [9] introduce auxiliary losses on multimodal data to refine ID embeddings through graph structures. Recent self-supervised learning models such as BM3 [10] adopt joint training of content-basee and user-item objectives. AlignRec [12 proposes unified alignment framework that aligns visual, textual, and categorical modalities using pretraining and contrastive learning.
effectively foster the performance of recommendation system 8 9
Despite these advances, integrating ID and MLLM-based representations remains challenging due to convergence inconsistency during joint training. This stems from semantic gap ID embeddings encode co-occurrence patterns, while MLLM features capture high-level semantics and an opti- mization imbalance, as ID embeddings are trainable whereas MILM parameters are typically frozen. As result, the ID branch quickly dominates training, leading to biased gradients that suppress the multimodal branch and ultimately degrade overall performance. To address this issue, although AlignRec [12 alleviates this convergence speed mismatch between multimodal and ID features by adopting a two-stage strategy: first, pre-training the alignment of content modes and then performing joint training, does not fundamentally address imbalance during the joint optimization phase. In contrast, take more fine- mne-grained approach by dynamically balancing gradient updates between ID and multimodal branches.
In more detail, we propose two-stage framework consisting of multimodal representation learning and joint training optimization. Firstly, we fine-tune pretrained Multimodal Large Language Model (MLLM) generate unified multimodal representations for items and users from visual and textual content. Specifically, Item representations are obtained from multimodal prompts through designated output token, while user representations are derived by aggregating historical item representations. Besides, to bridge the semantic gap between ID and MM representations, we introduce collaborative alignment by post-aligning multimodal embeddings with their corresponding ID embeddings. the second stage, we jointly train the recommendation model by combining ID and MLLM-based representations under the binary cross- cooss-yntrpyy objective, and propose an Adaptive Gradient Masking (AGM) strategy to dynamically regulate their parameter updates during optimization. AGM estimates the informativeness of each representation through mutual information, and applies non-uniform gradient masking at the subnetwork level to encourage balanced convergence. This adaptive mechanism prevents the ID branch from dominating training and ensures consistent convergence across both branches. To demonstrate the effectiveness of our approach, we provide a theoretical analysis showing that AGM leads to more balanced gradient updates, thereby promoting consistent convergence of both branches. Furthermore, we propose an unbiased variant, AGM* which improves training stability by correcting the bias introduced by binary masking.
Our contributions are summarized as follows:
We analyze convergence inconsistency issue between ID and MLLM based representations during joint training in recommendations, and propose a two-stage method address it.
We propose Adaptive Gradient Masking (AGM), subnetwork-level optimization strategy that dynamically balances gradient updates between and multimodal branches. We also introduce AGM*, an unbiased variant that enhances stability.

---

We fine-tune MLLMs to generate unified user and item representations, and introduce collaborative alignment mechanism bridge the semantic gap between ID and MM embeddings.
We provide theoretical analysis on the convergence properties of our method, and validate their effectiveness through extensive offline experiments and online A/B testing in a large- scale industrial recommendation system.
Related works
Multi-modal recommendation Multi-modal recommendation extends the classic collaborative filtering paradigm by integrating diverse content modalities (e.g., images, text, videos) to capture richer contextual signals and thereby enhance recommendation accuracy. Early approaches, such as VBPR [13] and methods fusing visual features with ID embeddings [14] showed that combining basic item side information can significantly improve user-item matching. Subsequently, attention- based architectures, including VECF [16] and MAML 17 explored finer-grained user preferences by leveraging mechanisms like image segmentation
18 and multi-modal feature interactions. With the surge of Graph Neural Networks (GNNs) in recommendation [19) models like MMGCN (20), GRCN [21] and DualGNN [22] pushed multi-modal recommendation further by injecting high- order neighbor relationships or user attentions across item modalities into node representations. To
better reveal item-item semantic similarities, LATTICE [23] constructs separate item-item graphs for each modality and fuses them into latent graph, while MVGAE (24] employs modality- specific variational graph autoencoder to combine molti-modal embeddings. Later, MGCN [8] constructs separate graph views to fuse text, image, and user-item interactions more effectively. Recent work like GUME (25 focuses on leveraging semantic neighbors and refining user modality embeddings to strengthen long-tail item connectivity, while LGMRec (26) separates local user-item interactions from global attribute relationships via hypergraph modeling. FREEDOM 9 tackles noisy item-item structures by freezing precomputed graphs and pruning user-item edges. Self- supervised learning method such as BM3 [IO] proposes a self-supervised learning framework that relies latent embedding dropout create view augmentations. AlignRec (12) addresses alignment challenges across different modalities by unifying multi-modal content and ID-based features through multi-stage alignment process. In addition, the remarkable progress of foundation models in various modalities [27 28 [9] has prompted researchers to adopt large-scale pretrained encoders for capturing more holistic multi-modal representations. Typical examples include VIPS [30], which extends the text-based P5 [3 by incorporating a CLIP image encoder, and MMGRec 33 which reveals item IDs from both collaborative and multi-modal signals via Graph RQ- RQEE Moreover, IISAN 33 proposes a lightweight Decoupled PEFT architecture that simultaneously tackles intra- and moter-modal adaptation in a plug-an and and-play play manner.
Multi-modal Large Language Model Multi-modal Large Language Models (MLLMs) have recently achieved significant progress in integrating language with other modalities, driven by the surge large-scale pretraining [29] 34 35] (6).. Research efforts generally begin with multimodal
understanding and text generation, with representative models such as BLIP-2 (37] and LLAVA (38)
Models like LLaMA-Adapter [39 40 and mPLUG-OW 41 42 align text and image features extensive image- text pairs, while InstructBLIP formats. Despite such progress, enhancing the visual encoder resolution 44 45
43 reshapes multiple tasks into instruction based
48 49
can result in prohibitive memory overhead, especially in multi-page scenarios. To address such problem, TextMonkey [50] employs token resampling to reduce the visual token load. Similarly, more recent models such as Qwen2-VL. [5I) and GPT-4 52 have exhibited outstanding proficiency multimodal reasoning and generation.
3 Methodology
3.1 Problem formulation
We consider the Click-through Rate (CTR) task defined on a dataset D {(u,, Us wo) where each sample consists of (e)d e emm), an item = (eld emm), and a binary label y {0. 1} indicating whether the user engaged with the item. Here, ej! and e denote trainable ID embeddings
3

---

for user and item, respectively, while en and eh represent and item, extracted from MLLM. The goal is learn a prediction function f(u,, Vy 0) that estimates the probability of user-item interaction. The model is trained to minimize binary cross-entropy loss.
representations of user
3.2 Multimodal representation learning
Multimodal information, such as text, images, and other item-related metadata, provides substantial advantages by enhancing the representation of both items and users recommendation systems. However, although pre-trained Multimodal Large Language Models (MLLMs) excel at understanding the data representation [52] [53 54], their original evaluation metrics are not specifically designed to meet the unique demands of recommendation tasks. As result, when faced with extensive user and item information, these pre- pre-traines trained models often struggle to extract key, effective feature embeddings. To address this limitation and improve the extraction of multimodal features, we introduce novel approach to efficiently generate task-relevant embeddings by harnessing multimodal features.
3.2.1 Item embedding
In this section, we fine-tune the MLLM using three novel alignment objectives aimed at enhancing cross-modal consistency. Furthermore, we append special token, [[tem cls), to the end of each item description, which allows the model to condense lengthy multimodal token sequences into compact and informative embeddings.
For item i, we first combine its textual and visual attributes into a unified input description. This accomplished using specific prompt template designed to guide the model's multimodal under- standing: "Integrate text and visual information into an embedding representation. Textual: [Text], Visual: "Image/videds
Then the MILMM encodes the input and generates corresponding token sequence including ]item_cls], in the form of [t1, f2.... tm (tem__lss] J. Finally, the hidden state associated with the [Item_cle _cl____]_]]_______]c___ token is extracted the multimodal embedding for item i.
mm (txxM image,),
where e,! donates the multimodal item embedding of item v.
In the fine-tuning phase, we introduce three specialized alignment tasks for multimodal recommenda- tion, aimed at improving the MILM's performance and suitability in recommendation scenarios.
Text-image alignment: To align visual and textual features, we adopt a method inspired by BERT [55]. For item with image Vú and text T, we mask 20% of Ti's tokens with a special [MASK] token, obtaining f,. The model then takes (V,, f.) input, with the corresponding original description Ti as the target output. This reconstruction task compels the model to leverage visual information to infer missing textual content, thereby learning the meaningful relationship between visual features and textual context for improved cross-modal understanding.
Recommendation systems leverage both structured metadata (e.g., title, price, tags) and unstructured descriptions. Since metadata directly reflects item characteristics, its effective processing enhances MLLMS' encoding performance. Thus, for item we propose predicting its detailed description T; from its metadata, establishing a robust mapping between structured attributes and unstructured text.
Meta-data processing:
User behavior understanding:
The model explicitly captures interest evolution patterns by predicting users' future interactions based their multimodal historical behavior sequences, enabling adaptive optimization of recommendation strategies. For this purpose, we create fine-tuning samples where a user's interaction history (containing both textual and visual item features) serves input, while the next interacted item provides the supervision signal.
3.2.2 User embedding
Despite the basic user information, analyzing historical item sequences is also crucial for predicting user preferences. However, handling extensive user histories and aligning textual information with corresponding images presents significant challenge in multimodal recommendation scenarios. To address the efficient aggregation of long multimodal sequences, we propose the User Embedding Gen- erator (UEG). This module is designed to efficiently aggregate the sequence multimodal information

---

Fine-tune
Content-based Embedding
Aatign
ID-based Embodding
Text-image Al Algnneet
Multimodal LLM
User Embedding Generator
ID-embedding
Mota-data processing
behavior
historical representations
uid-[1307] Idd11228,...917
understanding
Prompt. Teatuali[exx) Visual:
[mage|[ite--lld Imagel [item-cls)
Figure 2: Multimodal representation learning
Trainable
Freeze
Backward
Fisher
ex
0/1 -mmas
NAS
uid=f1302i lid-[0099]
ID-embedding
BackBone1
id
Fusion
BCE
amm
y mm
[Text] [Imgge]
Multimodal LLM
BackBone2
Forward
NAS
Backward
Figure 3: Adaptive Gradient Masking(AGMM,
of user historical items into a unified user multimodal representation. For user u,
mm
UEG(e 77177 mm
mm CV,
(2)
As illustrated in Fig. 2 the UEG a learnable neural network module that takes both the multimodal representations of a user's historical interactions and their unique identifier as input, producing comprehensive multimodal user embedding erm. To stabilize the learning multimodal user representations, we further incorporate pre-trained ID-based embedding layer with frozen parame- ters), which generates an ID-based user representation eid from the user's identifier and historical interactions. The UEG module is optimized via an alignment loss:
Latign
l dd
e
This objective ensures the Iearned multimodal representations maintain consistency with established ID-based embeddings while capturing rich multimodal patterns.
3.3 AGM
Forward propagation and convergence inconsistency Before computing interaction logits, first obtain ID-based embeddings (e e through trainable embedding layers and multimodal embeddings (e e using the method described in Section 3.2. These processed through
separate backbones (. :; Ord) and gmm 0m,), with their outputs concatenated and fused through to produce the final logit:
"m f(u 0; 0) = (((g" eid, oid): mm (e emmm gmm)))
(4)
We train using binary cross-entropy loss:
L N
Σ (y log 0s + (1 - y:) log(1 (.)).
(5)
where ú;
a (f v; 0)). By decomposing sps weight matrix into Widd and Wmm. express
Equation as:
f(u, v) Wid (ee) e 0²d) Wmmm mm (e mu mm + b
(6)
5

---

As shown in Appendix C the ID and MM branches update nearly independently. However, ID features converge faster with stronger signals, dominating predictions and gradient updates. This creates a feedback loop where MM components receive weakened optimization signals, remaining under-utilized at convergence. Consequently, the pathway determines the final logit while the MM pathway stays inadequately optimized, causing the observed convergence inconsistency.
Mask ratio via modal significance As discussed, ID and MM representations carry signals of varying strengths during training. To quantify the influence of different representations within the training objective, introduce contribution score For given sample, the contribution score is formulated as:
y +*p+ (1- y) (1 p), where p (W g(c., C,: + b/2)
The relative contribution ratios pdd and pr are calculated per mini-batch B:
E. 8 E stam?
mam = 1/pdd
(8)
To prevent abrupt fluctuations. we employ exponential moving average (EMA) smoothing with momentum À when updating these ratios across iterations.
Apt + (1 A)P:-1
(9)
In order to mitigate optimization imbalance, we need to provide adequate optimization opportunities to the non-dominant modality while suppressing the parameter updates of the dominant modality. Therefore, inspired by softmax normalization [56], we define the update ratio 7dd and -m as follows: exp(pmmm
mm
(10)
exp( + exp(p"mm)
higher value of y indicates fewer parameters are frozen and more parameters are updated in the corresponding branch.
Adaptive Gradient Masking To implement modality-specific gradient updates, we utilize the Fisher Information Matrix (FIM) [57] donated as F(Ø), which allows us to effectively measure relative importance model parameters across modalities. Specifically, the Fisher Information Matrix is defined as:
F(0) = E ( [llo P(ù r; (
i log P(ù
x; 0)
(11)
00
a0
Following 588 given a batch of data, we estimate the importance of parameters using the diagonal elements of F(0). Formally, the Fisher information for the j-th parameter calculated follows:
[8|
a logggi p(O:
x4 0)
F (0) = B
Σ
(12)
00,
1=1
Subsequently, normalize these diagonal elements and denote π the importance score of the j-th parameter. Aggregating importance scores of all parameters, we obtain probability distribution over the parameter space:
F)
π
[Ti T2, x. Ja where TT
(13)
Σ 0
()
Given the update ratio y of each modality and the parameter-wise probability distribution π, we employ the non-uniform adaptive sampling (59] to generate the gradient mask m(() E {0, 13101, where indicates the parameter will be updated during backpropagation and 0 means it remains frozen. This sampling method primarily directs our focus toward parameters carrying richer information. Concurrently, probabilistic sampling extends coverage broader range of parameters, promoting more thorough exploration and enhancing the model's generalization capability across the entire parameter space.
Finally, the parameter update rule with gradient masking thus becomes:
0 + O(t) 17 ((((t)) m(t)
(14)

---

3,4 Theoretical analysis and AGM*
In this section, we present our theoretical analysis of the Asymmetric Gradient Masking (AGM) approach and its improved variant AGM. We begin by establishing the convergence properties of the original AGM method, then introduce an importance weighting scheme that leads better convergence guarantees in AGM. The convergence of AGM is complicated by the bias introduced through gradient masking. Theorem formalizes this behavior:
Theorem 1 (Convergence of AGM). Suppose the loss function L() is L-smooth and vi()(t)) is unbiased, i.e. E(VI(((t))) v (t)()), which is commonly used in non-comvex optimization. However, 0/1 mask makes Ve()(t)) m(t) biased, i.e., ([()((((t) o m(t)] * VL(O(t)), since (I)((()) and m(l) are independent. Under the Mask-Incurred Error assumption, we have the following convergence result for AGM over T steps:
1+ + (1=+))(+ (1
(15)
where 5 E (0,1) and V 2 O are two constants.
The key limitation here is the bias in gradient estimates caused by the interaction between the mask m(t) and the stochastic gradients. This bias manifests in the (1 - -8) term in the denominator, which slows down convergence. To address this issue, we propose AGM which incorporates importance weighting through modified mask mi))). The weights are defined as:
1
if m, 1
(t) = T + 0,
(16)
otherwise.
where ny represents the probability of the j-th parameter being unmasked and c is small constant for numerical stability. This weighting scheme helps compensate for the bias introduced by the original masking operation. The update rule for AGM becomes:
(t + 1) = (() - 7/V ((O()) O mtt).
(17)
The importance weighting in AGM leads to better theoretical guarantees, as shown in Theorem Theorem 2 (Convergence of AGM*). Under some assumptions for Ve(((t)) O m(t), we have:
2
BIIICCI (1+2)VT) (
(18)
Comparing Theorems and we that AGM removes the problematic (1 5²) term from the denominator, leading to faster convergence. This improvement comes from the fact that the importance weights in mi() help maintain the unbiasedness of the gradient estimates despite the masking operation. The complete proofs and additional technical details can be found in the Appendix.
4 Experiments
4.1 Setup
Dataset We conduct offline experiments on four open-source datasets from diverse recommendation domains. First, we choose the Microlens dataset [60], which features user-item interactions, video introductions, and video cover images. In addition, dataset-Bale Baby, Sports, and Electronics 61 62) which contain user-item interactions, product descrip- tions, and images. All datasets preprocessed with 5-core setting both items and users, as described in 12 10 Detailed statistics of the datasets are provided in Appendix A.1
adopt three categories from the Amazon
Baselines and Evaluation In our experiments, we conduct two parts of evaluation. We first compare AGM with several recent advanced multimodal recommendation models, including VBPR [3] BM3 [10], FREEDOM (9, AlignRec [12) MGCN [8], LGMRec (26). GUME (25), and MM-Rec 63 to
demonstrate its effectiveness. Next, to examine the generalization capability of AGM, we test our
framework with diverse backbones, including MLP (64), DCN [65,, and Fibinet [66]. To evaluate the
performance of all models, we adopt two widely-used classification metrics: AUC (Area Under the
ROC Curve)
67 and LogLoss (Logarithmic Loss) (68).
7

---

Table Performance comparison of AGM and ID, MM, ID+MM models across different backbone architectures, measured by AUC.
MI.P
IXN
Fibinet
Model
Baby
Elec.
Sports
Micro.
Baby
Elec.
Sports
Micro.
Baby
Elec.
Sports
Micro.
0.6741
0.7215
0.7012
0.6696
0.7192
0.6854
0.6825
0.6792
0.7196
0.6901
MM
0.6237
0.6577
0.6295
0.6279
0.6202
0.6527
0.6251
0.6227
0.6154
0.6548
0.6223
0.6245
ID+MM
0.6719
0.7218
0.7072
0.6904
0.6685
0.7123
0.6827
0.6857
0.6641
0.7269
0.7115
0.6915
AGM
0.6864
0.7308
0.7145
0.6992
0.6827
0.7256
0.7062
0.6965
0.6832
0.7310
0.7129
0.6973
Table 2: Comparison of AUC (higher is better) and LogLoss (lower is better) between AGM and other multimodal recommendation methods, Boldface indicates the best performance, and underlined values indicate the second-best.
Baby
Electronics
Sports
MicroLens
Model
AUC
Logloss
AUC
Logloss
AUC
Logloss
AUC
Logloss
VEPR
0.6729
0.6739
0.7158
0.6032
0.6985
0.6533
0.6758
0.6054
FREEDOM
0,6802
0.6708
0.7221
0.5973
0.7023
0.6472
0.6772
0.5957
BM3
0.6715
0.6712
0.7119
0.6084
0.6932
0.6515
0.6705
0.6021
RlignRec
0.6832
0.6681
0.7274
0,5988
0.7101
0.6438
0.6869
0,5906
MGCN
0.6810
0.6695
0.7239
0.5994
0.7085
0.6417
0.6851
0,5881
LGMRec
0.6823
0.6687
0.7247
0.6012
0.7009
0.6480
0.6778
0,5935
GUME
0.6834
0.6679
0.7270
0.5991
0.7119
0.6399
0.6968
0.5876
MM-Rec
0.6643
0.669T
0.7136
0,5987
0.6703
0.6415
0.6735
0.5885
AGM
0.6852
0,6683
0.7285
0,5974
0.7126
0.6405
0.6974
0.5856
AGM
0.6864
0.6656
0.7310
0,5969
0.7145
0.6391
0.6992
0,5841
A AGM--AGM
+0.0012
-0.0027
+0.0025
-0.0005
+0.0019
-0.0014
+0.0018
-0.0015
4.2 Performance Comparison
Compared to different backbones Table illustrates the AUC performance of four methods (AGM" and other three traditional model frameworks) evaluated on different backbone architectures. More specifically, to analyze the individual and combined effects of different training features and compare their performance to AGM*, we conducted experiments on three traditional model frameworks: (i) ID: A baseline model that utilizes only ID features (e.g.. user ID, item ID). (ii) MM: variant that relies solely on multimodal features (e.g., image, text). (iti) ID+MM: A straightforward combination of ID and multimodal features, without specialized fusion or alignment mechanisms.
The results reveal the following key insights: (i) AGM consistently achieves the highest AUC across all backbone architectures, demonstrating that AGM not only achieves superior performance but also maintains robustness and generalizability across various backbones. (ii) Models that rely solely on multimodal features (MM) consistently exhibit the lowest AUC scores across all settings. This suggests that multimodal signals alone are insufficient to capture user preferences, likely due noise and sparse semantics in text image modalities.
Compared to different baselines Table 2 presents the AUC and LogLoss results of our proposed AGM* and AGM framework in comparison with several state-of-the-ard multimodal recommendation baselines across four benchmark datasets. From the experimental results, we derive the following observations: (i) AGM* consistently achieves the best performance on all datasets, although it is based on relatively simple neural network architectures. These gains can be attributed to AGM**s ability adaptively modulate feature contributions during training.
(ii) Compared with other MLLM-based methods such as AlignRec [ and GUME (25), AGM achieves consistently better performance across all datasets. This superiority can be partially attributed our fine-tuning strategy, which enhances the semantic alignment of multimodal features and ensures better adaptation of the pretrained MLLM to the recommendation domain. (iii) Experimental results prove the effectiveness of our proposed unbiased version A as it outperforms the biased AGM.
4.3 Ablation Study
In this part, we conduct ablation studies to evaluate the contribution of each core component in AGM and *GM*. Specifically, compare the full models with the following variants: For AGM: (i)
8

---

Table 3:
Ablation results (AUC) of different modules in AGM.
Dataset
ID+MM
Random
w/o BM
w/o FM
AGM
w/o BM
w/o FM
AGM
Baby
0.6719
0.6792
0.6773
0.6815
0.6852
0.6789
0.6837
0.6864
Elec.
0.7218
0.7267
0.7252
0.7271
0.7285
0.7254
0.7281
0.7308
Sports
0.7072
0.7103
0.7095
0.7112
0.7126
0.7107
0.7121
0.7145
Micro.
0.6904
0.6949
0.6938
0.6954
0.6974
0.6946
0.6967
0.6992
Table 4: The performance of Online A/B Testing at the platform.
Main
watch-time
app usage
long view
short view
+MM
+0.022%
+0.008%
13.%% +0.678%
-0.160% -0.235%
AGM
+0.175%
+0.124%
Random:
When generating the gradient mask m(!), this variant adopts purely random sampling that
ignores parameter importance distributions, instead of the non-uniform adaptive sampling [59 (ii)
w/o Backbone Masking (-BM)) This variant disables gradient modulation on the backbone in
Fig. 3
other words, no additional gradient masking is applied to the backbone network layers;
the gradients flow through these two layers in their original form. (iii) w/o Fusion Masking (-FM))
This variant omits gradient modulation on the fusion block in Fig. 3] so the gradients are propagated through this block in their original form without adaptive gradient masking. For AGM we apply the same ablations: (iv) w/o Backbone Masking (-BM*): Gradient modulation on the backbone is removed in * (v) w/o Fusion Masking (-FM*): Gradient modulation the fusion block is removed in AGM*
Table 3 presents the experimental results, demonstrating two key observations: (i) The removal of any module leads noticeable drop AUC performance, from which we can conclude that all components make contributions AGM and AGM (ii) Among all the ablation variants, removing the dynamic gradient masking on the backbone 1,2 (-BM/-MM*) results in the most significant performance drop. This may be because without proper gradient regulation at this early level, imbalanced learning signals can lead to biased feature extraction from each modality. Consequently, these biases carried forward and accumulated through the subsequent layers which substantially undermines the downstream fusion process, leading to a more pronounced overall performance loss compared to removing other modules.
4.4 Industrial Application
To further assess the real-world effectiveness of our model, integrate AGM into the industrial recommendation system of a large-scale short video platform that serves hundreds of millions of users. The model is deployed in 14-day online A/B test evaluate its performance in a production environment.
We adopt widely-used industry metrics, such app usage time and watch time, measure per- formance. As shown in Table our model achieves substantial improvements over the baseline, further confirming AGM's effectiveness. Notably, the model has now been fully deployed across the platform, actively serving hundreds of millions of users every day.
5
Conclusion
In this paper, we tackle the convergence inconsistency problem in joint training of ID-based and MLLM-based representations within large-scale recommendation systems. We propose a two- stage framework that first learns semantically aligned multimodal representations through MLLM fine-tuning and post-alignment with ID features, and then introduces a novel Adaptive Gradient Masking (AGM) strategy to balance optimization across modalities. Our theoretical analysis and extensive empirical results-acros both offline benchmarks and real-world A/B Besting-demonstrat that the proposed framework effectively mitigates the convergence gap, stabilizes training, and significantly boosts recommendation performance. These findings highlight the importance of

---

coordinated optimization in multimodal recommendation and pave the way for more robust integration of pretrained models into industrial systems.
Acknowledgement
This work was partially supported by the Yonghua Foundation.
References
[1] S. Rendle, "Factorization machines," in 2010 IEEE International conference data mining. IEEE, 2010, pp. 995-1000.
[2] R. Wang, R. Shivanma, D. Cheng, Jain, D. Lin, L. Hong, and E. Chi, "Den v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems," in Proceedings of the web conference 2021, 2021, pp. 1785-1797.
[3] G. Zhou, X. Zhu, C. Song. Y. Fan, H. Zhu, X. Ma, Y. Yan, J. Jin, Li, and K. Gai, "Deep interest network for click-through rate prediction," in Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 2018, pp. 1059-1068.
[4] Chang, C. Zhang, Z. Fu, X. Zang, L. Guan, Lu, Hui, D. Leng, Y. Niu, Song el al., "Twin: Two-stage interest network for lifelong user behavior modeling in etr prediction at kuaishou," " in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023, pp. 7785-3794.
[5] X.-R. Sheng, F. Yang, L. Gong, B. Wang. Z. Chan, Y. Zhang, Y. Cheng, Y.-N. Zhu, T. Ge, H. Zhu et al., "Enhancing taobao display advertising with multimodal representations: Chal- lenges, approaches and insights," in Proceedings the 33rd ACM International Conference Information and Knowledge Management, 2024, pp. 4858 4865.
[6] Q. Liu, J. Zhu, Yang. Q. Dai, Z. Du, X.-M. Wu, Z. Zhao, R. Zhang, and Dong, "Multimodal pretraining, adaptation, and generation for recommendation: A survey," in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024, pp. 6566 6576.
[7] Y. Ye, Z. Zheng, Y. Shen, T. Wang. H. Zhang, P. Zhu, R. Yu, K. Zhang, and H. Xiong, "Harnessing multimodal large language models for multimodal sequential recommendation, in Proceedings of the AAAI Conference Artificial Intelligence, vol. 39, no. 12, 2025, pp.
13 33009-13777 077.
[8] Yu, Z. Tan, G. Lu, and B.-K. Bao, "Multi-vien graph convolutional network for multimedia recommendation." in Proceedings of the 31st ACM international conference multimedia. 2023, pp. 6576-6585.
[9] X. Zhou and Z. Shen, "A tale of two graphs: Freezing and denoising graph structures for multimodal recommendation." in Proceedings of the 31 ACM International Conference Multimedia, 2023, pp. 935-943.
[10] X. Zhou, H. Zhou, Y. Liu, Z. Zeng, C. Miao, Wang. Y. You, and F. Jiang, "Bootstrap latent representations for multi-modal recommendation." in Proceedings of the ACM web conference 2023, 2023, pp. 845-854.
[11] K. Zhang, Y. Qin, Jin, Liu, R. Su, W. Zhang, and Y. Yu, "Dream: A dual representation learning model for multimodal recommendation." arXiv preprint arXiv:2404. 1.04.111113 11119, 2024.
[12] Y. Liu, K. Zhang, X. Ren, Y. Huang, J. Jin, Y. Qin, R. Su, R. Xu, Y. Yu, and W. Zhang, "Alignrec: Aligning and training in multimodal recommendations,"
Proceedings of the 33rd ACM International Conference Information and Knowledge Management, 2024, pp.
1503-1512.
[13] R. He and J. McAuley, "Vbpr: visual bayesian personalized ranking from implicit feedback," in Proceedings of the AAAI conference artificial intelligence, vol. 30, no. 1, 2016.
[14] Q Liu, S. Wu, and L. Wang. "Deepstyle: Learning user preferences for visual recommendation." in Proceedings of the 40th international acm sigir conference on research and development in information retrieval, 2017, pp. 841-844.
10

---

[15] S. Rendle, C. Freudenthaler, Z. Gantner, and L. Schmidt-Thiemes "Bpr: Bayesian personalized ranking from implicit feedback," arXiv preprint arXiv:/205.2613. 2012.
[16] X. Chen, H. Chen, H. Xu, Y. Zhang, Y. Cao, Z. Qin, and H. Zha, "Personalized fashion recommendation with visual explanations based on multimodal attention network: Towards visually explainable recommendation," in Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2019, pp. 765-774.
[17] F. Liu, Z. Cheng. C. Sun, Y. Wang. L. Nie, and M. Kankanhalli, "User diverse preference mod- cling by multimodal attentive metrie learning," in Proceedings of the 27th ACM international conference on multimedia, 2019, pp. 1526-1534.
[18] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition." arXiv preprint arXiv: 1409. /556, 2014.
[19] S. Wu, F. Sun, W. Zhang, X. Xie, and B. Cui, "Graph neural networks in recommender systems: survey," ACM Computing Surveys, vol. 55, no. 5, pp. 1-37, 2022.
[20] Y. Wei, X. Wang. L. Nie, X. He, R. Hong, and T.-S. Chua, "Mmgen: Multi-modal graph convolution network for personalized recommendation of micro-video," in Proceedings of the 27th ACM international conference on multimedia, 2019, pp. 1437-1445.
[21] Y. Wei, X. Wang. I.. Nie, X. He, and T.-S. Chua, "Graph-refined convolutional network for multimedia recommendation with implicit feedback," in Proceedings of the 28th ACM international conference on multimedia, 2020, pp. 3541-3549.
[22] Q- Wang, Y. Wei, J. Yin, J. Wu, X. Song, and I.. Nie, "Dualgnn: Dual graph neural network for multimedia recommendation. IEEE Transactions on Multimedia, vol. 25, pp. 1074-1084, 2021.
[23] J. Zhang, Y. Zhu, Q. Liu, S. Wu, S. Wang, and L. Wang. "Mining latent structures for multimedia recommendation," in Proceedings of the 29th ACM international conference on multimedia, 2021, pp. 3872-3880.
[24] Yi and Z. Chen, "Multi-modal variational graph auto-encoder for recommendation systems." IEEE Transactions on Multimedia, vol. 24, pp. 1067-1079, 2021.
[25] G. Lin, M. Zhen, D. Wang, Q. Long, Y. Zhou, and M. Xiao, "Gume: Graphs and user modalities enhancement for long-tail multimodal recommendation," in Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, 2024, pp. 1400-1409.
[26] Z. Guo, J. Li, G. Li, C. Wang, S. Shi, and B. Ruan, "Lgmrec: local and global graph learning for multimodal recommendation." in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 8, 2024, pp. 8454-8446 8462.
[27] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. Liu, "Exploring the limits of transfer learning with unified text-to-text transformer," Journal of machine learning research, vol. 21, no. 140, pp. 1-67, 2020.
[28] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell el al., "Language models are few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
[29] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark et al., "Learning transferable visual models from natural language supervi- sion," in International conference on machine learning. PmLR, 2021, pp. 8748-8763.
[30]= S. Geng. J. Tan, S. Liu, Z. Fu, and Y. Zhang, "Vip5: Towards multimodal foundation models for recommendation," " arXiv preprini urXiv:2305. 14302, 2023.
[31] S. Geng, S. Liu, Z. Fu, Y. Ge, and Y. Zhang. "Recommendation language processing (rlp):
A unified pretrain. personalized prompt & predict paradigm (p5)," in Proceedings of the 16th
ACM conference on recommender systems, 2022, pp. 299-315.
[32] H. Liu, Y. Wei, X. Song, W. Guan, Y.-F Li. and L. Nie, "Mmgrec: Multimodal generative recommendation with transformer model." arXiv preprint arXiv:2404.1653 2024.
[33] Fu, X. Ge, X. Xin, A. Karatzoglou, I. Arapakis, J. Wang, and M. Jose, "lisan: Efficiently adapting multimodal representation for sequential recommendation with decoupled peft," in Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2024, pp. 687-697.
11

---

[34] J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. C. H. Hoi, "Align before fuse: Vision and language representation learning with momentum distillation." Advances in neural information processing systems, vol. 34, pp. 9694-9705, 2021.
[35] Li, D. Li, C. Xiong, and S. Hoi, "Blip: Bootstrapping language-image pre-training for unified vision- -ainugla understanding and generation," in International conference machine learning. PMLR, 2022. pp. 18888-12 900.
[36] E. Kasneci, K. Seßler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. Günnemann, E. Hüllermeier el al., "Chatgpt for good? on opportunities and challenges of large language models for education," Learning and individual differences, vol. 103, p. 102274, 2023.
[37] Li, D. Li, Savarese, and S. Hoi, "Blip-2: Bootstrapping eaaguage-imagee pre-training with frozen image encoders and large language models," in International conference on machine learning. PMIR, 2023, pp. 730--11 742.
[38] H. Liu, C. Li, Q Wu, and Y. J. Lee, "Visual instruction tuning." Advances in neural information processing systems, vol. 36, pp. 34 898-949116 2023.
[39] R. Zhang, J. Han, C. Liu, P. Gao, A. Zhou, X. Hu, S. Yan, P. Lu, H. Li, and Y. Qiao, "Llama- adapter: Efficient fine-tuning of language models with zero-init attention," arXiv preprint arXiv:2303. 16199, 2023.
[40] P. Gao, J. Han, R. Zhang. Z. Lin, S. Geng, A. Zhou, W. Zhang, P. Lu, C. He, X. Yue et al., "Llama- adapter v2: Parameter-efficient visual instruction model," arXiv preprint arXiv:2304.15010 2023.
[41] Q. Ye, H. Xu, G. Xu, Ye, M. Yan, Y. Zhou, J. Wang, A. Hu, Shi, Y. Shi et al., "mplug- owl: Modularization empowers large language models with multimodality." arXiv preprint arXiv:2304. 14178, 2023.
[42] J. Ye, A. Hu, H. Xu, Q. Ye, M. Yan, G. Xu, C. L., Tian, Q Qian, J. Zhang et al., "Ureader: Universal ocr-free visually-situated language understanding with multimodal large language 2023.
model," arXiv preprint arXiv:2310.00
[43] W. J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi, "Instructblip: Towards general-purpose vision-language models with instruction tuning," 2023. [Online]. Available: https:/aaxxxvorr//b//////3
[44] Bai, S. Bai, S. Yang. S. Wang, S. Tan, Wang. J. Lin, C. Zhou, and J. Zhou, "Qwen-vl: versatile vision-languaa
model for understanding, localization, text reading, and beyond," 2023. [Online]. Available: https://axiivoo//a//233...2
[45] H. Liu, C. Li, Y. Li. and Y. J. Lee, "Improved baselines with visual instruction tuning." in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2024,
pp. 26 2662966662 306.
[46] H. Liu, C. Li, Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee, "Llavanext: Improved reasoning, ocr, and world knowledge," 2024.
[47] X. Dong, P. Zhang, Y. Zang, Y. Cao, B. Wang, L. Ouyang, X. Wei, S. Zhang, H. Duan, M. Cao al., "Inlernln-xcompposee:: Mastering free-form text-image composition and comprehension in vision-language large model," arXiv preprint arXiv:2401. 16420, 2024.
[48] Z. Guo, R. Xu, Y. Yao, J. Cui, Z. Ni, C. Ge, T.-S. Chua, Z. Liu, and G. Huang, "Llava-uhd: an Imm perceiving any aspect ratio and high-resolution images," in European Conference on Computer Vision. Springer, 2024, pp. 390-406.
[49] G. Luo, Y. Zhou, Y. Zhang, X. Zheng, X. Sun, and R. Ji. "Feast your eyes: Mixture-of-resolutio adaptation for multimodal large language models," " arXiv preprint arXiv:2403. 03003, 2024.
[50] Y. Liu, B. Yang. Q. Liu, 7. Li, 7. Ma, S. Zhang, and X. Bai, "Textmonkey: An ocr-free large multimodal model for understanding document," arXiv preprint arXiv:2403.044477 2024.
[51] Wang, S. Bai, S. Tan, S. Wang. Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge al., "Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution," arXiv preprint arXiv:2409.12199 2024.
[52] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, Altenschmidt, S. Altman, S. Anadkat et al., "Gpt-4 technical report," arXiv preprint a7Xiv:2303.08574.2000 2023.
12

---

[53] G. Team, R. Anil. S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican et al., "Gemini: family of highly capable multimodal models," arXiv preprint 1xXiv:2312./1803 2023.
[54] S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen, "A survey on multimodal large language models," arXiv preprini arXiv:2306. 13549, 2023.
[55] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding." in Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technolo- gies, volume I (long and short papers), 2019, pp. 4171-4186.
[56] Bridle, "Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition," in Neurocomputing: Algorithms, Architectures and Applications. Springer, 1990, pp. 227-236.
[57] R. A. Fisher, "On the mathematical foundations of theoretical statistics." " Philosophical Trans- actions of the Royal Society of London. Series A, Containing Papers of Mathematical or Physical Character, vol. 222, pp. 309-368, 1922.
[58) J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Frabska-Barwinsk. D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell, "Overcoming catastrophic forgetting in neural networks," Proceedings of the National Academy of Sciences, vol. 114, no. 13, pp. 3521-3526, 2017.
[59] S. Gopal, "Adaptive sampling for sgd by exploiting side information," " in Proceedings of The 33rd International Conference on Machine Learning, Proceedings of Machine Learning Research, M. F. Balcan and K. Q. Weinberger, Eds., vol. 48. New York, New York, USA: PMLR, 20-22 Jun 2016, pp. 364-372
[60] Y. Ni, Y. Cheng, X. Liu, J. Fu, Y. Li, X. He, Y. Zhang, and F. Yuan, "A content-driven micro- video recommendation dataset at scale," arXiv preprini urXiv:2309. /5379, 2023.
[61] McAuley, C. Targett, Q. Shi, and A. Van Den Hengel, "Image-based recommendations on styles and substitutes." in Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, 2015, pp. 42.52.
[62] R. He and J. McAuley, "Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering," in proceedings of the 25th international conference on world wide web, 2016, pp. 507-517.
[63] C. Wu, F. Wu, T. Qi, C. Zhang, Y. Huang, and T. Xu, "Mm-rec: Visiolinguistic model empowered multimodal news recommendation," " in Proceedings of the 45th international ACM SIGIR conference research and development in information retrieval, 2022, pp. 2560-2564
[64] Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," nature, vol. 521, no. 7553, pp. 136-444, 2015.
[65] R. Wang, B. Fu, G. Fu, and M. Wang, "Deep & cross network for ad click predictions," in Proceedings of the ADKDD 17, 2017, pp- 1-7.
[66] T. Huang, Z. Zhang, and Zhang. "Fibinet: combining feature importance and bilinear feature interaction for click-through rate prediction," in Proceedings of the 13th ACM conference recommender systems, 2019, pp. 169-177.
[67] Huang and C. X. Ling, "Using auc and accuracy in evaluating learning algorithms," IEEE Transactions on knowledge and Data Engineering, vol. 17, no. 3, pp. 299-310, 2005.
[68] J. Friedman, T. Hastie, and R. Tibshirani, "Additive logistic regression: a statistical view of boosting (with discussion and a rejoinder by the authors)," The annals of statisties, vol. 28,
no. 2, pp. 337-407, 2000.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The contributions and scope of the paper included in the abstract and Please refer to the first and last paragraph Section I for scope and contributions.
Section I
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitations of work in the Appendix P
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

We provide required assumptions and some proof' in Section 3 and the detailed proofs are shown in Appendix D and
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Model architectures can be seen Section 3 and datasets as well as experi- mental hyperparameters can be seen in Section 4 and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
A public code repository is included in Appendix A.3
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wttpcccpppppppppeeeeeeeeeeeeeeee httpa///giiieee/eeeeebiiissssss /Cc////oddiiisssseeeeeee ssionolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Datesets and training details can be seen Appendix A Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The statistical metrics used are all officially defined and well-established. The experimental results are the average of five experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Information on the computer resources can be seen in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We read and followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential positive impacts that our model will bring Ap- pendix G
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release high risk models and the datasets used in the paper open-sourced.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited the original paper for all the existing assets used in this paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We don not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemoodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: MLLM is utilized part of our model for multimodal embeddings.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy battps://neuree h0tps///ec.oooeeeeeee/////////// for what should or should be described.
20

---

Supplemental Material: Adaptive Gradient Masking for Balancing ID and MLLM-based Representations in Recommendation
A Experimental details
A.1 Dataset
The detailed descriptions of the datasets used in the main text are follows:
Amazon Baby: This dataset consists of user-generated reviews on baby-related products sold Amazon, such bottles, diapers, and infant toys. It typically includes product names, review texts, and star ratings, making it valuable resource for research in recommendation systems.
Amazon Sports: This dataset includes Amazon user reviews pertaining to sports and outdoor products, including equipment, camping gear, and fitness devices. Alongside textual reviews and ratings, the dataset provides insights into consumer preferences and opinions, supporting various applications recommendation modeling.
Amazon Electronics: This dataset is derived from Amazon reviews electronic products, covering items such as mobile phones, cameras, laptops, and home appliances. addition to review content and ratings, it contains product metadata, facilitating extensive research in product recommendation.
Microlens: This dataset contains user -eer-item interaction records, video introductions, and video cover images. Each video within MicroLens contains multiple modalities, including text descriptions, images, audio, and raw video information. This extensive coverage enables robust benchmarking of both classical and state-of-the-ar recommendation systems.
Table 5: Statistical Information of datasets.
Dataset
Baby
Sports
Electronics
Microlens
#User
19,445
35,598
192,403
25,411
#Item
7,050
18.357
63,001
20,276
#Interaction
160,792
296,337
1,689,188
223,263
Sparsity
99.88%
99.95%
99.99%
99.96%
A.2 Baselines
We summarize the key characteristics of the baseline methods used in our comparative evaluation:
VBPR 13:
This model incorporates visual features into matrix factorization by treating them as auxiliary information for user preference, learning with BPR loss.
This model simplifies the self-supervised multimodal recommendation frame- work by using latent representation dropout mechanism instead of graph augmentation to generate contrastive views, enhancing representation learning.
BM3 [00]
This model refines ID-based item representations through auxiliary con- trastive losses on multimodal data, and leverages graph structures to align different modalities in a unified embedding space.
FREEDOM 9
This model addresses modality misalignment and optimization imbalance by proposing two-stage training strategy that first pre-trains inter-content alignment, then jointly optimizes with the recommendation task.
Rei 11
21

---

MGCN 8::
This model purifies modality features using item behavior data to reduce noise, and models user-modal preference through multi-view graph convolution networks.
LGMRee (6):
This model enhances multimodal recommendation by learning both local and global semantic relations in ttem-user graphs, effectively capturing fine-graing teee-graied user preferences.
GUME [25]: This model improves long-
loig-aiil tail multimodal recommendation by incorporating user-specific modality preferences and behavior graphs to enhance personalized modeling.
MM-Ree 633 This model enhances multimodal news recommendation by jointly encoding news text and image ROIs using a visiolinguistic model, and introduces a candidate aware attention mechanism to identify relevant historical news.
A.3 Training details
The code and model are available at: AGM
In the fine-tuning phase, we adopted Qwen2vl-2b[51 as the backbone model. The model was fine-tuned for 5 epochs across four distinct datasets, utiliziig batch size of 128 on A100 GPUs.
For AGM, offline evaluations were conducted using TensorFlow 2.15.0 on a single RTX 4090 GPU, selecting Adam the optimizer. Hyperparameters, including batch size and learning rate, were systematically tuned across candidate sets of (256, 512, 1024, 2048) and {le - 3, le 4, 1e - 5}. respectively. The best model was selected based the minimum validation loss, and early stopping was applied with patience of 5 to prevent over-fitting.
B In-depth analysis
B.1 Evaluation of multimodal representations
To assess the quality of the multimodal representations generated by AGM, we conduct zero-shot recommendation experiments on the Amazon dataset following the protocol of AlignRec [12]. For each user, the last interacted item is regarded as the target item, and the rest form the historical sequence. We average the multimodal features of historical items to construct the user representation, then compute its similarity with the candidate items' features to evaluate if the target item ranks within top-K. K. We compare the following methods: (i) Amazon, which uses separately trained visual (CNN) and textual (Transformer) encoders; (ii) MILM, which directly uses the frozen MLLM outputs; (iii) w/o Lutign. variant of AGM that disables the feature alignment loss; and (iv) Ours (AGM), which includes all proposed components. We report Recall @ 20 and Recall 50 Table 6 The results reveal three key observations: (i) AGM significantly outperforms traditional feature extractors and raw MILM features, indicating the benefit of task-specific representation refinement. (ii) Removing the alignment loss leads to noticeable performance drops, highlighting its importance in guiding effective feature selection and fusion. (iii) AGM achieves consistent improvements across all categories, demonstrating its robust capability in modeling multimodal user-item relationships in a zero-shot scenario.
Table 6: Evaluation of multimodal representations
Baby
Elec.
Sports
Generation Methods
R@20
R@50
R@20
R@50
R@20
R@50
Amazon
0,0052
0,0150
0.0093
0.0135
0.0040
0.0072
MLLM
0.0140
0.0345
0.0202
0.0364
0.0053
0.0089
w/o Lalign
0.0225
0.0426
0.0251
0.0417
0.0120
0.0159
Ours
0.0276
0.0509
0.0293
0.0478
0.0175
0.0206
B.2
Convergence analysis
To better illustrate the convergence performance of different methods, we plot the AUC values across training epochs for AGM, AGM*, and the combined model (ID-MM) on Amazon Baby, Electronics, Sports and Microlens. As shown in 4ig44 both AGM and *GM* demonstrate increasing
22

---

0.80
0.80
0.79-
ID-MM
0.79
ID-MM
0.78
0.78
0.77-
AGM
0.77
AGM
0.76-
0.76
0.75
0.75
AUC
0.73
0.74
AUC
0.73
0.74
0.72
0.72
0.71
0.71
0.70
0.70
0.69
0.69
0.68
0.68
0.67
0.67
0.66-
0.66
0.65
10
1
5
10
Epoch
Epoch
(a) Baby
(h) Elec.
0.80
0.80
0.79
ID-MM
0.79
ID-MM
0.78-
AGM
0.78
AGM
0.77
AGM
0.77
0.76
0.76
0.75
0.75
AUC
0.73
0.74
AUC
0.73
0.74
0.72
0.72
0.71
0.71
0.70
0.70
0.69
0.69
0.68
0.68
0.67
0.67
0.66
0.65
1
10
1
5
10
Epoch
Epoch
(c) Sports
(d) Micro.
Figure 4: Validation-set AUC comparison of the combined model (ID-MM), AGM, and AGM across four datasets
AUC throughout training epochs, indicating effective joint learning. In contrast, ID-MM exhibits downward trend in AUC as training progresses. This degradation can be attributed to the ID features dominating the training process of ID-MM, especially in the absence of any modulation mechanism on gradients during the backpropagation phase. Consequently, as the model overfits the ID space over time, the performance of ID-MM degrades. This dominance suppresses the contribution of multimodal features, resulting in suboptimal representations and overall performance decline. In contrast, our AGM and *GM * introduce Gradient Modulation, which dynamically balances the gradient flow between the ID and multimodal branches, preventing ID features from overwhelming MM features, allowing both to contribute meaningfully to the final prediction.
In addition, AGM shows better overall performance compared to AGM, especially in larger datasets (e.g.. Amazon Electronics). Specifically, by compensating for the gradient masking bias introduced by the original masking operation in the AGM, AGM* improves the gradient update process, promoting faster convergence, and ultimately improving the final AUC of the model.
C Convergence inconsistency analysis of ID-MM combine model
In the joint learning of ID and multimodal representations for recommendation. we observe an optimization imbalance phenomenon, where one representation dominates the learning process, causing the other to be under- ootimizp. We introduce the analysis of the optimization imbalance phenomenon for the model with concatenation as fusion method. In our recommendation model, the logits output formulated as:
p((,)) Wif(e"d, ('d) g(e mm
(19)
23

---

To observe the optimization process of each component individually, W can be represented as the combination of two matrix: Widd and Wmm.. The Equation 19 can be rewritten as:
(#:)) W f(e"d 0id) ; + Wmm * g( 0mm) +
(20)
The model is trained using binary cross-entropy loss:
1 N
L
(y log ús + - vi) log(1 (s))
(21)
where the predicted probability is ú = a (p((z)) ()): The gradient of the loss with respect to the logit is:
aL
o ((p(x))) Vi-
(22)
0p(z,)
Applying the chain rule, the gradients for the ID weights and backbone parameters of the ID representation are follows:
aL
N
OL
Me )
(23)
oW
N
0e(a,,)
OD @p(x;) agid N
OL
N
Wiid af(eid; gía)
(24)
ag
According to Equations 23 and 24 the optimization of Wid and f(-) is nearly independent of the optimization of multimodal parameters Wmm and g(.), except for the term associated with the training loss. As result, the backbone have limited ability to make adjustments based on feedback from one another. The analysis of the feed-forwart and back pook-propagation stages reveals that both the model predictions and gradients are governed by the sum of ID and multimodal (MM) components. Since ID features converge faster and contain stronger discriminative information, they OL through Wird f(e" 0id)) Even if the MM representations remain under-optimized and produce erroneous outputs during training, the more informative ID components can still "correct" these errors through summation, thereby influencing both the feed-forward and pack-propagation processes. Consequently, according Eq. (9) and Eq. (11), the MM, which has relatively lower confidence in the correct category, receives limited optimization, leading its under-utilization. Based on this analysis, ID features play dominant role in the optimization process. As the model approaches convergence, MM components may still require further training to compensate for their under-optimized features.
dominate the model prediction p(z;) and gradient ap(z;;
D Convergence analysis of AGM
In this section, we give a detailed proof of Theorem 1. Recall update step stochastic gradient descent (SGD) for AGM is:
(t) n (t() @ m(),
(25)
where ())) is the stochastic gradient of (()(())) and r > is the learning rate, and m(t) is binary mask vector. To analyze the convergence of AGM, have the three following common assumptions for (()).
Assumption 1 (Smoothness). The loss function C is L-smooth, which is common for non-comvex optimization. That is, for any 0, O', we have:
(( - L(0') S (VL(6'), e - 80) - 10-8810
I.
(26)
Assumption 2 (Bounded Variance). We assume that the stochastic gradient v m(t) is biased and its variance is bounded. That is, for any (tt and m(t), we have
E (V(((()) m()] vL(O())) b(0()),
(27)
and
E ([(((((((t) C m(t E (((((()) c mtt)]l| < +VV[(((() where o² and 2 are constants.
(28)
Assumption Mlask-Innurred Error). For any (tt and mtt), we have ||E (V)( @ m(t) ((((()))] < [((((()) where the constant 8 E [0, 1].
(29)
24

---

Proof of Theorem 1
As Theorem claims:
1+ (1+
Σ
|C((()) s0
(30)
(1- uUVT
1=1
By Assumption !
and let h(t) = v((((t)) @ mt), we have
- < (V2(0()), ((++ 0(1)) + 10 | 0
- (VZ(0(t), T
[[((t)]2, (using eq. 25
(31)
Taking expectation over both sides of 31 and by using Assumption 2
r E [II(()) (2((0)) +(0(())) (000)) +
E[C(0(L+ +
= -- (VC((())). + T
(Eili((t) -
2
S -1 (VC(1(()).VV((((()))))) +(((0) + L
2 + +
s 7) V((((() ((())) + ! +
(32)
2
where the last inequality is due 77 S (1+t)
| Since - (a. b) + 2 = |a=bl|
|||||
then 32
implies that
2
T
B|c(O(++ 1))] [(0(t)) < - (VC((())) (VC(0)) VC((())) + 2
E [116(()
マク ((0())) (VC((())) Lo
(33)
2
27 Next, by27 in Assumption and Assumption 3
we know
= b(((t))) = [E())(((t)) o m()] (E(((t)) s =
(34)
Therefore, by
33 and (34)
have
E [(((((+ - ((0(t))) < 711 711 82)
|L(((()) + 1T
(35)
2
2
which implies
(L(((()) < 2B(((0(t)) L(0 + 1))
nLo²
(36)
m71 52)
1-
By summing up for L 1.....77 T, we have
VC(00)) 2E 2E(((0(1)) - C(O(T + 1))]
LLo?
+
TT(1 52)
1-
< 2C(O(1)) nLo2
(37)
Tn(1 82)
By setting 1) (1+D)LV7* get
2(1 (1+)))C(((1))
x²
1 (
(1++11+ (1+2)2
1+
(C(0()))
+
=0
VT(18")
(1+2)VT(-8" 82)
(1-
+
V)vT
(38)
25

---

E Convergence analysis of AGM
For unbiased stochastic gradient, recall that the update step of SGD is
( + + = ( - v()) @ m(().
(39)
where (t((((t) the stochastic version the gradient of loss function VZ(((()) at ( and rj > is the learning rate.
The element mm() is given by
if my)
(t) =
(1(1)*
(40)
0, otherwise.
Suppose that stochastic gradient V((((tt) is unbiased, i.., [VV VL(((t)) Then we have
E[V((((t)) @ mt) E
Fm [V((0(t) = Bm())) V((O(t))]]
V(((()
m(() V(((()))
=[[{(((tt)]
VL(0(L)).
(41)
indicating that VE(O(t)) O m(t) is also unbiased.
Assumption 4 (Bounded Variance), We assume that the stochastic gradient Ve(0)@m(() is unbiased and its variance is bounded. That is, for any O(t) and rin(),, we have
[(Ve(0()) m() VL((())).
(42)
and
(I(((((()) @ mm(( (V((((()) o (()]I = +
(43)
where x² and are two constants.
Proof of Theorem
As Theorem
2 claims: under assumptions
we have:
So ++ (
Σ |VCOO()
(44)
(1+2)VT
where the learning rate is set 77 (I+2)EVF*
By Assumption 1 and we h(t) == V((((tt) 3 m(). then we have
1)) C(((t) (VL(((()) O( + 1) 0()) + |1(t++)) (()]
- (())))))))))))))))))))))))) + 222 |
(45)
2
Taking expectation over both sides of 45 and by using Assumption we have
L
BIC(O(L= + 1))] (()()) -
B(1(((0)17)
-= LLL E ([((((()/c(CC) -
I
S -=
((1 =) +0?)
T2
- 2 [
(46)
2
26

---

where the last inequality is due to S (L+D)L
Therefore, we have
V2(0 2E(C(0(0) L(O(1 + 1))1 + =n/oo
(47)
By summing up for t 1..... T, have
(C((()) < 2BIC(0(1)) - L(O(T + 1))]
+ nLoo
Tn
2C()(1))
+nT.0 Tn
(48)
Since 77
1 (1t6)Ly7*
get
T
2(1 + (1)+L((((())
d² (1+t)VT
T V2(0(())
(49)
VT
F
Limitation
While our proposed AGM framework demonstrates promising results, several limitations remain. First, the framework's performance may depend on the quality and diversity of the multimodal data available during training. Second, the current experiments focus recommendation tasks, and extending the approach other multimodal applications may require further adaptation.
G Broader Impacts
Our work Adaptive Gradient Masking for recommendation systems presents several important societal implications. The improved ability to handle multimodal content could significantly enhance recommendation quality, partieularly for niche and cold-start items, potentially benefiting both users through more relevant suggestions and content creators through better exposure. The framework's ability to balance different feature types may also lead to more diverse recommendations, mitigating some common filter bubble effects.
27


Reviews:
[{"review_id": "f9Qza3gHoM", "reviewer": "Reviewer_QngB", "summary": "The paper addresses the convergence inconsistency in large-scale recommendation systems by proposing a two-stage framework involving multimodal representation learning and joint training optimizations. The authors also provide theoretical analysis on the convergence properties of the proposed method.  Both offline and online experiments validate the proposed AGM strategy.", "questions": "1. Table 2 presents the overall recommendation performance, where the proposed method shows an advantage over GUME, though the margin is relatively small. This raises the question of how the proposed method compares to long-tail-optimized methods like GUME specifically on long-tail items.\n\n2. Table 6 shows the evaluation of multimodal representations, which demonstrates the proposed method outperforms the \"MLLM\" method with a large margin. But I am concerned about the fairness of the evaluation. If I understand correctly, AGM fine-tunes MLLM with user behavior, which could be a major contributing factor to its performance under the Intermediate Evaluation Protocols of AlignRec.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The motivation of the proposed framework is convincing.\n\n2. The proposed AGM strategy is innovative. It uses mutual information to estimate representation contributions and apply non-uniform gradient masking to implement modality-specific gradient updates.\n\n3. The theoretical analysis of AGM adds rigor to the work.\n\n4. The experiments demonstrate significant performance across different categories of datasets. The inclusion of online A/B testing results also demonstrates the practical effectiveness of the proposed method in real-world industrial applications.\n\nWeaknesses\n\n1. Although the authors provide implementation code in the supplementary material, it appears that the implementation of the proposed multimodal representation learning method is missing. Including this part would enhance the reproducibility and credibility of the work.\n\n2. In Lines 23-24, the paper mentions that “they suffer from limited generalization, making them inadequate for addressing long-tail items, data sparsity, and cold-start scenarios.” However, it appears that the experimental evaluation does not include these scenarios. Including such evaluations would strengthen the empirical validation of the proposed method.", "was_revised": true, "final_justification": "The authors’ rebuttal has successfully addressed all of my concerns. I have also reviewed the discussions between the authors and other reviewers. Based on the clarifications provided and the overall discourse, I believe that a borderline accept is a reasonable and fair recommendation.", "rebuttal": "Dear reviwer QngB,\n\nWe sincerely thank you for your thoughtful feedback and positive score. We truly appreciate the time and effort you dedicated to evaluating our work. Below, we address each of your concerns in detail and provide clarifications where necessary.\n\n**Q1: Supplement code**  \n\nDue to the NeurIPS 2025 official policy, updating the submitted github repository is not permitted during the rebuttal phase. We will update the repository to include the missing multimodal representation learning module immediately after the rebuttal process.\n\n**Q2: Long-tail evaluation**  \n\nWe have added additional experiments to directly evaluate our method’s performance on long-tail items. Specifically, for each dataset, we sort items by their total number of user interactions in descending order. We then partition the items into the following frequency bands:\n- High-frequency: top 20% most popular items.\n- Mid-frequency: items ranked in the 20%–50% range.\n- Low-frequency: the remaining 50% of items.\n\nWe present the results in the table below. We select GUME as the primary baseline and conduct evaluations on three datasets. The results show that AGM achieves significant performance improvements on low-frequency items, demonstrating its effectiveness in addressing the long-tail challenge.\n\n| Dataset     | Frequency  | AUC (GUME) | AUC (AGM*) | LogLoss (GUME) | LogLoss (AGM*) |\n|-------------|------------|------------|------------|----------------|----------------|\n| Baby        | Overall    | 0.6832     | 0.6864     | 0.6680         | 0.6656         |\n|             | High Freq  | 0.6861     | 0.6873     | 0.6562         | 0.6544         |\n|             | Mid Freq   | 0.6810     | 0.6847     | 0.6793         | 0.6760         |\n|             | Low Freq   | 0.6752     | 0.6801     | 0.6960         | 0.6936         |\n| Sports      | Overall    | 0.7119     | 0.7145     | 0.6399         | 0.6391         |\n|             | High Freq  | 0.7182     | 0.7211     | 0.6201         | 0.6186         |\n|             | Mid Freq   | 0.7065     | 0.7090     | 0.6575         | 0.6556         |\n|             | Low Freq   | 0.6902     | 0.7010     | 0.6842         | 0.6823         |\n| Elec.       | Overall    | 0.7271     | 0.7310     | 0.5991         | 0.5969         |\n|             | High Freq  | 0.7295     | 0.7333     | 0.5904         | 0.5886         |\n|             | Mid Freq   | 0.7231     | 0.7255     | 0.6122         | 0.6103         |\n|             | Low Freq   | 0.7110     | 0.7169     | 0.6235         | 0.6210         |\n\n**Q3: Performance improvement seems relatively small**  \n\nIn the CTR predictions, an improvement as small as 0.001 in AUC is generally considered significant, as even minor gains in offline metrics can lead to substantial performance boosts in online serving scenarios [1]. For example, in [2], the Wide & Deep model achieved only a 0.275% AUC improvement over logistic regression in offline evaluation, but led to a 3.9% increase in online CTR.\n\n**Q4: Fairness of the evaluation**  \n\nWe sincerely thank you for your thoughtful question. We understand the concern regarding the fairness of the evaluation and offer the following clarifications:\n- Our evaluation protocols follow AlignRec exactly. We clarify that the evaluation protocols used in table 6 follow AlignRec exactly, in both the dataset split and the Intermediate Evaluation Protocol setup. Consistent with AlignRec, we compare the representation quality of different multimodal encoders under the same user behavior-based evaluation.\n \n- AlignRec itself also fine-tunes its MLLM. Different models adopt different strategies for using multimodal features. AlignRec itself includes a fine-tuned MLLM encoder (MMEnc) as part of its pipeline. Similarly, in Table 6, we report results from (i) the base frozen MLLM encoder, (ii) our fine-tuned version, and (iii) our model without the alignment loss. AlignRec’s Table 3 makes an analogous comparison between (i) Amazon, (ii) CLIP, and (iii) their MMEnc, which also shows significant gains from fine-tuning.\n\n- The key difference lies in the fine-tuning strategy employed. Given that both methods generate comparable and consistent multimodal embeddings, the performance gains achieved by AGM over AlignRec can be attributed to the effectiveness of our subsequent representation learning and aggregation mechanisms. Table 2 shows that AGM still outperforms AlignRec, indicating that our performance gains cannot be solely attributed to MLLM fine-tuning. Rather, they result from AGM’s ability to dynamically integrate ID and multimodal features.\n\nFinally, we truly appreciate all your valuable feedback. We hope our responses have addressed your concerns, and we would be grateful if you could take these clarifications into account in your overall evaluation. Thank you again for your time and insights — we look forward to any further comments you may have.\n\n[1] Guo H, Tang R, Ye Y, et al. DeepFM: a factorization-machine based neural network for CTR prediction[J]. arXiv preprint arXiv:1703.04247, 2017.\n\n[2] Cheng H T, Koc L, Harmsen J, et al. Wide & deep learning for recommender systems[C]//Proceedings of the 1st workshop on deep learning for recommender systems. 2016: 7-10."}, {"review_id": "m7krKtMhh4", "reviewer": "Reviewer_2Y2o", "summary": "To address the convergence inconsistency problem in the joint training of ID-based and MLLM-based representations in recommendation systems, the authors propose a two-stage framework: (1) fine-tuning MLLMs to generate unified user/item representations while aligning them with ID features, and (2) introducing an Adaptive Gradient Masking (AGM) strategy to dynamically balance parameter updates between ID and multimodal branches during joint training. The method is evaluated through extensive offline experiments and online A/B tests, demonstrating effectiveness in mitigating convergence issues and improving recommendation performance.", "questions": "1) Could you provide more details about the computational resources required for MLLM fine-tuning? This would help assess practical deployment feasibility.\n2) Have you considered testing your method on recommendation domains beyond e-commerce/short videos (e.g., music or news recommendation) to demonstrate broader applicability?\n3) The paper mentions AGM* improves upon AGM's bias - could you quantify how much performance gain comes specifically from this unbiased variant?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1) The proposed AGM mechanism is innovative, using mutual information to estimate representation contributions and applying non-uniform gradient masking at the sub-network level. The theoretical analysis of AGM's convergence properties adds rigor to the approach.\n2) Comprehensive experimental validation across multiple datasets and backbone architectures demonstrates the method's robustness. The online A/B test results showing significant improvements are particularly convincing for real-world applicability.\n3) The paper provides thorough ablation studies (Table 3) analyzing each component's contribution, and includes a zero-shot evaluation (Table 6) to assess multimodal representation quality, both of which strengthen the methodological claims.\n\nWeaknesses\n1) The computational cost of fine-tuning MLLMs for recommendation tasks isn't adequately discussed. Given the large parameter size of MLLMs, this could be a practical limitation for some applications.\n2) While multiple datasets are used, they're all from similar domains (e-commerce and short videos). Testing on more diverse recommendation scenarios (e.g., news, music) would better demonstrate generalizability.\n3) The comparison with baseline methods (Table 2) could be enhanced by including more recent MLLM-based recommendation approaches beyond those cited.", "was_revised": true, "final_justification": "The authors addressed all my concerns from Weaknesses and Questions, so I lifted my final score.", "rebuttal": "Dear Reviewer 2Y2o,\n\nWe sincerely thank you for your thoughtful feedback and positive score. We truly appreciate the time and effort you dedicated to evaluating our work. Below, we address each of your concerns in detail and provide clarifications where necessary.\n\n**Q1: Computational cost of fine-tuning**  \n\nWe thank the reviewer for raising the concern regarding the computational cost of fine-tuning large MLLMs. As noted in Lines 575–576 of our paper, we adopt Qwen2vl-2b as the backbone model and fine-tune it for 5 epochs across four datasets, using a batch size of 128 on 4 A100 GPUs. This provides a concrete reference for the resource requirements in our setting.\n\nAdditionally, we emphasize that our method adopts a two-stage franework: the MLLM is fine-tuned offline in the first stage to extract high-quality multimodal embeddings, while the downstream recommendation model is trained separately in the second stage. For practical deployment, we cache all item multimodal embeddings, thus eliminating the need for repeated MLLM inference during training or inference. As a result, the computational burden of fine-tuning is incurred only once offline and has minimal impact on online efficiency.\n\n**Q2: Evaluation of more datasets**  \n\nTo further evaluate the generalizability of our method beyond e-commerce and short video domains, we include the MIND-small dataset, a widely-used benchmark for news recommendation.\n\nThe table below summarizes the performance of AGM*, AlignRec and GUME on MIND-small:\n\n| Metric   | AlignRec | GUME   | AGM*   |\n|----------|----------|--------|--------|\n| AUC      | 0.6823   | 0.6835 | 0.6862 |\n| LogLoss  | 0.5934  | 0.5902 | 0.5857 |\n\n**Q3: Comparison with more MLLM methods**  \n\nWe additionally include LIRDRec [1] as a baseline in our evaluation. LIRDRec [1] is designed to learn item representations directly from multimodal content. The following table summarizes the Recall@10 and Recall@20 results across three datasets, which illustrates that AGM* performs on par with or slightly better than LIRDRec and other baselines.\n\n| Dataset     | Metric       | AlignRec | LIRDRec | GUME   | AGM*   |\n|-------------|--------------|----------|---------|--------|--------|\n| Baby        | Recall@10    | 0.0672   | 0.0707  | 0.0671 | 0.0710 |\n|             | Recall@20    | 0.1047   | 0.1068  | 0.1042 | 0.1069 |\n| Sports      | Recall@10    | 0.0758   | 0.0815  | 0.0778 | 0.0813 |\n|             | Recall@20    | 0.1163   | 0.1216  | 0.1166 | 0.1209 |\n| Electronics | Recall@10    | 0.0472   | 0.0493  | 0.0458 | 0.0490 |\n|             | Recall@20    | 0.0699   | 0.0736  | 0.0680 | 0.0742 |\n\n**Q4: Quantification of AGM\\*'s improvement beyond AGM**  \n\nWe provide a clearer quantification of AGM\\*’s improvement over AGM in Table 2. For example, on the Amazon Baby dataset, AUC increases from 0.6852 (AGM) to 0.6864 (AGM\\*), and LogLoss decreases from 0.6683 (AGM) to 0.6656 (AGM\\*). Similar gains are observed across other datasets. We apologize that Table 2 may not have illustrated this improvement clearly enough. In the revised version, we will include a delta row in the table to highlight AGM\\*’s performance gains more explicitly.\n\nFinally, we truly appreciate all your valuable feedback. We hope our responses have addressed your concerns, and we would be grateful if you could take these clarifications into account in your overall evaluation. Thank you again for your time and insights — we look forward to any further comments you may have.\n\n[1] Learning Item Representations Directly from Multimodal Features for Effective Recommendation"}, {"review_id": "4xRfLPbM5h", "reviewer": "Reviewer_a6WN", "summary": "This paper proposes a two-stage framework for recommendation systems that first fine-tunes multimodal large language models (MLLMs) to generate unified user and item embeddings and then introduces Adaptive Gradient Masking (AGM) to dynamically balance gradient updates between ID-based and multimodal representations during joint training. The approach mitigates convergence inconsistency, leading to improved recommendation performance, as demonstrated by theoretical analysis and both offline and online experiments.", "questions": "1. AGM seems to need to calculate the Fisher Information Matrix and dynamic sampling mask. Will it cause unacceptable training/inference delays?\n2. How does AGM perform when one type of information is of poor quality?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Pros:\n1. The author's motivation is clear and reasonable.\n2. The proposed method corresponds intuitively to the motivation.\n3. A theoretical analysis of the method is provided.\n\nCons：\n1. Lack of time complexity analysis.\n2. In the experiment, the author used AUC as the evaluation metric. Adding more recommendation-related evaluation metrics will help to keep consistent with the comparison method.\n3. Regarding Section 4.4 Industrial Application, a more descriptive discussion would help to enhance the persuasiveness.", "was_revised": true, "final_justification": "After reading the rebuttal and other reviews, I would like to maintain my score.", "rebuttal": "Dear reviewer  a6WN,\n\nWe sincerely thank you for your thoughtful feedback and positive score. We truly appreciate the time and effort you dedicated to evaluating our work. Below, we address each of your concerns in detail and provide clarifications where necessary.\n\n**Q1: Time complexity analysis**  \n\nFirst, we clarify that our method adopts a two-stage framework, where the MLLM is fine-tuned offline to extract multimodal features prior to training the downstream model. Since this step is performed once and does not affect online training or inference, so it is reasonable to omit a detailed time complexity analysis for this stage.\n\nWe provide a time complexity analysis of the AGM training process below:\n\nFor a batch of size $|\\mathcal{B}|$, the training time complexity of AGM consists of three components:  \n1.forward propagation: $\\mathcal{O}(|\\mathcal{B}|)$.\n\n2.Fisher Information Matrix (FIM) computation: $\\mathcal{O}(|\\mathcal{B}| \\cdot |\\mathcal{P}|)$.  Here, $\\mathcal{P}$ denotes the selected parameters for update, $\\theta$ denotes the full parameter set, $|\\mathcal{P}| \\ll |\\theta|$.\n\n3.importance-based sampling via heap-based sorting: $\\mathcal{O}(|\\mathcal{P}| \\log |\\mathcal{P}|)$.\n\nThus, we get the total training complexity per batch: $\\boxed{\\mathcal{O}(|\\mathcal{B}| + |\\mathcal{B}| \\cdot |\\mathcal{P}| + |\\mathcal{P}| \\log |\\mathcal{P}|)}$.\n\nAlthough AGM introduces lightweight sampling and importance estimation overhead compared to standard training, its overall complexity remains near-linear with respect to the batch size. Therefore, AGM retains comparable scalability to standard models.\n\n**At inference stage, AGM incurs no additional complexity compared to conventional models, since neither FIM computation nor sampling is involved**. The inference complexity remains $\\mathcal{O}(|\\mathcal{B}|)$, consistent with standard recommendation architectures.\n\n**Q2: More recommendation-related evaluation metrics**  \nIn our current experiments, we adopt AUC and LogLoss as evaluation metrics. because our focus is on click-through rate (CTR) prediction, where these two metrics are widely used. We include Recall@10 and  Recall@20 results in the table below as a complementary evaluation:\n\n| Dataset     | Metric       | AlignRec | GUME   | AGM*   |\n|-------------|--------------|----------|--------|--------|\n| Baby        | Recall@10    | 0.0672    | 0.0671  | 0.0710  |\n|             | Recall@20    | 0.1047    | 0.1042  | 0.1069  |\n| Sports      | Recall@10    | 0.0758    | 0.0778  | 0.0813  |\n|             | Recall@20    | 0.1163    | 0.1166  | 0.1209  |\n| Electronics | Recall@10    | 0.0472    | 0.0458  | 0.0490  |\n|             | Recall@20    | 0.0699   | 0.0680  | 0.7260  |\n\n**Q3: More details of industrial application**  \n\n\nWe deploy our method in a large-scale short-video platform in China, which serves over 300 million DAUs. This application scenario involves high-throughput real-time recommendations across various traffic entry points. Given the immense volume and diversity of video content, real-time efficiency and embedding consistency are critical for system performance. Our proposed pipeline effectively addresses these challenges: by caching item-level multimodal embeddings extracted by the fine-tuned MLLM, we significantly reduce computational overhead during training and inference, enabling scalable deployment under stringent latency constraints. This architecture has been verified to support stable online serving and model updates on a weekly basis.\n\n**Q4: Robustness to low-quality multimodal representations**  \n\nWe agree that the quality of multimodal features can significantly influence recommendation performance. As discussed in Line 688 of the paper, the effectiveness of AGM depends in part on the quality and diversity of multimodal data.\n\nHowever, AGM is designed to remain robust even when one modality is degraded or missing. Specifically, the model includes an ID-based embedding branch and an alignment loss between multimodal and ID-based representations. This setup enables AGM to fall back on reliable ID features when multimodal features are noisy or incomplete, avoiding overfitting to poor-quality inputs.\n\nTo further validate AGM’s robustness to low-quality multimodal features, we conducted ablation experiments using the three types of multimodal representations described in Table 6—(i) the base MLLM encoder without fine-tuning, (ii) our fine-tuned MLLM encoder, and (iii) our variant without the alignment loss—to evaluate the robustness of AGM to degraded or unaligned multimodal inputs. \n\n| Dataset | Metric     |  MLLM (i)     | w/o Lalign (iii)  |  AGM* (ii)  |\n |---------|----------|----------|--------|--------|\n |   Baby    | AUC        |  0.6743  | 0.6804  | 0.6864 |\n |                | LogLoss  |   0.6696  |   0.6683 |  0.6656|\n |   Sports    | AUC      |  0.6907  |   0.6981 |  0.7145|\n  |                | LogLoss  |  0.6450    |   0.6414 | 0.6391 |\n |   Electronics | AUC  |  0.7082    |   0.7160 | 0.7310 |\n |                | LogLoss  |   0.6076  |  0.5985 | 0.5969 |\n\nFinally, we truly appreciate all your valuable feedback. We hope our responses have addressed your concerns, and we would be grateful if you could take these clarifications into account in your overall evaluation. Thank you again for your time and insights — we look forward to any further comments you may have."}, {"review_id": "8RrJ0A8npj", "reviewer": "Reviewer_fBa2", "summary": "The semantic distribution gap between ID features and multimodal representations leads to convergence inconsistency during joint training: the ID branch converges quickly, whereas the multimodal branch requires more training rounds, which limits overall performance. To tackle this issue, the paper proposes a two-stage framework consisting of multimodal representation learning and joint training optimization. Specifically, it first fine-tunes a multimodal large language model (MLLM) to generate unified user and item representations, introducing collaborative signals via post-alignment of user ID embeddings to mitigate semantic discrepancies. Then, it introduces an Adaptive Gradient Masking (AGM) training strategy that dynamically adjusts parameter updates between the ID and MLLM branches. AGM estimates the contribution of each representation via mutual information and applies non-uniform gradient masks at the sub-network level to balance optimization.", "questions": "Please see the Weaknesses section for detailed questions and suggestions.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n* The proposed method effectively addresses the convergence inconsistency between ID and MLLM representations in large-scale recommender systems, leading to improved recommendation performance.\n* The paper introduces Adaptive Gradient Masking (AGM) and its theoretically motivated unbiased variant, AGM\\*.\n* Extensive offline experiments and online A/B testing validate the effectiveness of the proposed approach.\n\n## Weaknesses\n\n* In Section 3.2.1, the authors only describe aligning text-image embeddings within the multimodal item embedding space. Why is there no alignment between the item multimodal embedding and the item ID embedding?\n* Additionally, the backbone function $g()$ operates on $e_u$ and $e_i$ for downstream tasks. How does the UEG module ensure that the learned $e_u^{mm}$ and $e_v^{mm}$ reside in the same space? The training tasks for UEG in Section 3.2.2 rely on aligning user multimodal embeddings and ID embeddings, but this does not guarantee that the final $e_u^{mm}$ and $e_v^{mm}$ are in the same space.\n* Why does Equation (3) use the L2 norm as the alignment loss? Have the authors considered alternative distance metrics?\n* The description in Lines 213–219 is rather vague. For example, “non-uniform adaptive sampling” is not properly introduced. The phrase “This sampling method primarily directs our focus toward parameters carrying richer information” is unclear — is the goal to update these parameters or to preserve them? Finally, what does “probabilistic sampling extends coverage to a broader range of parameters” specifically refer to? Why does this provide broader coverage compared to uniform sampling?\n* Could the authors clarify how the range of $\\eta$ in Line 661 is determined and why $\\eta$ is assigned such a value in Line 667?\n* The notation system in Section 3 is confusing:\n  * In the problem definition in Section 3.1, the dataset $\\mathcal{D}$ uses $u$ and $v$ to indicate clicks $y$, but also uses subscripts $(u_i, v_i, y_i)$, which can lead to inconsistencies if the same user/item appears with different subscripts.\n  * In subsequent descriptions, the notation mixes indexed and non-indexed forms for users/items. For instance, in the formula in Line 185, $\\hat{y}_i$ is used together with unindexed $u$ and $v$.\n  * The variable $x_i$ in Equation (8) is not clearly introduced in context.\n  * …\n* There are minor writing issues throughout the paper:\n  * For example, the abbreviation “MM” in Line 4 should be defined as “multimodal (MM)” upon first use.\n  * There are inconsistent capitalization issues, such as in Line 50.\n  * …", "was_revised": true, "final_justification": "After considering the authors' rebuttal, I find that most of my concerns have been satisfactorily addressed. Specifically: I still have some reservations regarding the semantic consistency between $e_u^{mm}$ and $e_v^{mm}$. While the authors argue that $e_u^{mm}$ is derived from historical interactions involving $e_v^{mm}$, and that the alignment loss encourages compatibility, there remains no explicit modeling component (e.g., joint interaction, dot product, or similarity constraint) that ensures both embeddings reside in the same latent space. This may affect downstream interaction modeling, though empirical results do suggest the method works in practice.\n\n\n  Given that the main methodological concerns have been addressed and the remaining issue is relatively minor and partially mitigated by empirical evidence, I have increased my score from a 3 to a 4. I believe the paper makes a meaningful contribution and could be improved further with minor clarifications and refinements.", "rebuttal": "Dear reviewer fBa2,\n\nWe sincerely thank you for your careful reading and thoughtful feedback. We truly appreciate the time and effort you dedicated to evaluating our work. Below, we address each of your concerns in detail and provide clarifications where necessary.\n\n**Q1: No alignment between item MM and ID represetations**  \n\nUser ID and MM representations are both derived from users’ historical behavior sequences, and thus implicitly encode similar semantics — namely, the user’s preferences. Aligning them helps bridge sparse collaborative signals with rich content-derived features, especially in cold-start scenarios.\n\nIn contrast, item multimodal features represent \"intrinsic, static content semantics\", while item ID embeddings are learned via collaborative signals and are continually updated based on user interactions. These two representations capture fundamentally different information: the former reflects what the item is, the latter who it appeals to. Aligning them may dilute the semantic integrity of content features. Therefore, we align user ID and MM but deliberately keep item ID and MM disentangled.\n\n\n**Q2: Latent Space Between $e^{mm}_u$ and $e^{mm}_v$**  \n\nIn our model,   $e^{mm}_u$ is generated from historical behavior sequences consist of $e^{mm}_v$ , which ensures that they are naturally embedded in a shared latent space compatible for downstream interaction modeling. While the UEG module includes an alignment loss to guide $e^{mm}_u$ toward the ID-based user embedding space, we note that the alignment loss is assigned a small weight in the total objective. This design choice ensures that the core semantic structure of $e^{mm}_u$ remains largely intact and does not significantly deviate from the original MM space.\n\nFurthermore, we provide evidence in Table 6: when we remove the alignment loss, the quality of MM representations drops significantly in the zero-shot recommendation task, which is largely determined by the semantic compatibility between  $e^{mm}_u$ and $e^{mm}_v$. This degradation suggests that adding alignment process between $e^{mm}_u$ and $e^{id}_u$ does not significantly distort the originally consistent semantic space of $e^{mm}_u$ and $e^{mm}_v$, otherwise adding this alignment process would only result in performance loss in MM representation quality in the zero-shot recommendation task. Even if the alignment might slightly influence the original semantic space of $e^{mm}_u$, the benefit of better aligning user representations outweighs any potential disruption, leading to improved recommendation quality.\n\n**Q3: Choice of L2 Norm for Alignment Loss in Equation (3)**  \n\nTo be honest, we chose the L2 norm as the alignment loss just because it provides a simple, smooth, and well-behaved metric that equally penalizes deviations across all embedding dimensions. This decision aligns with many prior works in embedding alignment, such as AlignRec [1].\n\n**Q4: Unclear description in Lines 213–219**  \n\nWe apologize for the confusion and clarify here. The “non-uniform adaptive sampling” refers to an importance-based random selection of parameters during training. Concretely, we bias the sampling toward those with higher scores in distribution $\\pi$ when we choose which parameters to update during the backforward propagation. In other words, parameters carrying more information are updated more frequently. Therefore, The phrase “directs our focus toward parameters carrying richer information” means that we tend to update these important parameters more often (instead of preserve them). \n\nThe second phrase, “probabilistic sampling extends coverage to a broader range of parameters” confused you because the uniform sampling in this occasion is not clear. We apologize for the confusion here. Actully, AGM is fundamentally designed to prioritize the update of more important parameters, and in this context, our “probabilistic sampling” is intended as an improvement over traditional top-k masking strategies commonly used in importance-based sparse updates. Therefore, the sentence“probabilistic sampling extends coverage to a broader range of parameters” means compared to top-k methods, our sampling approach achieves broader coverage of the parameter space.\n\n\n**Q5: $\\eta$ in Lines 661 and 667**  \n\nTo ensure the expected decrease in loss, we consider the worst-case alignment where the bias term $ b(\\theta) $ is in the same direction as the loss gradient $ \\nabla \\mathcal{L}(\\theta) $, i.e., $ b = k \\nabla \\mathcal{L} $.  \nIn this case, the inner product and norm simplify as:  \n$$\n\\langle \\nabla \\mathcal{L}, \\nabla \\mathcal{L} + b \\rangle = (1 + k)\\||\\nabla \\mathcal{L}\\||^2,\\quad \\||\\nabla \\mathcal{L} + b\\||^2 = (1 + k)^2 \\||\\nabla \\mathcal{L}\\||^2.\n$$  \nBy substituting into Eq. (32) (second-to-last line) and neglecting $\\sigma$, we obtain:\n$$\n-\\eta (1 + k) \\||\\nabla \\mathcal{L}\\||^2 + \\frac{\\eta^2 L}{2}(1 + \\nu)(1 + k)^2 \\||\\nabla \\mathcal{L}\\||^2 \\leq 0.\n$$  \nDividing both sides by $ (1 + k)\\||\\nabla \\mathcal{L}\\||^2 $, we get:\n$$\n\\eta \\leq \\frac{2}{L(1 + \\nu)(1 + k)}.\n$$  \nBy conservatively taking $ k = 1 $, we obtain the sufficient condition:\n$$\n\\eta \\leq \\frac{1}{L(1 + \\nu)}.\n$$  \nThis justifies the learning rate constraint adopted in the analysis.\n\nThe upper bound in Line 661, $ \\eta \\leq \\frac{1}{(1 + \\nu)L} $, ensures descent of the expected loss under worst-case bias alignment.  \nIn Line 667, this setting serves two purposes:\n1. It satisfies the sufficient condition for convergence established in Line 661.\n2. It follows the standard choice in stochastic optimization to achieve an $ \\mathcal{O}(1/\\sqrt{T}) $ convergence rate for the average iterate. This rate matches the optimal bound under noisy gradient conditions, and the $ 1/(1 + \\nu) $ scaling further accounts for the additional bias variance introduced in our problem setting.\n\n**Q6: Confusing notation**\n\nWe sincerely apologize for the confusion this may have caused. Regarding the specific points you raised:\n(i) The variable $x_i$ in Equation (8) refers to the current training sample, and corresponds to the tuple ($u_i$, $v_i$). We will clarify this in the revised version.\n(ii) The notation alternates between indexed and unindexed forms (e.g., $u_i$, $v_i$ vs. $u$, $v$), which may lead to ambiguity. In the revision, we will ensure that all sample-specific variables are consistently indexed, and clarify where variables represent arbitrary users/items versus specific samples.\n\nWe appreciate your attention to these important details and will revise Section 3 accordingly to improvev readability and consistency.\n\n**Q7: Writing issues**  \n\nWe sincerely thank you for pointing out the writing and formatting issues, including the missing definition of “MM” (Line 4), inconsistent capitalization (Line 50), and other stylistic inconsistencies. We apologize for these oversights. These corrections will be carefully incorporated into the revised version to ensure clarity, consistency, and overall presentation quality.\n\nFinally, we truly appreciate all your valuable feedback. We hope our responses have addressed your concerns, and we would be grateful if you could take these clarifications into account in your overall evaluation. Thank you again for your time and insights — we look forward to any further comments you may have.\n\n[1] AlignRec: Aligning and Training in Multimodal Recommendations"}]

Author rebuttal:

Meta-review: This paper introduces the Adaptive Gradient Masking technique for recommendation tasks. Initially, the reviewers raised concerns regarding the lack of implementation details, insufficient experimental results, and the computational cost of fine-tuning MLLMs. However, after the rebuttal, the authors successfully addressed most of these concerns, including clarifications of novelty, and all reviewers provided positive feedback. The AC has carefully reviewed the paper, the reviewer comments, and the rebuttal, and agrees that the paper is well-motivated, clearly written, and supported by thorough experiments. Therefore, the AC recommends acceptance. It is encouraged that the authors incorporate all additional experiments and discussions from the rebuttal into the final version.
Decision: Accept (poster)