Catalog-Native LLM: Speaking Item-ID dialect with Less Entanglement for Recommendation
Authors: Reza Shirkavand
Xiaokai Wei
Chen Wang
Zheng Hui
Heng Huang
Michelle Gong
Venue: ICLR 2026 Poster (2026)
Keywords: Recommender Systems
Large Language Models
Mixture of Experts
Primary area: generative models
TL;DR: “IDIOMoE decouples item-ID and language processing inside an LLM via token-type MoE, reducing interference and improving recommendation quality at roughly the same compute.”

Abstract: While collaborative filtering delivers predictive accuracy and efficiency, and Large Language Models (LLMs) enable expressive and generalizable reasoning, modern recommendation systems must bring these strengths together. Growing user expectations, such as natural-language queries and transparent explanations, further highlight the need for a unified approach. However, doing so is nontrivial. Collaborative signals are often token-efficient but semantically opaque, while LLMs are semantically rich but struggle to model implicit user preferences when trained only on textual inputs. This paper introduces  Item-ID + Natural-language Mixture-of-Experts Language Model (IDIOMoE), which treats item interaction histories as a native dialect within the language space, enabling collaborative signals to be understood in the same way as natural language. By splitting the Feed Forward Network of each block of a pretrained LLM into a separate text expert and an item expert with token-type gating, our method avoids destructive interference between text and catalog modalities. 
IDIOMoE demonstrates strong recommendation performance across both public and proprietary datasets, while preserving the text understanding of the pretrained model.

Published as a conference paper at ICLR 2026
CATALOG-NATIVE LLM: SPEAKING ITEM-ID DIALECT WITH LESS ENTANGLEMENT FOR RECOMMENDATION
Reza Shirkavand", Xiaokai Wei", Chen Wang', Zheng Hui* Heng Huang', Michelle Gong?
Universiiyy of Maryland College Park (rezashkv, é umd. edu,
2ooloox (xwei, ewang, ) 0 rooblox... com University of Cambridge zh248 122443@columbia..o
ABSTRACT
While collaborative filtering delivers predictive accuracy and efficiency, and Large Language Models (LLMs) enable expressive and generalizable reasoning, modern recommendation systems must bring these strengths together. Growing pectations, such as natural-language queries and transparent explanations, further highlight the need for unified approach. However, doing is nontrivial. Col- laborative signals are often token-efficient but semantically opaque, while LLMs are semantically rich but struggle to model implicit user preferences when trained only on textual imputs. This paper introduces Item-ID + Natural- language Mixture- of-Experts Language Model (IDIOMoF)), which treats item interaction histories native dialect within the language space, enabling collaborative signals to be understood in the same way as natural language. By splitting the Feed Forward Network of each block of a pretrained LLM into separate text expert and an item expert with token-type gating, our method avoids destructive interference between text and catalog modalities. IDIOMoE demonstrates strong recommendation per- formance across both public and proprietary datasets, while preserving the text understanding of the pretrained model.
INTRODUCTION
Recommendation systems shape what people read, watch, buy, learn, and play. As Al shifts from static predictors to reasoning agents capable of following instructions, recommendation is also evolving from ranking fixed lists to assisting users in exploring, planning, and deciding. This trend is visible in practice: Amazon's 's Rufus provides LLM-powered conversational shopping (Amazon 2024); Meta's Llama-3 assistant is embedded in WhatsApp, Instagram, and Facebook for task planning (Metal 2024); and Netflix is adopting foundation-mode approaches for personalization and LLMM-bsedd conversational retrieval (Netflix, 2025) /Zuu et al.] (025).. These examples motivate bringing LLM knowledge and instruction-followw into recommenders while preserving the collaborative patterns that make them accurate at scale.
Conventional recommenders like collaborative filtering (Creen et al. 2009). ) content-based (CB) et al.) [211), and sequential models (Kang & McAuley| 2018; Sun et al. 2019) Zhai et al.) 2024) perform well within their scope when data are abundant, but they depend heavily on the quality of logs and item attributes. They remain vulnerable to popularity bias (Abdollahpouri et al. 2019). struggle to integrate heterogeneous signals (text, behavior, and context), and cannot support natural language queries.
Pre-trained LLMS offer complementary strengths: they bring broad world knowledge, can follow natural-language instructions, and can reason about multi-objective trade-offs. Yet fundamental gap remains. LLM pretraining centers on semantic understanding, whereas recommendation requires modeling collaborative preference patterns. The key challenge is leveraging LLMs for preference understanding without disrupting their semantic competence.
Recent work has tried to bridge this gap by extending LLM vocabularies with item IDs (Cao et al. 2024 Zhu et al. 2024 Jiang et al. 2025 Zhang et al. 2025). enabling direct ID-level generation.
Work done during internship Roblox

---

Published as a conference paper at ICLR 2026
Jmm temolliting
(a)
(d)
Figure 1: Four designs for recommendation with Transformers/LLMM (a) ID-only Transformer: trained from scratch on item-ID sequences, with pretrained LLM involved. (b) Text-derived bias: pretrained LLM on IDs, with an external text encoder providing side features that bias item scores. (c) Explicit text tokens: pretrained LLM that directly consumes both item-ID tokens and (possibly) text tokens in the same sequence. (d) Explicit text tokens + extra capacity: like (c), but adds item-specific parameters better handle IDs. IDIOMoE is special case of (d).
While effective in principle, such naive integration often causes knowledge interference: collaborative signals entangle with linguistic semantics, leading to degraded performance both sides. As we'll show, this interference does not vanish by simply scaling up parameters (c.g. adding more parameters naively) and thus calls for more principled architectural solutions.
Inspired by mixture-of-experts (MoE) (Shazeer et al. 2017 Lepikhin et al. 2020 Fedus et al. 2022 view ItemID modeling dialect distinct from natural language. But unlike standard MoË, which routes tokens indiscriminately. we design targeted Item-ID Natural lanuuall Mixture-oue Experts Language Model (IDIOMoE) that assigns dedicated collaborative expert for IDs alongside preserved text expert for language. A token-type gate orchestrates their interaction, mitigating interference while retaining pretraining knowledge. Evaluations on both public benchmarks and real-world industrial dataset from a leading online platform with hundreds of millions of users show that IDIOMoE consistently outperforms text-only adapters and item-only baselines. Our main contributions are:
Disentangled MoE architecture for recommendation. We propose a Mixture-of- -fiee-ertts--es design that treats Item-IDs native dialect. To the best of knowledge, this is the first attempt separating collaborative filtering from semantic processing, with a router that activates text experts only when useful.
Robust performance on real-world scale. Our method achieves compelling results on public datasets and on our large proprictary dataset with more hundreds of millions of users, while maintain- ing the natural language understanding of pre-trained L.I.M.
Extensive ablations isolating the source of gains. study model capacity and matched-capacity non-MoE baselines showing that improvements arise from expert specialization and routing, not just added parameters.
Analysis of expert specialization. Through a key-value memory lens of FFN neurons, we show that MoE separation yields clearer item-text affinity, higher category purity, and more clustered neurons than a non-MoE baseline, providing evidence that expert disentanglement leads to more interpretable and modular representations.
RELATED WORK
2.1 CONVENTIONAL RECOMMENDATION METHODS
Traditional recommendation models fall into collaborative filtering (CF), content-based (CB), and sequential paradigms. CF leams from user-item interactions to model latent preferences (Koren et al. 2009, while CB leverages item attributes improve personalization and mitigate cold-start
2

---

Published as a conference paper at ICLR 2026
issues Lops al. 2011 Sequential models further capture temporal dynamics, using models such as RNNs (Hidasi et al. 2013), SASRec (Kang & McAuley 2018). and BERT4Rec (Sun et al. 2019). Though these models achieve strong performance under sufficient data, they operate opaque ID sequences and require hand-crafted features or specialized architectures to incorporate diverse signals like language intent. They also struggle with long-tail exposure (Abdollahpouri et al. 2019).
2.2 GENERATIVE RECOMMENDATION
Some works treat recommendation sequence generation, unifying retrieval and ranking under generative objective Yang et al. 2025). This includes large-scale decoder models such as HSTU (Zhai et al. 2024), which scales trillions of parameters, and OneRec (Deng al.) 2025b), which uses a sparse MoE. encoder-decoder architecture for scalable training. These approaches improve novelty, fluency, and explainability, but essource-intensiv and require careful objective and data design to fully exploit collaborative interaction signals. They also do not support conversational recommendation.
2.2.1 LLM-BASED RECOMMENDATION AND SEMANTIC-ID ALIGNMENT
Large language models (LLMs) offer world knowledge and instructioonfollowiin capabilities that are appealing for building explainable recommenders. Recent frameworks such as P5 (Geng et al. 2022) reframe recommendation tasks as text-to-text generation, supporting few-shot generalization. Prompt-based methods (Hou et al. (024b) further explore LLMs as zero-shot rankers. However, these methods require verbose text inputs and often discard raw user-item interaction data, missing collaborative patterns entirely. To bridge this semantic collaborative gap, prior work fine tunes on interactions (Cao et al. 2024). aligns with rewards (Lu et al., 2024), or unifies modalities in shared token spaces (Zhai al.] [2025). A complementary direction embeds item IDs as tokens (e.g.. CoVE (Zhang et al. 2025). CLLM4Rec (Zhu et al. 2024). URM (Jiang et al. 2025 ), enabling token efficient generation and retrieval. However, designs like URM that drop explicit text tokens, hinder conversational recommendation and instruction handling, And when ID tokens and text tokens share parameters, interference emerges: language and collaborative signals entangle, degrading both.
2.3 MULTIMODAL MoE LLMS
Recent work integrates MoE into multimodal LLMs (MILMs) and LVLMs Bao et al. 2002 Shen et al. 2023): Diao et al. (0025) Deng et al. (025a)). MoE-LLaVA (Lin et al. 2024a) adds a sparse MoE backbone LLaVA (Liu et 2023a), converting feed-forward blocks into experts match exceed larger dense variants while activating fewer parameters and reducing visual hallucinations. Uni-MoE (Li et al. 2025) scales unified multimodal LLMs across many modalities and tasks with MoE layers. MoME (Xu et al. 2024) further mitigates task interference by factorizing the model into a Mixture of Vision Experts (MoVE) and a Mixture of Language Experts (MoLE), with MoVE aggregating multi-encoder vision features via an instruction-conditione router and MoL.E using sparsely gated adapter experts.
2.4 MOTIVATION AND POSITIONING
While prior work has shown the potential of combining semantic understanding with collaborative signals, existing methods lack clear mechanisms to separate and preserve these distinct forms of knowledge. Text can be incorporated via (a) text-as-features (pre-encoded embeddings/biases attached to IDs); or (b) explicil text tokens (Figure []. We choose the latter to preserve conversational capabilities of the I.I.M. In this setting, interference between language understanding and ID-level preference modeling remains undercxplored bottleneck. Simply mixing tokens or scaling capacity does not solve it.
We address this challenge by introducing a Item-ID + Natural-language Mixture-of-Expeer Language Model (IDIOMeE) that treats item interactions as a native dialect. IDIOMoE dedicates separate pathways to item and text processing in each block, with a lightweight token-type gate that reduces interference while retaining language understanding. This design enables efficient ID-level modeling and better alignment with both semantie and collaborative objectives.
3

---

Published as a conference paper at ICLR 2026
Multinodal Large Language Model
Text Emhhedding
Embordding
Varahulary extended Tokenizer
Skamm
The user has Interacted with Reus <R azap ttle Chow a.. zone &.A, <iit 509 tite Broth genre is
Figure 2: Overview of our proposed IDIOMoE. We extend the LLM tokenizer with new "i "it tem-idd tokens and introduce dedicated item embedding layer. The Normalization and Attention layers shared across all token types, while tokens are routed to distinet FFN layers depending on their type.
Table I:
Improvements over the ID-only base- line when adding text features.
Figure 3: Language understanding retention.
Vartant
Industial
HIESS 10
NDCURr1 10
HBK
NDC08
ID only (basoline) only next derved blss explicii atsibrutes
+2.8%
+26
+18.115
+13.9%
24.05
+11.4%
IDOOM
QUARR
3
METHOD
3.1 PRELIMINARY
We study how incorporating item textual attributes affects performance given a user's interaction history. We start from the pretrained Qwen /wwe//eeeeeeeeeeeeeeeeeeeee..... 5-0 5B (Qwen et al.) (025), extend its vocabulary with item-ID tokens, and compare three variants that differ only in input format and the source of item embeddings. In all variants, instruction text tokens are embedded with the LLM's native token embedding matrix.
ID-only (learned embeddings). Input:
"The user has interacted with < -333> </item-11 Each item token is embedded via learned item embedding table.
2. ID-only text-derived bias. Following Jiang et al. 2025) this variant has same input as (a). However, each item token embedding the sum of a learned ID vector and (ii) text-derived vector computed from the item's title and category using general-purpose sentence-embedding model.
3. ID explicit attributes. Input interleaves IDs with attributes: "The user has interacted with < em em-53 title: X, category: Y; </item-11 /> ...". Item-ID tokens use the learned item embedding table; Text tokens embedded by the LLM's token embeddings.
We evaluate the above on two datasets: Amazon-Arts (Ni 2019) and our industrial dataset. The results are presented in [able In both datasets adding item textual attributes improves performance. The text-derived bias approach performs better is easier for the model handle as adds some semantic signal without making the sequence longer more complex. In contrast, giving the model full attribute text makes the input longer and harder learn from. But there is key reason still include explicit text: it enables capabilities that the bias method can't. Conversing with users and generating friendly explanations all rely on having real text.
To evaluate whether the variants preserve the pretrained model's linguistic ability, we measure negative log-likelihood (NLL) on 5,000 samples from the wi k exx validation set (Merity et al. 2016) and further assess performance four benchmarks: BBH (Suzgun et al.) 2022). HellaS wag (Zellers)
et al. 2019), MMLU (Hendrycks et al., 2021). and WinoGrande (Sakaguchi et al. 2019). As shown Figure 3 ID+Text achieves substantially lower NLL and significantly higher benchmark results compared to the text-derived bias variant. While the bias method provides strong recommendation accuracy, does at the cost of language degradation, reflected in much poorer performance
4

---

Published as a conference paper at ICLR 2026
language understanding tasks. This points the need for better approach: that preserves the advantages of explicit text for conversational recommendation while still achieving strong performance on standard recommendation tasks.
In this paper, we propose to divide responsibilities rather than forcing a single model to handle everything. One expert is dedicated to IDs and collaborative filtering, while another is responsible for text. This design allows us to retain the benefits of explicit text when needed, without sacrificing efficiency or accuracy when is not. We show IDIOMoE preserves the language understanding of the model, while delivering the best recommendation performance (TTble Mooleeaane and Figure 3 confirming that separating experts by token type reduces cemantic-collaborate interference.
3.2 IDIOMoE
We present the Item-ID Natural-language cetture-of-Expertt Language Model (IDIOMoE), pretrained decoder-only LLM augmented with item-specialized experts and native item tokens. IDIOMeE keeps the language skills of the base model intact while learning collaborative patterns directly from user-item sequences. We start from a pretrained causal transformer and replace each feed-forward network (FFN) with two-expert module:
Text Expert: the original FFN from the pretrained LLM, preserved as-is.
Item Expert: new FFN similar the text expert, optionally shrunk (e.g., x xt) add capacity efficiently.
Routing is handled by static token-type gate: We use simple static routing scheme: only item-ID tokens < <lit.. >> are routed to the item expert, and all other tokens (titles, attributes, etc.) are routed to the text expert. All tokens share the same self-attention layers at every depth, so IDs and text always attend to each other, and the MoE split only affects the FFN sublayers, i.c., where ID- vs. text-specific information is stored. This design lets the model jointly reason over blended textual attributes and item IDs while allocating separate capacity for catalog structure. Moreover, since only expert is active per token, compute stays comparable to the base model (See Appendix BB6.4 for a discussion of efficiency results). Figure [egurreproove provides an overview of our framework.
3.2. NATIVE ITEM TOKENS AND HYBRID HEAD
We augment the tokenizer with special item tokens < < idd> >> and attach a hybrid embedding layer that combines the frozen text embeddings with trainable item embedding table. The output head reuses the same hybrid parameterization so the model can generate item IDs directly.
3.3
FFN KEY-VALUE MEMORY ANALYSIS
3.3. SETUP
Following Geva et al. 20022, we view each feed-forwant network (FFN) transformer block as key-value memory, where hidden states act as queries and FFN neurons contribute value vectors. Our goal is to probe whether Mixture-of-Expeee item semantics distinct from the text expert, and how this differs from non-MoE baseline.
(MoB) separation encourages the item expert to encode
the FFN consist of two linear projections with activation in between. We denote the second projection wi()) R/xd where / is the FFN hidden dimension and d the model dimension. Each row w E Rd of v=(() treated as value vector associated with neuron in layer é. To study how these rows align with model embeddings, we construct two sets of reference vectors:
For transformer layer (1...., I.).
(f)
Item embeddings: Etems E B. JR-Numxx x taken from the learned item embedding table used for ID tokens.
Text token embeddings: Ejext R Xeu taken from the backbone's 's input embedding matrix for standard vocabulary tokens (excluding items).
5

---

Published as a conference paper at ICLR 2026
Table 2: Results on small Amazon catalogs. Highlight LLM-Based
Bold best; underline
second best; unreported. Zhai et al. 2025). 2
Cao et al. 2024).
Zhang et al. 2025).
Methed
Games
Instruments
Sports
Beanty
Tuya
NDCGOF 20
HR
HRR400
BersARec!
0.0191
0.0347
0.1990
0/400
53 Rech
0.0668
0,9901
0.0133
0.1123
00630
0.1030
0,0240
0,0385
0.0027
0.6647
0.0776
0.0000
TIGER
0.0453
0.0857
0,050
0.1221
0.0806
0.1167
0,0225
0,0400
0.0084
0.0688
0.0432
0.012
VQ VQ-Ree'
0.1329
0,0679
0.0891
0.1357
IJ944
0.1386
MiSSiRea!
0.0499
1065
0.0880
961
00815
01331
PS CID
Q10154
0:0834
0,0704
19
00062
0U999
VIPS
0.0118
0,0758
0,0872
0.0071
0.0355
0.8399
MQLAGReel
0.1548
0,1033
0.1060
0.1327
ReAT"
0.0232
0/0422
0.0835
D10723
010461
0/0776
B4SRtec
0.4100
0,0758
0.0198
IDGunRee
00574
COVE
0JU359
00624
0.0393
0.300
ID Transfrmmz
0.0669
0.7611
0.1025
0.023
0.405
LEM
011534
INOOMeF
Given value vector w E R& compute cosine similarities to both sets:
Siems (w) Eitems
Stex1 Sier(W))
Liext
assuming all vectors are no-mozall We then retrieve the top-k most similar item IDs and text tokens for analysis.
3.3.2 METRICS
We define three metrics to quantify the specialization of each neuron's 's value vector w:
Affinity: a(w) = median Lop-k (w)) - median mediann(see lop-k
(feett (w)),
(2)
1 i E top-k(w) cat(1) c}| E [0,1],
Purity:
p(w)= max Te
Clustered row: Letester [[([w) 2 r]. for threshold - [0, 1).
(4)
Here, C denotes the set of item categories, (i) returns the category of item i, and t controls the strictness of cluster assignment. simple terms, affinity quantifies the relative alignment of an FFN neuron's 's value vector with item versus text embeddings, thereby indicating modality preference. Purity measures the concentration neuron's top-k nearest neighbors within a single item category, reflecting category-specific specialization. Clustered rows are those neurons whose purity exceeds threshold t, identifying dimensions the FFN value space that form coherent category-level clusters.
4 EXPERIMENTS
4.1
EXPERIMENTAL SETTINGS
Baselines Our main focus on LLM-based recommenders, so the most relevant baselines are different ways of adding recommendation capability to LLMs. We include established LLM-for- Rec baselines that are directly comparable to our setting: the P5/P5-CID family, which reframes recommendation as text-to-text generation over pretrained language model (Geng et al.) 2022 Hua et al. 2023): VIP5, a multimodal extension of P5 that adapts the LLM with parameter-efficien modules (Geng al.) 2023); E4SRec, which keeps the LLM largely frozen and adds a lightweight ID-side adapter for sequential recommendation (Li et al. 2023d); and ReAT, which aligns LLMs to recommendation objectives via auxiliary, recommendation speciffic generated tasks (Cao et al. 2024). These capture the main design choices for adding recommendation capability to LLMs (prompting, adapters, frozen-backbon adapters, alignment), and thus form most relevant comparison set. In

---

Published as a conference paper at ICLR 2026
Table 3: Results on large Amazon cat- alogs. Bold=best;
Figure 4: Results on our industrial dataset.
Highlight-LLM-Biee underline=second best;
21.1%
LLM
Method
Beawty
Books
Toys
NDCG9
HRO:0
NDCG@10
HR:610 NDCG
11.0%
SASRes
000101
0.0064
0,0125
GA812
0L004S
0.211
0U0499
Teae LLM
030105
08365
B8095
03829
03316
0.0079
2.7%
0,0032
00119
0,0144
0,051
60009
0.0143
INONEE
0,0132
00228
6/0166
0.361
MJCGG300
Higg1
addition, we compare three variants built on the same backbone: (i) ID Transformer, trained only on item tokens; (ii) Item-ID I.IM + text-derived bias (Jiang al. 2025 where IID embeddings are augmented with text features; and (iii) Item-LLMM, which integrates item text via vocabulary expansion but without MoE. These three variants are matched IDIOMoE in parameter count and trained under identical token budgets. For completeness, we also report results of classical sequence models (GRU4Rec (Hidasi et al. 2015), Bert4Rec (Sun et al., 2019). FDSA (Zhang et al.) 2019). S3-Rec (Zhou et al.] (0200), recent quantized/contrastive approaches (VQ-Rec (Hou et al.
2023b), MissRec (Wang et al.] 2023a), TIGER (Rajput al. 2023). MQL4GRee (Zhai et al. 2025
IDGenRec (Tan et al. 2024)), and strong transformer baselines (SASRec (Kang & McAuley 2018
HSTU (Zhai et 2024)). We further include CoVE (Zhang et al. 2025), which extends an L.LM with LoRA parameters to encode catalog items. While these embedding driven or classical models are not our primary comparison targets, we include them for completeness on smaller Amazon datasets. Full baseline details are in Appendix B.1
Datasets, Evaluation, & Backhone We use public Amazon Dataset: Games, Instruments and Arts (Ni et al.] 2019) as well as Sports, Beauty and Toys McAuley et al. (2015). We further report performance larger 2023 Amazon variants (Beauty, Books, and Toys) with substantially larger item vocabularies Hou et al. (2024)). We also train and evaluate on our in-house industrial-scat dataset with hundreds of millions of users and tens of thousands of items. We report NDCG@ 10, HR @ 10 and MRR. Metrics are computed over the full catalog on Amazon datasets and on 50000 samples in our industrial dataset. We follow the standard leave last item out procedure for separating train and test datasets. All LLM-based models that we train, use Qwen Qwen/Qwen2 Qwen2. 55-0 5B on text-analysis results, Amazon datasets. and for all ablations. We use owen/Owen2 5-1. 5B for main results on our proprictary dataset. See Appendix B for all details.
4.1.1 RESULTS: AMAZON CATALOGS
Table summarizes performance across six small Amazon datasets. We observe that classical sequence models such as GRU4Rec (Hidasi et al. 2013) and Bert4Rec [Sunnn et al. (2019) perform consistently worse than more recent architectures, confirming the difficulty of modeling sparse item interactions in these settings. Transformer- based methods with additional inductive biases, such FDSA (Zhang et al., 2019), S3-Rec /Zhou al. (2020), and TIGER Rajpptt et al.) (2023). provide moderate gains, while recent quantization and multi-mode modal approaches like VQ-Rec Hou al. (2023b). MISSRec Wang et al. (2023a), and MQLAGRec Zhai al. (2025) achieve stronger results. Compared to direct LLM- Based baselines (highlighted in gray) and classical sequence models, IDIOMoE delivers the most consistent improvements: it achieves the highest NDCG@ @ 10 and HR@ @ 10 in nearly all domains. These results highlight the robustness of our approach across diverse catalog sizes and domains, suggesting better generalization than prior methods that either overfit to specific datasets or fail to transfer across settings.
We evaluate SASRec (Kang & McAuley 2018), HSTU (Zhai et al. 2024). ID- ID-Trssfform,r, L.MM- based baselines and IDIOMoE on Larger Amazon datasets. Table 3 presents the results. IDIOMME is the strongest LLM-based method across all three catalogs: it is the top LLM on Beauty (2nd overall behind HSTU by a small margin), and it achieves the best overall scores Books and Toys. In contrast, Reem-LLM and Text-Attr LLM Jiang et al. (0255) lag behind IDIOMoE across metrics, indicating that simply mixing item/text tokens adding text-derived biases is insufficient. Together, these results support our claim that disentangling item and language pathways yields higher ranking quality than prior L.LM baselines while remaining competitive with the best non-LLM models.
7

---

Published as a conference paper at ICLR 2026
4.1.2 RESULTS: PROPRIETARY DATASET
While results on the Amazon datasets remain useful reference point, we acknowledge their limi- lations. The benchmarks are relatively small and may contain overlaps that make them easier than real-world scenarios. Therefore, although we report results on these datasets for comparability with prior work, we place greater weight on evaluations conducted on our large-scale proprietary dataset, which we consider a more realistic and meaningful test of recommendation quality.
Figure 4 (Table 9 shows results on our large-scale proprietary dataset as improvements over the SASRec Kang & McAuley (2018) baseline. ID-Transformer achieves strong gains, confirming that transformers can effectively capture collaborative filtering signals when grounded in IDs and given enough compute. Title-LLM, which relies solely on free-form item titles, collapses in performance, highlighting the limitations of text-only representations for recommendation. Item-LLM combines
IDs with textual features and yields further improvements, particularly on HR 10, demonstrating the value of jointly modeling collaborative and semantic signals. HSTU Zhai et al. (2024) provides modest gains but falls short compared to the L.-M-based approaches and doesn't support explainable recommendation. Finally,
method (IDIOMoE) achieves the largest improvements across all metries (+27.1 1% NDCG@ 10, +16.6% HR @ 10, +31 .2% MRR), showing that disentangling ID and text processing with specialized experts and routing not only preserves collaborative filtering strength but also better leverages semantic cues for robust sege-sccae recommendation.
4.2 ABLATIONS
4.2.1 NON-MoE CAPACITY CONTROLS.
To ensure that the improvements of IDIOMoE are not simply due to added parameters, we compare against non-MoE variants with matched capacity. Specifically, we consider three settings: (i) wide- FFN, where the feed-forwaid layers of the transformer blocks are widened to match IDIOMoE parameter count; (ii) sapend-blocks where additional transformer layers are added after the original stack; and (iii) prepend-blosd -lockks, where extra layers are inserted before the original stack. All models trained under the setup IDIOMoE with the hyperparameters and the same FLOPS. We also compare against a LoRA Hu et al. (2022) variant where low-rank adapters are added across all layers. Table summarizes the results.
We find that simply adding parameters in non- structured ways is insufficient. Wide-FFN im- proves performance on Amazon-Beauty but only marginally helps in the industrial set- ting. In contrast, append-blocks and prepend- blocks severely degrade performance across both datasets, likely due to disruption of pre- trained representations or training instability. LoRA-LLM, where low-rank adapters are added across all layers, helps slightly on Amazon- Beauty but fails drastically the industrial benchmark, highlighting its sensitivity to scale and signal sparsity.
Table 4: Non-MoE capacity controls Amazon- Beauty and Industrial datasets. All variants are matched to IDIOMoE in parameter count. Re- sults are shown as relative improvements over Item- L.M.
Method
Amazon Beauty 2(%)
Industrial (1%)
NDX00000
#R
NDCG0110
HR
Biem-LLM (buseline)
LoRA-LLM
+21.5%
+7.9%
-79.1%
-76.3%
WINE-FN
+27.0%
+24.9%
+3.8%
11.3%
Append-Mlacks
-87.8%
-90.3%
-5.5%
Prepend-
-97.2%
-95.9%
-15.3%
-16.2%
MoA
+8.33
-46.2%
+20.95
+27.1%
MoT
+49.3%
We also compare with various MoE designs. Both MoA (expert attention modules) and MoT (expertized full transformer blocks with cross attention) yield large improvements over all non-MoE controls. Importantly, IDIOMoE performs on par or better than both, despite using a simpler and more efficient expert design focused solely on FFNs with static routing. Although MoA and MoT are competitive on amazon-Beautg and occa- sionally match or slightly exceed IDIOMME there, we emphasize the industrial results as our primary evidence. On this large setting, the FFN-based MoB IDIOMoE consistently outperforms MoA/MoT variants. Nonetheless, the pattern we observe might be dataset-dependent. The core idea of IDIOMoE is to treat catalog items as first-class citizens and to separate where information about IDs and text is stored. All three MoE variants we ablate are consistent with this idea. Our choice to place MoE in the FFNs is guided by the stronger and more stable gains we see on the large-scale industrial dataset.
IDIOMOE
148.1%
+24.1%
+28.9%
8

---

Published as a conference paper at ICLR 2026
These results confirm that IDIOMoE's performance is not due to parameter count, but rather due to its intentional separation of item and language processing via token-type MoE routing. Compared to generic scaling or lightweight tuning (e.g., LoRA), the structured, disentangled pathways in IDIOMoE yield higher accuracy, especially in large-scale settings where interference between item IDSs and natural language is more pronounced.
4.2.2 ITEM EXPERT CAPACITY
We vary the intermediate width of the item expert per layer by applying different shrink factors to the middle layer the item FFN experts. Larger shrink factors reduce the parameter count and latency, but they also constrain the model's ability to capture rich collaborative signals. Ta- ble 5 presents the results. On Amazon- Amazon-Beauty, we see that moderate shrink values (2 and 4) provide substantial improvements capacity and efficiency (+41 18% NDCG@ 10, +26.6% HR 10). However, very aggressive shrink- ing (shrink=8) reduces gains, suggesting that the item expert becomes under-paramettrized... In contrast, results on the industrial dataset show a different trend: shrinking consistently hurts per- formance, with small but steady drops in both NDCG@ 10 and HR@ 10 as capacity decreases. These findings indicate that while smaller bench- marks can benefit from lighter experts, large- scale real-world data demands higher item- expert capacity to preserve recommendation ac- curacy. This motivates the need for adaptive capacity allocation, where expert width can be tuned to match the complexity and scale of the target domain. Our method provides this control on capacity allocation.
the baseline, with shrink=4 yielding the best balance of
Table 5: Impact of varying item expert capacity.
Amazon-Beaute Beauty A
Industrial (%)
Shrink
NDCG@10 10
HR@10
NDCG@10
HR@10 10
1 (baseline)
2
+21
+23.3%
-2.0%
-211%
4
+41.8%
+26.6%
-3.1%
-2.2%
8
+10.1%
4.5%
-3.6%
4.2.3 WHERE TO INSERT MOE LAYERS
To study where MoE layers are most effective, we conduct ablation by selecting different insertion strategies. Specifically, we activate MoE experts (i) the first 8 layers, (ii) the middle 8 layers, (iii) the last 8 layers, and (iv) every third layer throughout the model. This allows to compare the impact placing MoE capacity in shallow, intermediate, deep, or evenly distributed positions. We report results the Amazon-Arts dataset in Table 6
We observe clear differences depending on where MoE layers inserted. Using MoE in the first 8 layers yields the weakest perfor- mance, suggesting that early representations dominated by low-level token processing where additional capacity is less beneficial. Distribut- ing MoE every three layers achieves moderate improvements still falls short. Placing MoE in the middle 8 layers improves results, but the largest gains come from inserting MoE in the last 8 layers (+27.6% HR @ 10 and +28.4% NDCG@ 10 over baseline). This indicates that deeper layers (where task-specific semantics and collaborative filtering patterns are most prominent) benefit most from specialized experts, they directly shape the final ranking representations.
Table 6: Ablation on where to insert MoE layers.
Amazon Amazon-Beauty A(%)
Industrial A(%)
MoE Placement
NDCG@10
HR
NDCG@10 HR 10
First (huseline) Every Last
+17.7%
+10.3%
+2.0%
+5.3%
422
+17.2%
+3.1%
+6.9%
+28.4%
+27.6%
+9.6%
+9.0%
4.2.4 STATIC VS. DYNAMIC ROUTING
We find that a switch-style (Fedus et al. 2022) dynamic gating severely degrades recommen- dation quality, while static token-type routing performs much better (T) The likely rea- son is that static routing gives each expert clear, consistent role (language vs. item IDs) so they specialize without interference. In contrast, dynamic routing mixes assignments across ex- perts, leading greater entanglement between signals and weaker specialization. This highlights that
Table 7: Impact of static routing.
Amazon- Beauty 4(%)
Industrial ((%)
Routing Strategy
NDCG@10
NDCG@100 HR
Static Dynamic
-59.5%
.36.9
-24.2%
-24.4%

---

Published as a conference paper at ICLR 2026
Ieet Afreyy (nwar)
Not (main)
Pdf
(rzar)
Cleee Farty
(a) Amazon-Are
(b) Industrial dataset
Figure 5: FFN key-value memory analysis comparing MoE vs. non-MoF. Each subfigure shows
item-text affinity, cluster purity, and fraction clustered rows across transformer layers.
fixed separation by token type is not just simpler but also more effective for disentangling language and recommendation signals.
For each layer f, we report means/medians of a(w) (Equation 2 and p(w) (Equation 3) across rows, and the clustered fraction E[]cslsttr( (w)] (Equation 4.. In MoE, we compare the item expert. We extract W rows, compute top-k similarities to items and text, and summarize per layer and overall. We set 20 and 7=0.5.
4.3 FFN VALUE MEMORY ANALYSIS
The results in Figure show clear differences between MoE and non-MoE models when analyzing FFN neurons as key-value memories. In terms of item-text affinity, both models begin with weak modality preference, but deeper layers of the non-MoE. baseline drift toward negative affinity (favoring text), whereas the MoE model maintains more balanced alignment. This indicates that MoE preserves item sensitivity in upper layers, where recommendation decisions most critical (Table 6
For cluster purity, MoE consistently yields higher values across layers, meaning that its neurons are category-specifie: when neuron activates for items, tends retrieve items from the same category. Similarly, the fraction of clustered rows (neurons forming coherent category-level clusters) remains low and flat for the non-MoE baseline, is always higher in MoE and rises sharply in the later layers of MoE on the more challenging industrial dataset. Together, these results suggest that MoE separation leads to clearer item-text specialization, higher category purity, and more structured clustering than vanilla transformer, reinforcing our claim that expert separation enables more interpretable and modular representations of recommendation signals.
CONCLUSION
We introduced IDIOMoE, a dual-expert continued- prettaiied language model that processes text and item data through two specialized experts. Despite its simplicity, IDIOMoE outperforms both classical and recently proposed LLM-based recommendation models. It effectively preserves the pretrained knowledge of the LLM. Our findings highlight the importance of using specialized sub-networks for different modalities, rather than scaling indiscriminately with a single model for all inputs. We view IDIOMoE step toward more sustainable and adaptive LLMs for recommendation tasks, and believe this direction is crucial in our efforts to achieve better recommendation performance and interpretability without relying unnecessarily large models that exhibit diminishing returns.
6 ACKNOWLEDGMENTS
The authors would like to thank Zahra Miri for her assistance in preparing the figures. This work was partially supported by NSF IIS 2347592, 2348169, DBI 2405416, CCF 2348306, CNS 2347617, RISE 2536663.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Himan Abdollahpouri, Robin Burke, and Bamshad Mohasher. Managing popularity bias in recom- mender systems with personalized re-ranking. arXiv preprint arXiv: 2019.
Amazon. Amazon's rufus ai assistant now available to all u.s. customers, 2024. URL https:
/7 7/www. about amazon. com/ http://o....o///////aaaaa//oo-// -ww.sse-aaaa--aaa-- rufus Accessed
2025-08-28.
Hangbo Bao, Wenhui Wang. Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, and Furu Wei. Vlmo: Unified vision-language pre-training with mixture-of mixture-of-modality modality-experts, 2022. URL arxiv. org/ abs http:////////...////222222 02358
Keqin Bao, Jizhi Zhang, Wenjie Wang, Yang Zhang, Zhengyi Yang, Yancheng Luo, Chong Chen, Fuli Feng, and Qi Tian. A bi-step grounding paradigm for large language models recommendation systems, 2023a.
Keqin Bao, Jizhi Zhang. Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. Tallrec: An effective and efficient tuning framework to align large language model with recommendation. In Proceedings of the 17th ACM Conference Recommender Systems. ACM, 2023b.
Yuwei Cao, Nikhil Mehta, Xinyang Yi, Raghunandan Keshavan, Lukasz Heldt, Lichan Hong, Ed H Chi, and Maheswaran Sathiamoorthy. Aligning large language models with recommendation knowledge. arXiv preprint arXhv:2404.00245 2024.
Aldo Gael Carranza, Rezsa Farahani. Natalia Ponomareva, Alex Kurakin, Matthew Jagielski, and Milad Nasr. Synthetic query generation for privacy-preserving deep retrieval systems using differentially private language models, 2024.
Qibin Chen, Junyang Lin, Yichang Zhang, Ming Ding, Yukuo Cen, Hongxia Yang, and Jie Tang. Towards knowledge-based recommender dialog system, 2019.
Emile Contal and Garrin McGoldrick. Ragsys: ttem-cold-star recommender as rag system, 2024.
Zeyu Cui, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. M6-rec: Generative pretrained language models are open-ended recommender systems, 2022.
Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongxiang Sun, Xiao Zhang, and Jun Xu. Uncovering chatgpt's capabilities in recommender systems. In Proceedings of the 17th ACM Conference on Recommender Systems. ACM, 2023.
Gabriel de Souza Pereira Moreira, Sara Rabhi, Jeong Min Lee, Ronay Ak, and Even Oldridge. TransformersArec: Bridging the gap between nip and sequential / session-based recommendation. In Proceedings of the 15th ACM Conference on Recommender Systems. Association for Computing Machinery, 2021.
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang. Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan. Emerging properties in unified multimodal pretraining, 2025a. URL
//aarvv.aa/ii////aa org/ abs 2505. 14683
Jiaxin Deng, Shiyao Wang, Kuo Cai, Lejian Ren, Qigen Hu, Weifeng Ding, Qiang Luo, and Guorui Zhou. Onerec: Unifying retrieve and rank with generative recommender and iterative preference alignment. arXiv preprint arXiv:2502. 18965, 2025b.
Haiwen Diao, Xiaotong Li, Yufeng Cui, Yueze Wang. Haoge Deng, Ting Pan, Wenxuan Wang, Huchuan Lu, and Xinlong Wang. Evev2: Improved baselines for encoder-free vision-language models, 2025. URL htthttps:: arxiv. wrw. hwtp:///bb////....e///
06788
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
Luke Friedman, Sameer Ahuja, David Allen, Zhemning Tan, Hakim Sidahmed, Changbo Long, Jun Xie, Gabriel Schubiner, Ajay Patel, Harsh Lara, Brian Chu, Zexi Chen, and Manoj Tiwari. Leveraging large language models in conversational recommender systems, 2023.

---

Published as a conference paper at ICLR 2026
Junchen Fu, Fajie Yuan, Yu Song. Zheng Yuan, Mingyue Cheng, Shenghui Cheng. Jiaqi Zhang, Jie Wang. and Yunzhu Pan. Exploring adapter-based transfer learning for recommender systems: Empirical studies and practical insights. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining. ACM, 2024.
Alireza Ganjdanesh, Reza Shirkavand, Shangqian Gao, and Heng Huang. Not all prompts are made equal: Prompt-based pruning of text-to-image diffusion models. In The Thirteenth International Conference on Learning Representations, 2025.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Alain Le Noac"h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 2024.
Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). In Proceedings of the 16th ACM conference on recommender systems, pp. 299-315, 2022.
Shijie Geng, Juntao Tan, Shuchang Liu, Zuohui Fu, and Yongfeng Zhang. Vip5: Towards multimodal foundation models for recommendation. arXiv preprint arXiv:2305. 0..305//4302. /4302, 2023.
Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprint ar 2163 2022.
Ruidong Han, Bin Yin, Shangyu Chen, He Jiang, Fei Jiang, Xiang Li, Chi Ma, Mincong Huang, Xiaoguang Li, Chunzhen Jing, et al. Mtgr: Industrial-scale generative recommendation framework in meituan. arXiv preprint arXiv:2505. 18654, 2025.
Ruining He and Julian McAuley. Vbpr: Visual bayesian personalized ranking from implicit feedback. 2015.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-basee recommendations with recurrent neural networks. arXiv preprint arxiv:1511.06939, 2015.
Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. Towards universal sequence representation learning for recommender systems, 2022.
Yupeng Hou, Zhankui He, Julian McAuley, and Wayne Xin Zhao. Learning vector-quantized item representation for transferable sequential recommenders, 2023a.
Yupeng Hou, Zhankui He. Julian McAuley, and Wayne Xin Zhao. Learning vector-quantized item representation for transferable sequential recommenders. In Proceedings of the ACM Web Conference 2023, pp. 1162-1171, 2023b.
Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian McAuley. Bridging language and items for retrieval and recommendation. 2024a.
Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. Large language models are zero-shot rankers for recommender systems. In European Conference on Information Retrieval, pp. 364-381. Springer, 2024b.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Wenyue Hua, Shuyuan Xu, Yingqiang Ge, and Yongfeng Zhang. How to index item ids for recom- mendation foundation models. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, pp. 195-204
2023.
12

---

Published as a conference paper at ICLR 2026
Junguang Jiang, Yanwen Huang, Bin Liu, Xiaoyu Kong. Xinhang Li, Ziru Xu, Han Zhu, Jian Xu, and Bo Zheng. Large language model as universal retriever in industrial scale recommender system. arXiv preprint 2XXv:25522.0341,, 2025.
Wang-Cheng Kang and Julian McAuley. Self-attentive sequential recommendation. In 2018 IEEE international conference on data mining (ICDM), pp. 197-206. IEEE, 2018.
Sara Kemper, Justin Cui, Kai Dicarlantonio, Kathy Lin, Danjie Tang, Anton Korikov, and Scott Sanner. Retrieval-augmenner conversational recommendation with prompt-based semi-structured natural language state tracking. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2024.
Hai-Dang Kieu, Minh Duc Nguyen, Thanh-Son Nguyen, and Dung D. Le. Keyword-driven retrieval- augmented large language models for cold-start user recommendations, 2025.
Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, 42(8):30-33) 07.30-37,220 2009.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding, 2020.
Jinming Li, Wentao Zhang, Tian Wang, Guanglei Xiong, Alan Lu, and Gerard Medioni. Gpt4rec: A generative framework for personalized recommendation and user interests interpretation. arXiv preprint arXiv:2304.00 03387, 2023a.
Peibo Li, Maarten de Rijke, Hao Xue, Shuang Ao, Yang Song, and Flora D. Salim. Large language models for next point-of-interest recommendation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2024a.
Raymond Li, Samira Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. Towards deep conversational recommendations, 2019.
Ruyu Li, Wenhao Deng, Yu Cheng, Zheng Yuan, Jiaqi Zhang, and Fajie Yuan. Exploring the upper limits of text-based collaborative filtering using large language models: Discoveries and insights, 20236.
Tianyu Li, Ali Cevahir, Derek Cho, Hao Gong, DuyKhuong Nguyen, and Bjom Stenger. (ber(b Self-supervised user representation learning. 2021.
Xiangyang Li, Bo Chen, HuiFeng Guo, Jingjie Li, Chenxu Zhu, Xiang Long, Sujian Li, Yichao Wang, Wei Guo, Longxia Mao, Jinxing Liu, Zhenhua Dong, and Ruiming Tang. Inttower: the next generation two-tower model for pre-ranking system, 2022.
Xiangyang Li, Bo Chen, Lu Hou, and Ruiming Tang. Ctrl: Connect collaborative and language model for ctr prediction, 2023c.
Xinhang Li, Chong Chen, Xiangyu Zhao, Yong Zhang, and Chunxiao Xing. E4srec: An elegant effective efficient extensible solution large language models for sequential recommendation. arXiv preprint arXiv:2312.02445) 2023d.
Yaoyiran Li, Xiang Zhai, Moustafa Alzantot, Keyi Yu, Ivan Vulić, Anna Korhonen, and Mohamed Hammad, Calrec: Contrastive alignment generative Ilms for sequential recommendation. arXiv, 2024b.
Yunxin Li, Shenyuan Jiang, Baotian Hu, Longyue Wang, Wanqi Zhong, Wenhan Luo, Lin Ma, and Min Zhang. Uni-moe: Scaling unified multimodal Iims with mixture of experts. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025.
Yueqing Liang, Liangwei Yang, Chen Wang, Xiongxiao Xu, Philip S. Yu, and Kai Shu. Taxonomy- guided zero-shot recommendations with llms, 2025.
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang, Munan Ning, et al. Moe-Ilava: Mixture of experts for large vision-lane laiion-la models. arXiv preprint 4rXiv:2401.159.. 2024a.
13

---

Published as a conference paper at ICLR 2026
Guo Lin and Yongfeng Zhang. Sparks of artificial general recommender (agr): Early experiments with chatgpt, 2023.
Jianghao Lin, Xinyi Dai, Yunjia Xi, Weiwen Liu, Bo Chen, Hao Zhang, Yong Liu, Chuhan Wu, Xiangyang Li, Chenxu Zhu, Huifeng Guo, Yong Yu, Ruiming Tang, and Weinan Zhang. How can recommender systems benefit from large language models: A survey, 2024b.
Guang Liu, Jie Yang, and Ledell Wu. Ptab: Using the pre-trained language model for modeling tabular data, 2022.
Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023a.
Junling Liu, Chao Liu, Peilin Zhou, Renjie Lv, Kang Zhou, and Yan Zhang. Is chatgpt a good recommender? preliminary study, 2023b.
Peng Liu, Lemei Zhang, and Jon Atle Gulla. Pre-train, prompt and recommendation: comprehensive survey of language modelling paradigm adaptations in recommender systems, 2023c.
Pasquale Lops, Marco de Gemmis, and Giovanni Semeraro. Content-based Recommender Systems: State of the Art and Trends, pp. 73-105. Springer US, Boston, MA, 2011. ISBN 978-0-387-85820- 3.
Kezhi Lu, Qian Zhang, Guangquan Zhang, and Jie Lu. Bert-rs: A neural personalized recommender system with bert. In Machine Learning, Multi Agent and Cyber Physical Systems: Proceedings of the 15th International FLINS Conference (FLINS 2022), pp. 390-397. World Scientific, 2023.
Wensheng Lu, Jianxun Lian, Wei Zhang, Guanghua Li, Mingyang Zhou, Hao Liao, and Xing Xie. Aligning large language models for controllable recommendations. arXiv preprint arXiv:2403.088 050663 2024.
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relationships in multi-task task learning with multi-gate mixture-of cocuure-oo-eeeee -f-experss In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. pp. 1930-1939, 2018
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based recommendations on styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pp. 43-52, 2015.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Meta. Meet your assistant: Meta ai, built with llama 3, 2024. URL https: /about Fb. 2025-08-28.
wawssssistntt-uullll wwwuubbttttttttttteeeee meath-llanne
ama 11ama
3/ Accessed
Aashiq Muhamed, Iman Keivanloo, Sujan Perera, James Mracek, Yi Xu, Qingjun Cui, Santosh Ra- jagopalan, Belinda Zeng, and Trishul Chilimbi. Ctr-bert: Cost-effective knowledge distillation for billion-parameter teacher models. In NeurIPS Efficient Natural Language and Speech Processing Workshop, 2021.
Netflix. 2025.
Foundation
model
for
recommenda- httpss//nettllxxeccbbbog com/
personalized
tion,
URL
iiaandatio-moodd -or-l-ffr-prrsooe - www.rreccomeedatttoooaaaaaeeeeae Accessed 2025-08-28.
Jianmo Ni, Jiacheng Li, and Julian McAuley, Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Proceedings of the 2019 conference empirical methods in natural language processing and the 9th international joint conference on natural language processing GEMNLP-IJCNLP) pp. 188,197,10 2019.
Zhaopeng Qiu, Xian Wu, Jingyue Gao, and Wei Fan. U-bert: Pre-training representations for improved recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021.
14

---

Published as a conference paper at ICLR 2026
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei 7hang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan Hulikal Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Tran, Jonah Samost, et al. Recommender systems with generative retrieval. Advances in Neural Information Processing Systems, 36: 10299-10315, 2023.
Ruchit Rawal, Reza Shirkavand, Heng Huang, Gowthami Somepalli, and Tom Goldstein. Argus: Hallucination and omission evaluation video-Ilms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20280-20290, 2025.
Xubin Ren and Chao Huang. Easyrec: Simple yet effective language models for recommendation, 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-exxprrs layer, 2017.
Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. Scaling vision-language models with sparse mixture of experts, 2023. URL abs/2303. 07226
Kyuyong Shin, Hanock Kwak, Su Young Kim, Max Nihlen Ramstrom, Jisu Jeong, Jung- Woo Ha, and Kyung-Min Kim. Scaling law for recommendation models: Towards general-purgee purpose user representations, 2022.
Reza Shirkavand, Shangqian Gao, Peiran Yu, and Heng Huang. Cost-aware contrastive routing for 1lms. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025a.
Reza Shirkavand, Peiran Yu, Shangqian Gao, Gowthami Somepalli, Tom Goldstein, and Heng Huang. Efficient fine-tuning and concept suppression for pruned diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18619-18629, June 2025b.
Reza Shirkavand, Peiran Yu, Qi He, and Heng Huang. Bilevel zofo: Efficient I1m fine-tuning and meta-training. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025c.
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management, pp. 1441-1450, 2019.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.
Juntao Tan, Shuyuan Xu, Wenyue Hua, Yingqiang Ge, Zelong Li, and Yongfeng Zhang. Idgenrec: Lim-recsys alignment with textual id learning. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, 2024.
Hongyan Tang, Junning Liu, Ming Zhao, and Xudong Gong. Progressive layered extraction (ple): A novel multi-task learning (mtl model for personalized recommendations. In Proceedings of the 14th ACM Conference Recommender Systems. Association for Computing Machinery, 2020.
15

---

Published as a conference paper at ICLR 2026
Min Tang, Shujie Cui, Zhe Jin, Shiuan-ni Liang, Chenliang Li, and Lixin Zou. Sequential recom- mendation by reprogramming pretrained transformer. Inf. Process. Manage., 62,1), January 2025.
ISSN 0306-4573.
Jiachun Wang, Fajie Yuan, Jian Chen, Qingyao Wu, Min Yang, Yang Sun, and Guoxiao Zhang. Stackrec: Efficient training of very deep sequential recommender models by iterative stacking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2021.
Jie Wang. Fajie Yuan, Mingyue Cheng, Joemon M. Jose, Chenyun Yu, Beibei Kong, Zhijin Wang. Bo Hu, and Zang Li. Transrec: Learning transferable recommendation from mixture-of -f-ooaalittyy feedback, 2025a.
Jinpeng Wang, Ziyun Zeng, Yunxiao Wang, Yuting Wang, Xingyu Lu, Tianxiang Li, Jun Yuan, Rui Zhang, Hai-Tao Zheng, and Shu-Tao Xia. Missrec: Pre-training and transferring multi-modal interest-aware sequence representation for recommendation. In Proceedings of the 31st ACM International Conference on Multimedia, pp.
2023a.
Kaishen Wang, Hengrui Gu, Meijun Gao, and Kaixiong Zhou Damo: Decoding by accumulating activations momentum for mitigating hallucinations in vision-language models. In The Thirteenth International Conference on Learning Representations, 2025b.
Lei Wang, Songheng Zhang, Yun Wang, Ee-Peng Lim, and Yong Wang. Llm4vis: Explainable visualization recommendation using chatgpt, 2023b.
Wenjie Wang. Xinyu Lin, Fuli Feng, Xiangman He, and Tat-Seng Chua. Generative recommendation:
Towards cext-generation recommender paradigm, 2024a.
Xu Wang, Jiangxia Cao, Zhiyi Fu, Kun Gai, and Guorui Zhou. Home: Hierarchy of multi-gate experts for multi-task learning kuaishou, 2024b.
Wei Wei, Xubin Ren, Jiabin Tang, Qinyong Wang, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. LImrec: Large language models with graph augmentation for recommendation. 2024.
Xuansheng Wu, Huachi Zhou, Yucheng Shi, Wenlin Yao, Xiao Huang, and Ninghao Liu. Could small language models serve as recommenders? towards data-centric cold-start recommendations, 2024.
Chaojun Xiao, Ruobing Xie, Yuan Yao, Zhiyuan Liu, Maosong Sun, Xu Zhang, and Leyu Lin. Uprec: User-aware pre-training for recommender systems, 2021.
Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. Llava-critic: Learning to evaluate multimodal models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 11628-1362,, 2025.
Tianyi Xiong, Shihao Wang. Guilin Liu, Yi Dong, Ming Li, Heng Huang. Jan Kautz, and Zhiding Yu. Phycritic: Multimodal critic models for physical ai. arXiv preprint arXiv:2602.11124, 2026.
Jiahui Xu, Lu Sun, and Dengji Zhao. Mome: recommendation. SIGIR, pp. 2527-2531, 2024.
for efficient multi-task
Fan Yang, Zheng Chen, Ziyan Jiang, Eunah Cho, Xiaojiang Huang, and Yanbin Lu. Pair: Personal- ization aware lims for recommendation, 2023.
Zhen Yang, Haitao Lin, Ziji Zhang, al. Gr-lims: Recent advances in generative recommendation based on large language models. arXiv preprint wrXiv:2507.06507 2025.
Tiansheng Yao, Xinyang Yi, Derek Zhiyuan Cheng, Felix Yu, Ting Chen, Aditya Menon, Lichan Hong, Ed H. Chi, Steve Tjoa, Jieqi Kang, and Evan Ettinger. Self-supervised learning for large- scale item recommendations, 2021.
Fajie Yuan, Alexandros Karatzoglou, Ioannis Arapakis, Joemon Jose, and Xiangnan He. A simple convolutional generative network for next item recommendation, 2018
16

---

Published as a conference paper at ICLR 2026
Fajie Yuan, Xiangnan He, Alexandros Karatzoglou, and Liguang Zhang. Parameter-efficient transfer from sequential behaviors for user modeling and recommendation, 2020.
Fajie Yuan, Guoxiao Zhang, Alexandros Karatzoglou, Joemon Jose, Beibei Kong, and Yudong Li. One person, one model, one world: Learning continual user representation without forgetting, 2021.
Zheng Yuan, Fajie Yuan, Yu Song, Youhua Li, Junchen Fu, Fei Yang, Yunzhu Pan, and Yongxin Ni. Where to go next for recommender systems? id- vs. modality- based recommender models revisited, 2023.
Zhenrui Yue, Sara Rabhi, Gabriel de Souza Pereira Moreira, Dong Wang, and Even Oldridge. Llamarec: Two-stage recommendation using large language models for ranking, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Zheni Zeng, Chaojun Xiao, Yuan Yao, Ruobing Xie, Zhiyuan Liu, Fen Lin, Leyu Lin, and Maosong Sun. Knowledge transfer via pre-training for recommendation: A review and prospect, 2020.
Jianyang Zhai, Zi-Feng Mai, Chang-Dong Wang, Feidiao Yang, Xiawu Zheng, Hui Li, and Yonghong Tian. Multimodal quantitative language for generative recommendation. arXiv preprint arXiv:2504.053 14, 2025.
Jiaqi Zhai, Lucy Liao, Xing Liu,, Yueming Wang. Rui Li, Xuan Cao, Leon Gao, Zhaojie Gong, Fangda Gu, Michael He, et al. Actions speak louder than words: fiiliinnpaarmmet sequential transducers for generative recommendations. arXiv preprint arXiv:2402. /7/ 12,11,1522221 2024.
Haochen Zhang, Tianyi Zhang. Junze Yin, Oren Gal, Anshumali Shrivastava, and Vladimir Braverman. Cove: Compressed vocabulary expansion makes better llmebased recommender systems. arXiv preprint arXiv:2506. 19993, 2025.
Junjie Zhang, Ruobing Xie, Yupeng Hou, Wayne Xin Zhao, Leyu Lin, and Ji-Rong Wen. Recommen- dation as instruction following: large language model empowered recommendation approach, 2023.
Qi Zhang, Jingjie Li, Qinglin Jia, Chuyuan Wang, Jieming Zhu, Zhaowei Wang, and Xiuqiang He. Unbert: User-news matching bert for news recommendation. In IJCAI, volume 21, pp. 3356-3362. 2021a.
Tingting Zhang, Pengpeng Zhao, Yanchi Liu, Victor S Sheng. Jiajie Xu, Deqing Wang, Guanfeng Liu, Xiaofang Zhou, et al. Feature-level deeper self-attention network for sequential recommendation. In ISCAI, pp. 4320-4326, 2019.
Yuhui Zhang, HAO DING, Zeren Shui, Yifei Ma, James Zou, Anoop Deoras, and Hao Wang. Language models as recommender systems: Evaluations and limitations. In / (Still) Can't Believe It's Not Better! NeurIPS 202/ Workshop. 2021b.
Zijian Zhang, Shuchang Liu, Jiaao Yu, Qingpeng Cai, Xiangyu Zhao, Chunxu Zhang, Ziru Liu, Qidong Liu, Hongwei Zhao, Lantao Hu, Peng Jiang, and Kun Gai. Mßoe: Multi-domain multi-task mf experts recommendation framework. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2024.
Zizhuo Zhang and Bang Wang. Prompt learning for news recommendation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2023.
Yuying Zhao, Xiaodong Yang. Huiyuan Chen, Xiran Fan, Yu Wang, Yiwei Cai, and Tyler Demr. Simaug: Enhancing recommendation with pretrained language models for dense and balanced data augmentation. arXiv preprint 250.022505.01695, 2025.
17

---

Published as a conference paper at ICLR 2026
Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, and JinRong Wen. S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. In Proceedings of the 29th ACM international conference information & knowledge management, pp. 1893-1902, 2020.
Yaochen Zhu, Liang Wu, Qi Guo, Liangjie Hong, and Jundong Li. Collaborative large language model for recommender systems. In The Web Conference 2024, 2024.
Yaochen Zhu, Chao Wan, Harald Steck, Dawen Liang, Yesu Feng. Nathan Kallus, and Jundong Li. Collaborative retrieval for large language model-based conversational recommender systems, 2025.
18

---

Published as a conference paper at ICLR 2026
A EXTENDED RELATED WORK
A.1 CLASSIC RECOMMENDATION APPROACHES
Recommender systems have long relied on two complementary paradigms: collaborative filtering (CF) Yao et al., 2021: Wang al. 2025a Li al. 2022 He & McAuley 2015) and content-based (CB) methods. CF models exploit user-iten interaction patterns, such as ratings or clicks, to learn latent representations of users and items (Koren et al. 2009). This approach domain-agnostic and often yields high accuracy, but it suffers from well-known cold-start problems for new users or items and can exhibit strong popularity bias (Abdollappouri et al. (209),, over-recommending popular items at the expense of long-tail discovery. CB methods instead leverage explicit item features or descriptions to recommend similar items, which can address item cold-start but ignore collaborative patterns and the "wisdom of the crowd." These methods may produce over-specialized recommendations that limit serendipity.
Hybrid recommenders attempt to combine CF and CB to balance relevance, novelty, and coverage. However, even hybrid systems can be difficult to control with respect to multi-objective goals like fairness, diversity, or novelty without post-hoc re-ranking.
Sequential and Contextual Models. Moving beyond static recommendation. sequential mod- els (Yuan et al., 2018: Zhou et al. 2020; de Souza Pereira Morcira et al., 202T Hou et al. 2022 2023a Wang et al. 2023a) predict user's next interaction by modeling temporal dependencies in their history. Early neural solutions include GRU4Rec (Hidasi et al. 2013). which applied gated re-
current units capture sequence dynamics. The introduction of Transformers brought step-change: SASRec (Kang & McAuley 2018) was the first to model next-item prediction in an autoregressive fashion using self-attention, improving short-term preference modeling. BERT (Sun al. 2019) adapted bidirectional Transformers to better utilize context on both sides of a target position. These architectures form strong baselines in academic and industrial settings, yet they still rely on abstract IDs or dense embeddings, making it hard to integrate external semantic knowledge or to directly optimize multiple objectives beyond accuracy.
Recent work also explores fairness- and diversity- aware training, multi-objective loss formulations, and contextual augmentation, but these methods often require complex pipelines and lack the natural flexibility of a language interface.
A.2 LARGE LANGUAGE MODELS FOR RECOMMENDATION
The advent of large language models (LLMs) pretrained Yuan et al. 2020
Xiao et al. 2021 Qiu
et al. 2021 Li et al. 2021 Yuan et al. 2021 Shin et al. 2022; Shirkavand et al.) 2025a on massive
corpora has opened new opportunities for recommendation. Zeng et al. 2020 Liu et al. 2023c
Lin et al. 2024b Yuan et al. 2023 Wang al., 2024a; Fu et al. 2024) LLMs provide broad
world knowledge, reasoning skills, and instruction woweettioooooooowwiio following /Zhang et al. (2023): Li et al. (022a):
Contal & McGoldrick (2024) abilities that can extend beyond the pattern- maathiin of traditional
recommenders (Zhang et al. [2021]] Muhamed et al. 2021 Cui et al. 2022 Liu et al. 2022 Zhang
& <Wang 2023
Wei et al. 2024 Li al. 2023b Wang
2023b).
LLMs Recommenders. A pioneering example is P5 (Geng et al. 2022 which reformulates diverse recommendation tasks into a unified text-to-text format, allowing zero-shot Hou et al. (0024)) and few-shot transfer between tasks such as rating prediction, sequential recommendation, and explanation generation (Bao et al.] 2023a| Li et al. [203cc Yue et al. 2023 Lu et al., 2023 Zhang jt al., 2021a| Wu et al.
as textual descriptions 2024p. This unification facilitates integration of multiple modalities, such reviews, and enables natural-
queries Liu et al. (202bb; Bao ct al. (2023b); Dai et al. )023);; Lin & Zhang (2023); Zhang & Wang (2023); Yang et al. (2023); Carranza et al. (2024); Kieu et al. (2025). However, item representation in such setups is often token- iokeen-inefficienn especially for large catalogs because items must be described in text, and off-
the-shelf LLMs lack direct exposure to collaborative signals from user-item interactions (Cao et al.) 2024). This leads to a mismatch between the LLM's pretrained knowledge and the domain-specific collaborative knowledge needed for effective recommendation.
19

---

Published as a conference paper at ICLR 2026
Zero-Shot and Prompt-Based Approaches. Zero-shot prompting (Hou al., 2024b Liang et al., 2025) evaluates an LLM as a ranker given user's s history and a set of candidate items in the prompt. Such methods can achieve competitive performance without task-specific training, demonstrating strong generalization, but are sensitive to prompt design, prone to sequence-order biases, and often ignore subtle interaction semantics.
Vine-Tuning and Alignment. To address these limitations, fine-tuning methods adapt LLMs to recommendation tasks while preserving language capabilities Ren Huang (2024): Zhao al. (2025); Li et al. (2024b); Wang et al. (021)); [Shirkavand et al. (2025c). GDM (Cao et al. 2024) introduces auxiliary natural-language training tasks (e.g., masked item modeling. BPR) to inject collaborative patterns. MQL. (Zhai et al. 2025) encodes multimodal item attributes (text, images) into a shared quantitative token space, enhancing cold-start and cross-domain performance. RL-based alignment (Lu et al. 2024) further improves controllability by optimizing instruction-followin behavior with preference-basee based rewards, enabling conversational Friedman et al. (2023); Li ct al.
(2019); Chen et al. (2019); ; Kemper et al. (2024): Li et al. (2023a) Tang al. (2025) and constraint- aware recommendation.
Item ID Integration and Hybrid Representations. To avoid verbose item descriptions, several works embed item IDs directly into the LLM's vocabulary. CoVE (Zhang et al. (2025) expands the token set with unique item tokens, enabling single-token recommendations and compressed embeddings. CLLM4Rec (Zhu et al. 2024) extends this with both user and item tokens, combining soft and hard prompts integrate collaborative semantics. These ---uumeened models improve efficiency and accuracy but risk "knowledge entanglement": naive merging of ID and language tokens can cause interference, harming both recommendation accuracy and language fluency.
A.3 GENERATIVE AND HYBRID RECOMMENDER MODELS
Generative recommenders recast recommendation as sequence generation task (Yang et al. 2025), unifying retrieval and ranking in one model. HSTU (Zhai et al. 2024) employs a Transformer- based transducer, scaling up to 1.5T parameters and achieving large offline and online gains, while demonstrating NI.P-lkk scaling laws for recommendation. TIGER (Rajput et al. 2023) compresses item vocabularies via multi-code vector quantization. OneRec (Deng al. 2025b) unifies retrieval and ranking in an encoder-decoder Transformer with sparse oix-urs-of-Experts (MoE) Shazeer et al. (2017); Fedus et al. 2022); Ma et al. [2018); Tang et al. [2020); Xu et al. (002)) Zhang al. (2024); Wang et al. (2024b) for capacity scaling and adds Iterative Preference Optimization for alignment. These approaches offer novelty, explainability, and unified modeling. but require heavy compute and
careful fine-tuning strategies to retain collaborative memory.
Beyond Accuracy. Extensions like MTGR (Han et al. 2025) integrate hand-crafted features into generative architectures, while others focus on fairness, calibration, and bias mitigation in LLM-based recommenders (Yang et al. 2025). The generative format naturally supports novelty and explanation generation, which combat popularity bias and improve transparency, but system design remains challenging.
A.4 MULTIMODAL MoE LLMS
More recently, MoE has been integrated directly into multimodal large language models (MLLMs)
and large vision language models (LVLMs) Bao et al. (2022): ; Shen et al. (2023; Diao et al. 2025)
Deng al. (2025a); Xiong et al. (2025); ; Wang et al. (2025b); Xiong al. (2026 : Ganjdanesh et al.) (2025): Shirkavand et al. (2025b). MoF-LLaVA (Lin et al.) (024a) introduces sparse Moł backbone for VA (Liu et al. (023a) style LVLMS and proposes a three-stage MoE-tuning strategy that first builds a strong dense LVLM and then converts its beed-forward blocks into experts. The resulting MoE-LLaVA model achieves performance comparable to or better than substantially larger dense LLaVA variants, while activating only fraction of the parameters per token and reducing visual hallucinations Rawal et al. (2025).
Uni-MoE (Li et al. 2025 targets unified multimodal LLMs that support broad set of modalities and tasks, applying MoË layers to scale capacity while maintaining a single generalist model. Addressing task interference in instruction-tuned MILMM,, MoME (Mixture of Multimodal Experts) (Xu et al.
20

---

Published as a conference paper at ICLR 2026
Table 8: Statistics of Amazon datasets used.
Dataset
Total sequences
Num items
Games
42259
13839
Instruments
17112
6250
Arts
22171
9416
Sports
35598
18357
Beauty
22363
12101
Toys
35598
11924
Books(23)
776370
495063
Beauty(23)
729576
207649
Toys(23) 432264
162035
2024) decomposes the architecture into a Mixture of Vision Experts (MoVE) and a Mixture of Language Experts (MoLE). MoVE aggregates features from multiple vision encoders via adaptive deformable transformation and an mnsiuuttioncoondiionned router, while MoLE inserts sparsely gated adapter experts into LLM layers.
B EXPERIMENTS
B. BASELINES
We benchmark IDIOMoE against representative methods spanning classic sequence modeling and recent LLLM-based recommenders, with an emphasis on baselines that add recommendation capability LLMs.
Early sequential modeling, GRU4Rec (Hidasi et al. 2015) pioneers GRU-based session modeling: SASRec (Kang & McAuley 2018) introduces unidirectional self-attention; BERT4Rec (Sun al. adopts bidirectional masked modeling for sequences.
2019
Transformer extensions and self-supervision. FDSA (Zhang al. 2019) enriches feature de- pendencies within Transformers, and S3-Rec (Zhou et al. 2020) pretrains with sequence-aware self-supervision.
Representation design, multimodality, and framework-style comparatives. VO-Rec (Hou et al.
2023b) learns discrete item codes vector quantization: MissRec (Wang al. 2023a) explores mul- timodal pretraining and transfer: TIGER (Rajput et al., 2023) formulates autoregressive retrieval over semantic IDs. Framework baselines that unify text and recommendation include PS/P5-CID (Geng
2022; Hua et al. 2023) and its multimodal extension VIP5 (Geng et al. 2023). F4SRec (Li et al.
ct al.
2023d) targets efficient sequential recommendation with a largely frozen LLM. ReAT (Cao
al.
2024) aligns LLMs to recommendation through auxiliary, recommendation speciffc tasks, For
completer etneeeteness on small Amazon benchmarks, we also report CoVE (Zhang et al. 2025).
Our reproduced and controlled variants. To isolate architectural effects under identical capacity, tokenizer, and training budget, we implement three LLM-based variants on the same backbone IDIOMoF: (i) ID Transformer (item tokens only): (ii) Item-ID LLM text-derived bias (ID) embeddings augmented with text features); and (iii) Item-LI. (vocabulary expansion with explicit item text but MoE). We also reproduce strong non-L.LM and hybrid sequential baselines, including SASRec (Kang & McAuley 2018) and HSTU (Zhai et al. 2024). Unless stated otherwise, all LLM- based baselines are matched IDIOMoE in active parameter count and trained with the same token budget, optimizer, sequence length, and schedules.
B.2 DATASETS
We use public Amazon Dataset: Games, Intruments and Arts (Ni al.) 2019) well Sports, Beauty and Toys|McAule et al. (2013). See Table for dataset statistics. We also train and evaluate on our in-house industrial-scale dataset with millions of users and tens of thousands of items.
21

---

Published as a conference paper at ICLR 2026
B.3 PREPROCESSING
We take the preprocessed version of Games, Arts, and sports from Zhai et al. (2025). We take small Sports, Beauty and Toys from Zhang et al. (025). We download 2023 amazon variants from the official website Hou et al. (2024a). Following previous work Rajput et al. (2023) , we first filter out unpopular users and items with less than five interactions. Then, we create user behavior sequences based the chronological order. We use chronological leave-last-k splitting per user: last 1 for test, the preceding for validation, and the remainder for training. Item text comes from title and categories. Maximum item history length is 50 items (most recent first). Maximum total token length (items + text) is 1024. We truncate text first, then items if necessary to satisfy the context size. We pad shorter sequences to 1024 with special pad token; attention masks prevent loss on padded positions. We take the final unpadded position for evaluation.
B.4 OPTIMIZATION AND EVALUATION
Optimizer is Adam (betas (0.9, 0.9999), eps 1e-8, weight decay 1e-2). We use linear warmup of 3000 iterations, then a cosine decay learning rate schedule. We tune learning rate with a grid search over (1e 3, le 5, 1e 5) for IDIOMoE and baselines. Training NVIDIA 1100-80GB. Batch size is 128. We use standard next-token objectives that minimizes the KL divergence between the data distribution and the distribution of the LLM. We report NDCG@ 10/50, HR @ 10/50, and MRR. Metrics are computed over the full catalog. We train for 200 epochs on small amazon datasets and for 50 epochs on larger amazon datasets. For text benchmarks we use 1m-eval -harness Gao et al. (2024). We constrain the output space to the unseen token items for retrieval quality.
with boat on
B.5 IDIOMOE DETAILS
1. Experts per FFN block: 2 (ID expert Text expert).
2. Routing: static token-type routing (ID tokens + ID expert; text tokens - Text expert).
3. Shared components: attention, LayerNorms, positional embeddings.
4. Expert widths: tables.
Text expert width 1. ID expert width 1 for ablations. Tuned for main
5. Placement: all-layers become MoE for ablations. last-k with 4,8, 16 is tuned for main results.
6. Freezing Policy: For Table experiments (Text analysis) and ablations, LLM backbone is frozen. In other small-scale runs select the best among: freeze-all, freeze-text-exper only, and frreze-attention-nnly In industrial dataset, we freeze everything and only train the item experts and item embeddings.
7. Factorized Embedding: amazon datasets, instead of single embedding table IR Nnem Xd. we first project to lower dimensional space and then to the model dimension to reduce embedding parameters E = W; x W., where Wi F Nume X ddmid and W.. E dnd x
8. For main results (not ablations and not Table D we warm up the item expert with item-only sequences for 20% of epochs, then gradually mix in text tokens with linear schedule. Ablations with LLM-based models and Table do not use this warm-up to ensure fairness.
B.6 RESULTS
B.6.1 PROPRIETARY RESULTS
Table shows the results on our industrial dataset.
We also conduct additional experiement our industrial dataset to study the effect of scaling
the model using the Qwen 2.5 Owen et al. (2025) family (0.5B, 1.5B, 3B, 7B). Figure shows the results. We see that recommendation quality improves with LLM size given enough training data, and the gains of IDIOMoE over Item-LLM as the main baseline are persistent across all model scales considered.
22

---

Published as a conference paper at ICLR 2026
Table 9; Results on our industrial dataset.
Method
Industrial A(%)
NDCG@10
HR @ 10
MRR
SASRec (baseline)
HSTU
+10.5%
+2.7%
+13,2%
ID Transformer
+21.1%
+8.9%
+23.1%
Title-LMM
-81.8%
-87.6%
-98.4%
Text-Attr LLM
+25.4%
+14.1%
+25.9%
Item-LLM
+23.5%
+13.0%
+24.3%
IDIOMeE
+27.1%
+16.6%
+31.2%
HR@10 vs. model
NDCG@10 vs. model
Item-LLM 4HR@90 vs 0.5B
Relative NDCG@ imprevement (%)
Relative HR.B 1R imprevement (%)
20
25
20
10
15
10
tem-LLM 4NDCG@10 0.38
IDIOMoE ANDCG@10 vs iter-LLMM 0.58
0.58
1.58
0.58
1.58
78
Model size
Model size
Figure 6: Relative performance on the industrial dataset with Qwen 2.5 backbones of different sizes. All values are reported as relative improvements (%) over the 0.5B baseline Item-LLM.
B.6.2 SEMANTIC IDs
IDIOMoE is fully compatible with semantic schemes for handling new items. We conduct an experiments where we replace raw item IDs with semantic IDs from MQL4GRee (Zhai et al. 2025) showing that IDIOMoE 's gains persist in this setting (Table TOp.
We also conducted cold-start experiment following the steps described in the section 4.3 of TIGER (Rajput et al. (003),, where we remove 5% of test items from the training data and report the test performance overall and over the unseen set items. We set the ratio of unseen items items in the top-k items e = 0.1. Table II shows the results, demonstrating that our method extends naturally to standard cold-start mechanisms.
These results demonstrate that our method extends naturally to standard cold-start mechanisms and compatible with semantic-ID-baase handling of new items.
B.6.3 ATTENTION ANALYSIS ON TEXT-ONLY PROMPTS
We analyze the internal attention behavior of our Item LLM on text-only inputs. We use the same tokenizer and pretrained backbone as the deployed model, run the model and compute summary statistics per layer. We compare (i) IDIOMoE (ii) a freshly loaded pretrained backbone.
of text prompts,
For each transformer layer, we average heads, mask padding, and re-normalize per query. We report:
1. previous-token attention, Ali, i averaged over valid positions
2. attention to the first token, A(: 0]
3. the distance profile, Aji d] function of offset d
23

---

Published as a conference paper at ICLR 2026
Table 10: Performance with Semantic IDs on three Amazon datasets.
Arts
Games
Instrunets
Methed
HRae10
NUCGG0 10
Hiker to
INXG 10
HR
ND0G6110
IDIOM&E
Table 11: Cold-start evaluation following TIGER: 5% of test items are removed from training. We report overall test metrics (All) and metrics restricted to unseen items (Unseen) for three datasets.
Games
Unseen
Unseen
Unseen
Method
HR 63 10
NDC06P1
HR 10
NDOG61 10
HR
NDCG6
HRB 10
NDCGP10
HR
NDCO810
HR 10
NDCGR10
Backhone IDJOMaE
0,0508
0.0518 0,0543
0.0569
0,0395
0,0849
0.0534 0,0572
0,0478
0,0332
0,0642 0,0877
0.0433
0.0394
0.0249
0,0892
0,0416
0,0541
0,0422
0,0579
0,0530
0.0513
4. the entropy of the attention distribution over keys per query, averaged over queries.
We also aggregate distance profiles over carly/mid/late layer blocks for clarity.
Figures and show that the MoE model and the pretrained backbone exhibit near-identical attention patterns on text-only inputs across all layers. Layer-wise previous-token bias, first-token emphasis, and attention entropy overlap almost perfectly, and early/mid/late distance profiles coincide within visual resolution.
This alignment expected in our setting for two reasons:
1. The MoE architecture modifies the feed-forve fordard pathways, while the backbone self-attention blocks remain architecturally unchanged
2. The text-only inputs do not activate item-specific experts, the effective computation path closely matches the backbone.
Consequently, attention structure (diagonal strength, range of contextual aggregation) remains stable, even though token-level representations downstream of attention can still differ due to MoE expert routing within the MLPs. Under text-only prompts, our fine-tuned Item LLM preserves the backbone's s attention geometry. This suggests that improvements from MoE primarily arise in representation and computation within expert MLPs rather than from altering attention allocation.
B.6.4 EFFICIENCY RESULTS
Our model is evaluated with standard batched inference. Is not restricted to processing a single query at time. Just like a conventional LLM, IDIOMoE supports multi-query batches with appropriate padding and attention masking, and all of our reported results use evaluation batch sizes larger than Table [abblrr] latency and throughput values for both batched training and inference on three sequence lengths, showing that IDIOMoE achieves comparable performance values to the underlying backbone model at various sequence lengths. The MoE modification only changes the FFN sublayers (attention remains shared), so the per-token compute remains similar, and there additional online adaptation step at serving time beyond single forward pass. From Table 12 two trends stand out:
1. Overhead shrinks with sequence length. At short contexts (256 tokens), MoE adds modest training overhead (+6.5% latency, -6.1 tokens/s) and a larger inference overhead (+18.4% latency). As context grows, routing/pack-scatter costs amortize: 512 tokens the inference overhead drops to +12.5%, and at 1024 tokens it is only +3.8% with no memory increase. Training overhead is similarly small long sequences (< 0.7% tokens/s 1024).
2. Memory is neutral. Peak GPU memory is within +0.5G of the dense baseline across all settings, and identical at 1024 tokens for both training (29.4G) and inference (4.67G), consistent with activating one expert per token.
IDIOMoE achieves near-parity efficiency at long contexts (5 4% overhead at 1024) and acceptable overheads at short contexts (re 18% 256), while keeping memory effectively unchanged. In Section 4 we show these costs buy consistent quality gains placing IDIOMoE on a favorable quality-latency Pareto frontier.
24

---

Published as a conference paper at ICLR 2026
MM
E20
Figure 7: Layer-wise attention metrics text-only inputs. Left: previous-token attention. Middle: attention to the first token. Right: attention entropy. MoE (blue) and backbone (orange) overlap across layers, indicating preserved attention geometry.
bys's
vdd
Byels
16
26
83
Figure 8: Distance profiles aggregated over early, mid, and late layers. MoE and backbone curves are nearly identical, reflecting similar allocation of attention mass across short-, medium-, and long-range dependencies.
Table 12: Efficiency at batch size 8 for three sequence lengths with item ratio of 0.2. A is MoE relative to the dense baseline. Latency end-to-end per query; throughput is steady-state.
Soq
Phase
Latency (ms) 1
Examples/s 1
(O)
Base
Mob
Base
MoB
A
Base
MoB
Base
MoE
Train Inter
6.1%
16314.61
6.11%
256
36.13
42.78
+18.4%
221.44
186.99
11.6%
56689 83
47870.29
15.6%
2.81
2.82
+0.4%
512
Train
Infer
49.16
186.58
55.33
+12.5%
162.72
44.30
144.59
11.2%
33.%%
22681.76
83314,50
74028.48
23196.24
11.2%
3.43
3.43
0.0%
1024
Train
323
323.98
+0.2%
24.73
24.69
-0.2%
25324
25146.42
-0.7%
29,40
Infer
92.45
95.92
+3.8%
86.53
83.40
-3.6%
88607.00 00
85400.26
-3.6%
4.67
4.67
0.0%
25


Reviews:
[{"review_id": "xVDurUYwrR", "reviewer": "Reviewer_BYeB", "summary": "This paper proposes IDIOMoE, an item-ID and natural-language mixture-of-experts transformer that routes ID tokens to a dedicated collaborative expert and text tokens to the frozen pretrained expert, enabling seamless recommendation within a generative LLM while preserving linguistic ability.", "questions": "See Weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel ID/text mixture-of-experts architecture that treats item IDs as a native dialect, effectively disentangling collaborative and semantic signals.\n\n2. Static token-type routing keeps implementation lightweight while preserving the pretrained LLM’s language capabilities.\n\n3. The method is thoroughly evaluated against industrial-scale baselines on datasets, demonstrating consistent gains.", "weaknesses": "1. It seems that the rigid ID/text routing cannot jointly model attributes blended in user queries, leaving collaborative signals and side features separated. For exmaple, in queries like “white iPhone”, the tokenizer sends “white” to the Text-Expert and “iPhone\" to the Item-Expert. How can the proposed method avoid such effects?\n\n2. How can the proposed model handle new items? It seems that there is no clear zero-shot or cold-start mechanism is provided in the paper.\n\n3. How efficient is the algorithm proposed in the article? From the model's perspective, it can only process one user's query at a time. If there are a large number of users making recommendation requests, can the model be effectively deployed and adapted?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C5IQK3dAnq", "reviewer": "Reviewer_ZkdV", "summary": "This paper addresses the key challenge of integrating collaborative filtering signals (item IDs) and natural language understanding in recommendation systems without mutual interference. It proposes IDIOMoE (Item-ID + Natural-language Mixture-of-Experts Language Model), a dual-expert architecture that treats item interaction histories as a \"native dialect\" within the language space. The core design modifies the Feed Forward Network (FFN) of a pretrained LLM into separate text and item experts, with static token-type gating to route text tokens to the text expert and item-ID tokens to the item expert. Experiments on public Amazon datasets (6 small-scale, 3 large-scale) and a proprietary industrial dataset (hundreds of millions of users) demonstrate that IDIOMoE outperforms classical recommenders (e.g., SASRec, BERT4Rec) and LLM-based baselines (e.g., P5-CID, CoVE) in metrics like NDCG@10, HR@10, and MRR, while preserving the LLM’s natural language competence. The paper’s key contributions include: (1) a disentangled MoE architecture for separating collaborative filtering and semantic processing; (2) robust performance at industrial scale with retained language understanding; (3) extensive ablations verifying gains from expert specialization rather than parameter scaling; (4) FFN key-value memory analysis revealing more interpretable and modular representations via expert disentanglement.", "questions": "## 1. Questions Related to Technical Mechanisms.\n- Dynamic Adjustment of Item Expert Capacity: The paper observes that small-scale datasets (e.g., Amazon-Beauty) perform best with a smaller-capacity item expert (shrink=4), while large-scale industrial datasets require a full-capacity expert (shrink=1). However, it does not propose an adaptive capacity allocation algorithm that can automatically adjust based on data characteristics (e.g., item interaction sparsity, dataset size). Is it feasible to design such an algorithm?\n\n- What is the performance of IDIOMoE on cold-start items (no interaction data, only text attributes)? Please provide quantitative results on cold-start subsets of the Amazon or industrial dataset, comparing with content-based and LLM-based cold-start methods.\n\n## 2. Suggestions Related to Experimental Design.\n- Validation of Recommendation Explanation Generation: The abstract claims IDIOMoE preserves LLM capabilities to support \"transparent explanations,\" but experiments only verify general language understanding via metrics like WikiText NLL and BBH benchmarks. To validate this claim, the authors should supplement an \"explanation quality evaluation\" experiment.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "## 1. Originality.\n- The paper introduces a novel MoE-based disentanglement paradigm for recommendation systems, which is the first to explicitly model item IDs as a \"dialect\" separate from natural language. This addresses the long-standing \"knowledge entanglement\" issue in prior LLM-based recommenders (e.g., CoVE, CLLM4Rec) that naively merge ID and text tokens.\n- The static token-type gating design is a creative simplification of standard MoE (e.g., Switch Transformers), avoiding the inefficiency and instability of dynamic routing while achieving effective modality separation.\n\n## 2. Significance.\n- It establishes a new direction for LLM-recommender fusion by emphasizing \"modality-specific specialization\" over brute-force parameter scaling, inspiring follow-up work on disentangled multi-modal recommendation.\n\n## 3. Rigorous experimental design.\n- The study covers diverse datasets, comprehensive baselines (classical sequential models, LLM-based recommenders, capacity-matched non-MoE variants), and key ablations (expert capacity, MoE layer placement, static vs. dynamic routing).", "weaknesses": "## 1. Insufficient analysis of cold-start scenarios.\n- Cold-start is a critical challenge for recommenders, but the paper only mentions item textual attributes as a potential solution without quantitative evaluation. For example: How does IDIOMoE perform on new items with only text descriptions (no interaction data)? And can the item expert leverage text-derived signals to mitigate cold-start bias?\n\n## 2. Insufficient verification of recommendation explanation generation capability.\nThe paper claims that IDIOMoE retains the language ability of LLM to support \"transparent explanations\", but the experiments only verify the language understanding ability through general language benchmarks such as WikiText NLL and BBH, without conducting quantitative/qualitative evaluations on explanation generation in recommendation scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p6ZQZox56H", "reviewer": "Reviewer_jsF2", "summary": "This paper aims to integrate item ids into large language models (LLMs) for recommendation systems. The authors propose IDIOMoE, a Mixture-of-Experts architecture that treats item interactions as a distinct dialect within the language space. The core idea is to route item ids to addtional experts (FFN) and text tokens to the original experts, which allows the model to leverage both collaborative filtering signals and natural language understanding without interference. The paper demonstrates that IDIOMoE outperforms text-only and item-only baselines on both public benchmarks and a large proprietary dataset, while maintaining the semantic understanding of the pre-trained LLM. The authors also provide extensive ablations to show that the improvements come from expert specialization and routing, rather than just added parameters.", "questions": "Refer to Weaknesses and I can further raise my Rating if the authors can provide more results.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. **S1:Idea**: The idea of using a Mixture-of-Experts design to separate collaborative filtering from semantic processing is innovative. By assigning dedicated experts for item IDs and text tokens, the model can mitigate interference and preserve the strengths of both modalities.\n2. **S2:Rigorous Evaluation**: The paper provides extensive evaluations on both public benchmarks and a large proprietary dataset compared with various advanced baselines, demonstrating the effectiveness of the proposed method. And authors also conduct extensive ablations and analysis to isolate the source of gains, showing that improvements arise from expert specialization and routing, and providing evidences why IDIOMoE can preserve the LLM's utility.", "weaknesses": "1. **Scaling of Parameters**: While the paper shows that the proposed method outperforms baselines, it is not clear how the performance scales with LLM's size.\n2. **Ablation on MoE Architecture**: As is shown in Table4, MoA and MoT also achieve similar performance to IDIOMoE, but their parameter design on Attention instead of FFN, what's the advantage of using FFN for MoE instead of Attention? \n3. **Can IDIOMoE work with Various ID Types?**: As it shown that the proposed method works well with item IDs, can it also work with other types of IDs, like Semantic IDs in TIGER?\n4. **Can IDIOMoE Reasoning in recomendation?**: The paper focuses on the integration of item IDs and text tokens, and it would be interesting to see how well IDIOMoE can handle reasoning tasks that require understanding of both item interactions and natural language queries. For example, can it answer questions about user preferences or make recommendations based on complex user queries that involve multiple items and attributes?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YSdkS6CVXO", "reviewer": "Reviewer_jk5P", "summary": "This paper proposes a novel architecture for integrating collaborative filtering signals (item interactions) with LLMs. The authors argue that prior works suffer from semantic–collaborative entanglement when item IDs are added directly to LLM vocabularies, and their disentanglement largely improves recommendations while retaining general text reasoning.", "questions": "1. How does IDIOMoE perform in interactive conversational settings (e.g., user dialogue or open-ended preference elicitation)?\n\n2. Could the static routing scheme limit adaptability to unseen token types? How does the model handle cold-start items or dynamically changing catalogs?\n\n3. In lines 121-122, it says previous generative recommendation methods \"can forget collaborative structure if not carefully aligned with interaction data.\". Can you show more details or evidence?\n\n4. Despite using different experts, your method also integrates the two embeddings after FFN blocks. Can you explain why this is superior to other methods that directly add item tokens to LLM vocabularies (and with some alignment techniques)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The MoE-based separation of text and item experts is simple yet elegant, showing consistent performance gains on diverse datasets (small/large Amazon and industrial-scale).\n\n2. With only FFN layers modified, this method has the potential to be applied to industrial-scale recommendation systems with low overhead.\n\n3. The paper provides comprehensive ablations of architecture settings like MoE placement, static vs. dynamic routing, and expert shrinkage.", "weaknesses": "1. The backbone models are small (0.5B and 1.5B), so it's questionable whether the effectiveness can scale to larger models.\n\n2. The method has limitations on newly-coming items, which require extending the vocabulary and re-training.\n\n3. The idea of assigning different modalities to different experts to reduce modality interference has already been proposed in the domain of VLM / MLLM [1,2]. This paper directly uses this idea but doesn't mention it in related works.\n\n[1]. Li, Yunxin, et al. \"Uni-moe: Scaling unified multimodal llms with mixture of experts.\" TPAMI, 2025.\n\n[2]. Shen, Leyang, et al. \"Mome: Mixture of multimodal experts for generalist multimodal large language models.\" NeurIPS, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)