Title: Sparse Meets Dense: Unified Generative Recommendations with Cascaded Sparse-Dense Representations
Venue: NeurIPS 2025
Accepted: yes
Keywords: Generative Recommendation
Abstract: Generative models have recently gained attention in recommendation systems by directly predicting item identifiers from user interaction sequences. However, existing methods suffer from significant information loss due to the separation of stages such as quantization and sequence modeling, hindering their ability to achieve the modeling precision and accuracy of sequential dense retrieval techniques. Integrating generative and dense retrieval methods remains a critical challenge. To address this, we introduce the Cascaded Organized Bi-Represented generAtive retrieval (COBRA) framework, which innovatively integrates sparse semantic IDs and dense vectors through a cascading process. Our method alternates between generating these representations by first generating sparse IDs, which serve as conditions to aid in the generation of dense vectors. End-to-end training enables dynamic refinement of dense representations, capturing both semantic insights and collaborative signals from user-item interactions. During inference, COBRA employs a coarse-to-fine strategy, starting with sparse ID generation and refining them into dense vectors via the generative model. We further propose BeamFusion, an innovative approach combining beam search with nearest neighbor scores to enhance inference flexibility and recommendation diversity. Extensive experiments on public datasets and offline tests validate our method's robustness. Online A/B tests on a real-world advertising platform with over 200 million daily users demonstrate substantial improvements in key metrics, highlighting COBRA's practical advantages.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces COBRA, a generative recommendation framework that combines sparse semantic IDs and dense item vectors through a cascaded generation process. By leveraging a coarse-to-fine strategy and a novel BeamFusion inference method, COBRA improves both accuracy and diversity. Extensive evaluations on public and industrial datasets, including online A/B tests, demonstrate its effectiveness over strong baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper presents a well-motivated and novel cascaded representation learning strategy that tightly integrates sparse semantic IDs and dense vectors in both training and inference, addressing limitations of previous generative retrieval methods that rely solely on one modality.\n2. The model is not only validated through rigorous offline experiments but also deployed in a real-world recommendation platform.\nThe authors provide extensive quantitative evaluations, including ablations and diversity trade-off analysis, offering strong evidence for the effectiveness and robustness of their approach.\n\nWeaknesses:\n1. As far as I know, the only difference between GRU4Rec and SASRec is that the former uses RNN for sequential modeling, while the latter adopts attention mechanisms. While attention-based methods may have certain advantages, why is there such a significant performance gap between the two on all three datasets?\nI believe the baselines used in the experiments can be roughly divided into two categories: traditional sequential recommendation methods, such as GRU4Rec, and generative approaches, such as P5 and TIGER. However, the authors did not compare their method with the latest baselines from either category.\n2. The authors did not present each equation on a separate line with a corresponding number in the methodology section, which slightly deviates from standard academic writing conventions.\n3. While the empirical results are convincing, the paper lacks a formal theoretical analysis of why cascaded sparse-dense generation yields superior performance or how error propagation between stages is mitigated.\n\nQUESTIONS:\n1. Why is there such a large performance gap between GRU4Rec and SASRec across the three datasets?\n2. Why didn’t the authors include comparisons with the most recent methods in both traditional sequential recommendation and generative recommendation?\n3. I suggest that the authors revise the methodology section using more standard equation formatting practices.\n4. Although the paper presents corresponding experimental results, it fails to clearly illustrate the limitations of TIGER and explain why integrating sparse semantic IDs and dense vectors can effectively address the issues in TIGER.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents COBRA (Cascaded Organized Bi-Represented generAtive retrieval), a unified framework for generative recommendation that integrates both sparse semantic IDs and dense vector representations in a cascaded manner. The model takes a user’s interaction history encoded as a sequence of sparse-dense pairs and uses a Transformer decoder to alternately predict the next item’s sparse ID followed by its dense vector. The dense representations are trained end-to-end using contrastive learning, and the inference adopts a coarse-to-fine generation strategy supported by a BeamFusion module for diversity control. Experiments on standard benchmarks show COBRA outperforms previous state-of-the-art generative recommenders in both accuracy and diversity.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The use of sparse IDs as coarse-grained anchors helps reduce the learning difficulty of dense representations and improves controllability in inference.\n\n2. The introduction of BeamFusion adds flexibility and diversity to the generative recommendation process.\n\n3. Empirical results on public benchmarks demonstrate strong performance gains over existing generative recommendation models such as TIGER.\n\n\nWeaknesses\n\n1. The baselines primarily include sparse ID-based models (e.g., TIGER), but miss stronger or more comparable systems like Learnable Item Tokenization for Generative Recommendation, which also explore hybrid sparse-dense paradigms. The absence of such baselines makes it difficult to fully assess COBRA's relative effectiveness.\n\n2. The model architecture used in experiments is relatively small (1-layer encoder and 2-layer decoder), which raises concerns about scalability. Recent models like OneRec suggest that more expressive architectures are becoming standard, and it would strengthen the work to evaluate COBRA’s performance under larger-scale settings.\n\n3. The proposed method lacks sufficient theoretical analysis to justify why the cascaded structure improves performance. While empirical results are promising, scenarios like cold-start—where sparse IDs may be poorly learned—are not thoroughly discussed. The assumption that sparse IDs always enhance dense representations could break down in such cases. Further study on these edge cases would be valuable.\n\nQUESTIONS:\nSee above weakness\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes COBRA, a generative recommendation framework that integrates sparse IDs for coarse-grained item semantics and dense vectors for fine-grained details, enabling precise and personalized recommendations. During inference, COBRA introduces a BeamFusion mechanism, combining beam search with nearest neighbor scores to enhance inference flexibility and recommendation diversity. Extensive experiments on both public and industrial datasets demonstrate the effectiveness, robustness, and scalability of COBRA.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths** \n\n1. This paper integrates dense vectors to capture fine-grained details of items through end-to-end training, effectively mitigating the information loss in ID-based methods.\n2. This paper presents a BeamFusion mechanism during inference, which balances recommendation accuracy and diversity.\n3. Experimental results validate the COBRA's superiority over compared baselines, with detailed quantitative results and analyses.\n4. This paper is well-organized and clearly written, with a clear introduction, related work, methodology, and experimental evaluation.\n\n**Weaknesses**\n\n1. While the paper incorporates dense vectors to enhance sequential recommendation accuracy over TIGER [1], it also introduces additional computational overhead. Therefore, providing a comparative analysis of model complexity and training/inference runtime across baselines would be valuable for assessing the effectiveness and efficiency of COBRA.\n2. The paper lacks essential implementation details, including the embedding dimension, detailed architectures of the encoder and decoder, optimizer choices, learning rate, batch size, and pretraining settings for the residual quantization model (Figure 2). These omissions raise concerns about the reproducibility of the experimental results.\n3. Several important baselines in the generative recommendation are missing, such as ColaRec [2], LC-Rec [3], and RPG [4]. Including comparisons with these methods would better highlight the effectiveness of the proposed COBRA.\n\n4. While ablation studies are conducted on industrial datasets (Section 4.2), additional ablation analysis on public datasets would further validate the robustness and generalizability of the proposed COBRA.\n5. Some typos are present. \n6. In Lines 108 and 174, \"encoder Encoder\" should be revised for consistency in formatting—either uniformly bolded or unbolded; In Figure 1, the Transformer encoder should be revised—it should be either shown or omitted on both sides to avoid misleading readers into thinking the proposed COBRA omits the encoder compared to TIGER; In Line 170, $\\bold{v}_{j}$ should be revised to $\\bold{v}_{{item}_j}$; In Lines 153–155, obtaining $\\bold{e}_{t+1}$ from predicted $ID_{t+1}$ via the embedding layer Embed (as defined in Line 136) should be explicitly described to ensure clarity in the methodology.\n\n**Reference**\n\n[1] Recommender Systems with Generative Retrieval, NeurIPS, 2023.\n\n[2] Content-Based Collaborative Generation for Recommender Systems, CIKM, 2024.\n\n[3] Adapting Large Language Models by Integrating Collaborative Semantics for Recommendation, ICDE, 2024.\n\n[4] Generating Long Semantic IDs in Parallel for Recommendation, KDD, 2025.\n\nQUESTIONS:\n1. The paper claimed: \"However, unlike [19], which uses a different configuration, we employ a 3-level semantic ID structure, where each level corresponds to a codebook size of 32\" (Lines 227-228). What are the advantages of using the same codebook sizes across all levels, compared to varying codebook sizes?\n2. COBRA uses 3-level semantic IDs (32×32×32) for public datasets (Lines 227–228), while it uses 2-level semantic IDs (32×32) for industrial-scale datasets (Line 254). Does this imply that the number of levels is dataset-dependent? If so, the paper should provide empirical evidence and a guideline for choosing appropriate configurations under different conditions. This also raises concerns about the COBRA’s adaptability across different settings.\n3. In Section 4.2, COBRA uses 2-level semantic IDs (32×32), but its variant “COBRA w/o Dense” uses 3-level semantic IDs (256×256×256) (Line 260). What is the rationale for this difference? This setup may affect the fairness of the ablation.\n\nLIMITATIONS:\nSee the above **Weaknesses**.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes COBRA, a new model which integrates generative and dense retrieval method by formulating sparse ID generation and dense vector prediction through a cascading manner in sequential recommendation. COBRA is trained via optimizing both the CE loss of next token prediction for sparse IDs and the CE loss with in-batch negatives for dense vectors. During inference, COBRA leverages beam search to generate several distinct IDs and dense vectors, builds the candidate item set by combining items corresponding to these IDs and items retrieved by dense vectors through MIPS, and the final result is generated by selecting top-K items in the candidate set via a non-linear combination of beam search scores and cosine similarity scores. Experiments on public and private datasets demonstrate the effectiveness of COBRA.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. Writing is good and the paper is easy to follow\n2. Experimental results seems impressive\n\nWeakness:\n1. The evaluation of major contributions (cascade bi-represented formulation, coarse-to-fine generation) is not thorough. \n2. Lack of analysis of time complexity of training and inference, which is important for retrieval models.\n\nQUESTIONS:\n1. Experiment results do not verify the necessity and importance of the cascade bi-represented formulation in COBRA, which raises doubts about the novelty of the paper.  It seems the performance improvement comes from ensembling generative retrieval and dense retrieval methods. A natural question is, what is the performance of combining retrieved results of existing GR and DR methods through BeamFusion or other simple fusion strategy like linear combining beam search and NN scores?\n2. As COBRA leverages multiple dense embeddings for NN search, current dense retrieval methods with multiple embeddings should also considered as baseline for fair comparison. Examples includes MIND[1] and ComiRec[2].\n3. What is the ablation study result in the public dataset?\n4. What is the value of $M$ corresponding to Table 1&2? \n\n[1] Multi-Interest Network with Dynamic Routing for Recommendation at Tmall\n[2] Controllable Multi-Interest Framework for Recommendation\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-04"}