Multimodal Quantitative Language for Generative Recommendation

Authors: Jianyang Zhai, Zi-Feng Mai, Chang-Dong Wang, Feidiao Yang, Xiawu Zheng, Hui Li, Yonghong Tian
Primary area: generative models
Keywords: Recommendation System, Generative Recommendation

Abstract:
Generative recommendation has emerged as a promising paradigm aiming at directly generating the identifiers of the target candidates.
Most existing methods attempt to leverage prior knowledge embedded in Pre-trained Language Models (PLMs) to improve the recommendation performance. However, they often fail to accommodate the differences between the general linguistic knowledge of PLMs and the specific needs of recommendation systems. Moreover, they rarely consider the complementary knowledge between the multimodal information of items, which represents the multi-faceted preferences of users.  To facilitate efficient recommendation knowledge transfer, we propose a novel approach called Multimodal Quantitative Language for Generative Recommendation (MQL4GRec). Our key idea is to transform items from different domains and modalities into a unified language, which can serve as a bridge for transferring recommendation knowledge. Specifically, we first introduce quantitative translators to convert the text and image content of items from various domains into a new and concise language, known as quantitative language, with all items sharing the same vocabulary. Then, we design a series of quantitative language generation tasks to enrich quantitative language with semantic information and prior knowledge.  Finally, we achieve the transfer of recommendation knowledge from different domains and modalities to the recommendation task through pre-training and fine-tuning. We evaluate the effectiveness of MQL4GRec through extensive experiments and comparisons with existing methods, achieving improvements over the baseline by 11.18\%, 14.82\%, and 7.95\% on the NDCG metric across three different datasets, respectively.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730210629596

thread_type:
official_review

<thread_object>
creator: reviewer_rvkm

<content>
summary:
This paper introduces a novel approach, MQL4GRec, designed to convert item content from diverse domains and modalities into a unified quantitative language.

strengths:
1. The proposed concept of a multimodal quantitative language, together with the design of quantitative language generation tasks, represents a novel and innovative advancement in the field of generative recommendation

2.  The architecture design is elegant, presenting the idea in a straightforward way, yet experiments demonstrate its strong effectiveness. I personally appreciate this type of work and believe it can make a meaningful impact in the field of generative recommendation.

3. The availability of the code significantly enhances reproducibility.

weaknesses:
1.The proposed framework is rooted in the generative recommendation paradigm and aligns with a preprint in a similar research direction [1]. However, it still represents a valuable contribution in my view, even if not groundbreaking.

2.The authors should consider testing the statistical significance of MQL4GRec results.

3.There remains a limitation in zero-shot capability, which is a known challenge in the field of recommendation.

4.To enhance the comprehensiveness of the "Multi-modal Recommendation" section in the related work, the authors could consider including more recent state-of-the-art multimodal recommender system papers, such as [2,3]. The field of multimodal codebooks from other communities should also be included in the related work section to clarify the distinctions between the proposed MQL approach and existing methods, such as those in [4, 5]


[1] Liu, Han, et al. "MMGRec: Multimodal Generative Recommendation with Transformer Model." arXiv preprint arXiv:2404.16555 (2024).

[2] Fu, Junchen, et al. "IISAN: Efficiently adapting multimodal representation for sequential recommendation with decoupled PEFT." *Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval*. 2024.

[3] Liu, Han, et al. "MMGRec: Multimodal Generative Recommendation with Transformer Model." *arXiv preprint arXiv:2404.16555* (2024).

[4] Lan, Zhibin, et al. "Exploring better text image translation with multimodal codebook." arXiv preprint arXiv:2305.17415 (2023).

[5] Duan, Jiali, et al. "Multi-modal alignment using representation codebook." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

questions:
1.As an efficient framework, I am interested in understanding the training costs of MQL4GRec compared to other baselines. Specifically, how does it perform in terms of training time, VRAM usage, and inference time?

2.In the field of recommender systems, there is a lack of widely recognized pre-trained models. Could MQL4GRec potentially serve as a foundation model for other downstream tasks?

My concerns are addressed by the authors, therefore, I will raise my scores from 6 to 8.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730347007794

thread_type:
official_review

<thread_object>
creator: reviewer_rlim

<content>
summary:
This paper presents a novel multimodal generative recommendation method, MQL4GRec, which facilitates the effective transfer of recommendation knowledge by converting item content from different domains and modalities into a unified quantitative language. Specifically, MQL4GRec introduces a unified quantitative language representation to handle multimodal content, including text and images. Additionally, a series of quantitative language generation tasks are designed to enrich the semantic representation. Extensive experiments across three datasets show that this method significantly outperforms baseline approaches.

strengths:
* High Innovation: This work is the first to propose using a unified quantitative language to address knowledge transfer in multimodal recommendation, which is highly valuable for enhancing the generalization ability of recommendation systems.

* Thorough Experimental Validation: The paper conducts extensive experiments across three datasets, showcasing not only the overall performance of the method but also analyzing the role of individual components through ablation studies, indicating rigorous experimental design.

weaknesses:
* The model's high complexity poses significant computational and storage demands, which could lead to considerable costs in real-world deployment.

questions:
* Figure 3 shows that the performance on the Games dataset slightly declines as the amount of pre-training data increases. Does this suggest that the pre-training strategy has limitations when applied to domains with substantial cross-domain differences?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730558441196

thread_type:
official_review

<thread_object>
creator: reviewer_vnzn

<content>
summary:
MQL4GRec introduces a novel approach for generative recommendation by transforming multimodal content from different domains into a unified "quantitative language," facilitating cross-domain knowledge transfer in recommendation tasks. The method uses quantitative translators for text and image content, building a shared vocabulary to encode semantic information across modalities. A series of language generation tasks further enriches this vocabulary, enhancing the model's capacity to represent multi-faceted user preferences. Experimental results demonstrate notable performance improvements over baseline models on key metrics across multiple datasets, showcasing MQL4GRec's scalability and potential in multimodal recommendation.



However, its innovation appears limited, closely resembling existing methods like GenRet and MMGRec in both tokenization approach and generative structure.

strengths:
1. Innovation: The proposed MQL4GRec method translates content from different modalities into a unified “quantitative language,” enabling cross-domain and cross-modal recommendation knowledge transfer. This approach addresses limitations in handling multimodal data in existing generative recommendation models.


2. Superior Performance: Experimental results on multiple public datasets demonstrate that MQL4GRec outperforms baseline methods on key metrics such as NDCG.



3. Open-Source Availability: The paper provides a fully accessible code repository, facilitating reproducibility and further research within the community.

weaknesses:
1. Similarity to Existing Tokenization Approaches
- The unified vocabulary for multimodal information closely resembles the generative retrieval tokenization approach found in "Learning to Tokenize for Generative Retrieval," [1]  particularly the "GenRet" model, which uses discrete auto-encoding for compact identifiers. This "multimodal codebook" seems to be an adaptation of single-modality tokenization, relying on established techniques like RQ-VAE and offering only incremental improvements without substantial architectural or performance innovation.
- The motivational structure and initial figures in MQL4GRec are closely aligned with those in GenRet, which may diminish the perceived originality of the proposed approach.



2. Lack of Comparative Analysis
-  MQL4GRec’s generative approach appears heavily inspired by MMGRec [2], which also employs Graph RQ-VAE for multimodal representation through user-item interactions, raising concerns about the uniqueness of MQL4GRec’s contributions. 
- The paper does not clearly distinguish MQL4GRec's advancements over MMGRec, especially in terms of multimodal token-based representations. A more thorough comparison is needed to establish any unique contributions beyond MMGRec’s existing framework.


[1] Sun, W., Yan, L., Chen, Z., Wang, S., Zhu, H., Ren, P., ... & Ren, Z. (2024). Learning to tokenize for generative retrieval. Advances in Neural Information Processing Systems, 36.

[2 ] Liu, H., Wei, Y., Song, X., Guan, W., Li, Y. F., & Nie, L. (2024). MMGRec: Multimodal Generative Recommendation with Transformer Model. arXiv preprint arXiv:2404.16555.

The paper does not clearly distinguish MQL4GRec's advancements over MMGRec, especially in terms of multimodal token-based representations. A more thorough comparison is needed to establish any unique contributions beyond MMGRec’s existing framework.

questions:
- User ID Collision: With only 2000 tokens representing a large user base, there is a potential for ID collisions, which may lead to inaccuracies in recommendation results in real-world applications.
   - Domain Adaptability: The model performs poorly on certain domain-specific datasets, such as the Games dataset, suggesting limitations in domain transferability.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730658672455

thread_type:
official_review

<thread_object>
creator: reviewer_ab8j

<content>
summary:
The paper titled "Multimodal Quantitative Language for Generative Recommendation" introduces a novel approach to enhance generative recommendation systems by converting item content from multiple domains and modalities into a unified "quantitative language". This methodology seeks to bridge the gap between the generalized linguistic knowledge of pre-trained language models (PLMs) and the specialized needs of recommendation systems. The authors developed a new framework, MQL4GRec, which employs "quantitative translators" to convert textual and visual item data into a shared vocabulary. This shared language is then enriched with semantic information through various generation tasks to enable effective knowledge transfer from multimodal data to recommendation systems.

The paper's main contribution lies in its innovative method of integrating multimodal data to improve recommendation performance significantly, surpassing baseline methods by notable margins in terms of the NDCG metric across multiple datasets. Furthermore, the framework introduces a potential shift towards more universal recommendation systems that do not rely on traditional item IDs, thereby addressing common challenges like improving scalability and transferability across different domains.

strengths:
1. The approach to transform diverse item content from different domains and modalities into a unified quantitative language is highly innovative. This integration allows for a more robust and versatile recommendation system that can handle varied inputs effectively.

2. The paper introduces a well-structured design of pre-training and fine-tuning tasks that includes not only generative prediction but also alignment tasks, enhancing the robustness of the method. This comprehensive approach allows the model to effectively leverage both generative capabilities and alignment strategies to improve overall recommendation accuracy.

3. The proposed framework significantly outperforms existing models on several benchmark datasets, particularly in terms of the NDCG metric. This suggests that the method is not only theoretically sound but also practically effective.

weaknesses:
1. The paper does not discuss the impact of various training operations on the algorithm's time complexity, such as the handling of collisions. This oversight might leave questions about the scalability and efficiency of the proposed method in practical applications.

2. The methodology section lacks a clear explanation of how the generated quantitative vectors are used to retrieve corresponding items during the next item generation task. This omission could lead to ambiguity regarding the operational specifics of the model.

3. The model does not utilize multimodal information simultaneously to predict the next click item, which limits the paper's innovativeness and potential for expansion. Leveraging such multimodal data more effectively could enhance the model's predictive performance and applicability in more complex scenarios.

questions:
1. How do different training operations, such as collision handling and quantization, impact the computational time and resource requirements of the model?

2. What methods or algorithms are employed to map these quantitative vectors back to specific items in the dataset?

3. Why does the model not use multimodal information simultaneously to predict the next click item? What are the challenges or limitations that prevent this integration?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732281174494

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal by Authors

comment:
We thank all the reviewers for their time and effort in reviewing our paper and providing constructive feedback！

We are encouraged that **all reviewers have recognized the high novelty and innovation** of our proposed MQL4GRec, as well as the extensive experimental validation that demonstrates the effectiveness of our approach. We are grateful for the high praise from **all reviewers** regarding the openness of our anonymized code repository, which greatly enhances the reproducibility of our work. 

Additionally, we sincerely appreciate the reviewers' appreciation for our rigorous experimental design, including comprehensive experimental validation and ablation studies (Reviewer **RLim**), the elegance of our architecture (Reviewer **rVKM**), and the well-structured design of pre-training and fine-tuning tasks (Reviewer **AB8j**). We are especially thankful to the fourth reviewer for appreciating this type of work and believing that it can make a meaningful impact in the field of generative recommendation (Reviewer **rVKM**).

After carefully analyzing the reviewers' comments, we are more than glad to find these comments very insightful. We respond to the comments point by point in the rebuttal. We hope that our response can address the concerns raised by the reviewers.

Once again, we sincerely thank the reviewers for their valuable feedback and insightful suggestions, which undoubtedly contribute to enhancing the quality of our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a novel approach called Multimodal Quantitative Language for Generative Recommendation (MQL4GRec), which transforms items from different domains and modalities into a unified language and transfers recommendation knowledge across different domains. The authors have performed extensive experiments on three different datasets to demonstrate the effectiveness of the proposed method.

Overall, this paper is clearly written. The ideas of the proposed method are novel, e.g., transforming diverse item content from different domains and modalities into a unified quantitative language is novel, and the well-structured design of pre-training and fine-tuning tasks. The experimental evaluation is extensive and convincing. The proposed method outperforms existing methods on benchmark datasets.