Multimodal Quantitative Language for Generative Recommendation
Authors: Jianyang Zhai
Zi-Feng Mai
Chang-Dong Wang
Feidiao Yang
Xiawu Zheng
Hui Li
Yonghong Tian
Venue: ICLR 2025 Poster (2025)
Keywords: Recommendation System
Generative Recommendation
Primary area: generative models

Abstract: Generative recommendation has emerged as a promising paradigm aiming at directly generating the identifiers of the target candidates.
Most existing methods attempt to leverage prior knowledge embedded in Pre-trained Language Models (PLMs) to improve the recommendation performance. However, they often fail to accommodate the differences between the general linguistic knowledge of PLMs and the specific needs of recommendation systems. Moreover, they rarely consider the complementary knowledge between the multimodal information of items, which represents the multi-faceted preferences of users.  To facilitate efficient recommendation knowledge transfer, we propose a novel approach called Multimodal Quantitative Language for Generative Recommendation (MQL4GRec). Our key idea is to transform items from different domains and modalities into a unified language, which can serve as a bridge for transferring recommendation knowledge. Specifically, we first introduce quantitative translators to convert the text and image content of items from various domains into a new and concise language, known as quantitative language, with all items sharing the same vocabulary. Then, we design a series of quantitative language generation tasks to enrich quantitative language with semantic information and prior knowledge.  Finally, we achieve the transfer of recommendation knowledge from different domains and modalities to the recommendation task through pre-training and fine-tuning. We evaluate the effectiveness of MQL4GRec through extensive experiments and comparisons with existing methods, achieving improvements over the baseline by 11.18\%, 14.82\%, and 7.95\% on the NDCG metric across three different datasets, respectively.

Published as a conference paper at ICLR 2025
MULTIMODAL QUANTITATIVE LANGUAGE FOR GEN- ERATIVE RECOMMENDATION
Jianyang Zhai!!? Zi-Feng Mai! Chang-Dong " Yang* Xiawu Zheng?*, Hui Li Yonghong Tian
1 'Sun Yat-sen University, *Pengcheng Laboratory, "Guangdong Key Laboratory of Big Data Analysis and Processing,
*Xiamen University, 'Peking University (zhai jj01, yangfd) 8pc1 acc.c, changddngwannghhtmma com
ABSTRACT
Generative recommendation has emerged as promising paradigm aiming at di- rectly generating the identifiers the target candidates. Most existing methods attempt to leverage prior knowledge embedded in Pre-trained Language Models (PLMs) to improve the recommendation performance. However, they often fail to accommodate the differences between the general linguistic knowledge of PLMs and the specific needs of recommendation systems. Moreover, they rarely consider the complementary knowledge between the multimodal information of items, which represents the multi-faceted preferences of users. To facilitate efficient recommendation knowledge transfer, propose novel approach called Mul- timodal Quantitative Language for Generative Recommendation (MQL4GRec). Our key idea is transform items from different domains and modalities into a bridge for transferring recommendation
unified language, which can serve knowledge. Specifically, we first introduce quantitative translators convert the text and image content of items from various domains into a new and concise language, known as quantitative language, with all items sharing the same vocab- ulary. Then, we design series of quantitative language generation tasks to enrich quantitative language with semantic information and prior knowledge. Finally, we achieve the transfer of recommendation knowledge from different domains and modalities to the recommendation task through pre-training and fine-tuning. We evaluate the effectiveness of MQL4GRec through extensive experiments and comparisons with existing methods, achieving improvements over the baseline by 11.18%, 14.82%, and 7.95% the NDCG metric across three different datasets. respectively.
INTRODUCTION
Recommendation systems (RS) aim recommend items to users that they may be interested in, and are widely used on many online platforms, such as e- commerce and social networking (Chaves)
Piece Acrylic Paint Set
4
d.5
B A.4
Arts
Items
6.3 2.2
4.6
Sengoku Basara: The Party
A_1 B
D_6
Vocabulary
jet al., 2022 Covington et al.) 2016). For a long time, recommendation mod- els that represent users and items using their unique IDs (known
Movies
Quantitative Language
Illustration of our MQL4GRec. We translate items from different domains and modalities into new unified language, which can then serve as bridge for transferring recommendation knowledge.
Figure 1:
IDRec) have
been dominant in the field of RS (Kang & [Zhang
McAuley 2018a Sun et al. 2019 | et al. 2024). However, IDRec may en-
counter cold start and knowledge transferability issues due to its inherent properties. To address
"Corresponding authors. 'Our implementation is available at: http://ghhiiicccmhhaaaaaaggggggg

---

Published as a conference paper at ICLR 2025
these limitations, some literature (Hou et al. 2022 Sun et al. 2023) employs modal encoders (De-
vlin et al., 2018 He et al. 2016) to learn universal representations of items or sequences. While promising. these modal encoders are typically not specifically designed for recommendation tasks, resulting in suboptimal performance.
Recently, generative recommendation has emerged promising paradigm, which employs end-
to-end generative model to directly predict identifiers of target candidates (Geng et al. 2022 Rajput
al. | ).23). Due the success of PLMS in natural language generation (NLG) (Raffel et al. 2020a
Brown et al.
2020; Touvron et al. 2023), most existing methods attempt to leverage the prior knowl
edge of PLMs to improve the recommendation performance (Bao al. 2023] Zhang et al. 2023
Zheng et al.
2023). They formalize the recommendation task as a sequence- seuuuennee----quunnee genera- tion process, where the input sequence contains data of items interacted with users, and the output sequence represent identifiers of target items. Then they enable PLMs to perform recommendation tasks by adding instructions or prompts. Despite achieving decent performance, they suffer from the following limitations: 1) There are significant task differences between PLMs and RS, which may lead to inconsistencies between the general linguistic knowledge of PLMs and the specific re- quirements of RS; 2) They often overlook the complementary knowledge between the multimodal information of items, which is crucial for capturing the multi-faceted preferences of users.
To address these limitations, it is crucial to bridge the gaps between different domains and modali- ties, leveraging their recommendation knowledge to enhance the performance of the target domains. Inspired by significant advancements in NLG, such as pretraining- -iintinnninntniinggg (Devlin et al. 2018 Raffel et al. 2020b) and prompt- (Brown et al. 2020 Touvron et al. (023), we propose the idea of transforming items from various domains and modalities into a new and unified language. A key factor contributing to these significant advances is the use of a shared vocabulary, where tokens are endowed with rich semantic information and prior knowledge across various tasks, which can then be effectively transferred to downstream tasks. Thus, we aspire for this new language to en- compass a vocabulary in which tokens can represent items from various domains and modalities, as depicted in Figure [ Specifically, this language not only serves as bridge for knowledge transfer but also as identifiers of items, and should be more concise than the original modalities (text and image) to avoid issues in generation (Hua et al. 2023).
To this end, we propose a novel approach known as Quantitative Language for Multimodal Gen- first introduce quantitative translators to
erative Recommendation (MQL4GRec). Specifically, convert the content of items (text and images) into the quantitative language. We train a separate quantitative translator for each modality of the item, each consisting of modal encoder and vec- tor quantizer. Together, the codebooks of the two quantitative translators constitute the vocabulary. Then, we design series of quantitative language generation tasks aiming at endowing quantitative language with rich semantic information and prior knowledge, and these tasks can be viewed as microcosms of NLG tasks. Specifically, we additionally incorporate some special tokens as task prompts. Finally, we transfer the source domain and multimodal recommendation knowledge to the recommendation tasks through pre-training and fine-tuning. To evaluate the effectiveness of our proposed MQLAGRec, we conduct extensive experiments and comparisons with existing meth- ods. Relative to the baseline, we observe improvements of 11.18% 14.82%, and 7.95% on the NDCG metric across three datasets, respectively. In summary, our proposed MQL4GRee achieves the transfer of recommendation knowledge by breaking down barriers between items across differ- ent domains and modalities, demonstrating strong scalability and potential. Our main contributions can be summarized as follows:
We propose MQL4GRec, novel approach that translates items from various domains and modalities into unified quantitative language, thereby breaking down the barriers between them and facilitating the transfer of recommendation knowledge.
We design a series of quantitative language generation tasks that endow quantitative language with rich semantic information and prior knowledge, and enhance the performance of recom- mendation tasks through pre-trate veetrrannng and fine-tuning.
We conduct extensive experiments and analyses on three public datasets, and the results validate the effectiveness of our proposed method.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORKS
Generative Recommendation. Generative models are one of the hottest research topics in ma- chine learning, resulting in some representative works such as Variational AutoEncoders (VAEs)
(Kingma & Welling 2014), Generative Adversarial Networks (GANs) (Goodfellow al. 2014) and Diffusion models (Ho et al., 2020). Generally, generative models aim to learn the distribution
of the training data (P(x)))nd generate new samples
F(x). These generative models have also
been applied to recommendation, resulting in many remarkable works of VAE-based (Cai & Cai
2022 Shenbin al.||22021 2020), GAN-based (He et al. 2018 Guo al., 2022, Wang al. 2022 and
diffusion-based (Jiang et al. 2024 Wang et al. (023c) recommendation.
Recently, Transformer-basse PLMs such as LLaMA (Touvron et al.) 2023) and GPT (Brown al., 2020 have also shown promising capabilities in language generation. With the help of such power- ful generative PLMs, some PLM-based recommendation methods have also been proposed. Some early works, such as P5 (Geng et al. 2022) and M6-Rec (Cui et al., 2022), attempt to transform recommendation into a language generation task by designing prompts bridge the gap between the downstream task and the pretraining task of PL.Ms. Some works focus on leveraging the prior knowledge in PLMs for recommendation by various tuning techniques such as parameter-efficient fine-tuning (PEFT) (Bao et al. 2023) and instruction tuning (Zhang al. 2023).
One the most important tasks in PLM-based recommendation how to assign an unique se- quence of tokens to each item as its ID. Early works (Geng et al., 2022 Cui et al. 2022 directly use the original name of the item or randomly assign an integer for each item, which have weak transferability and are sometimes unintelligible PLMs. SEATER (Si al. 2023) constructs tree- structured item IDs from pretrained SASRec (Kang& & McAuley (018h) model. P5-ID (Hua et al., 2023) investigates the effect of different item IDs on recommendation. ColaRec (Wang et al.) 2024) captures the collaborative signals between items to construct generative item IDs. Notably, TIGER (Rajput et al. 2023) is the first attempt to use RQ-VAE to construct item IDs by quantizing the item embeddings.
Multi-modal Recommendation. Multi-modal side information of items, such descriptive text and images, has been shown to be effective improving recommendations by providing richer contexts for interactions. Early works such VBPR (He & McAuley 2016) extract visual features by matrix factorization to achieve more personalized ranking. Some works (Wei et al.) 2019 Sun
lt al. 2020 et al. 2020) leverage various types of graph neural network (GNN) to fuse the multi- modal features. For example, LATTICE (Zhang et al., 202T) designs a modality-aware learning layer to learn item-item structures for each modality and aggregates them obtain latent item graphs. DualGNN (Wang et al. (2023b) proposes a multi-modal representation learning module to model the user attentions across modalities and inductively learn the user preference. MVGAE (Yi & Chen (022) uses modality-specifff variational graph autoencoder to fuse the modality-spenn sppciifc node embeddings.
Recently, with the profound development of foundation models in different modalities (Radford al. 202T Brown et al. 2020, Raffel al., 20206), some recent works attempt to leverage pre- trained foundation models as feature encoders to encode the multi-modal side information. Follow- ing P5 (Geng et al. 2022), VIP5 (Geng et al., 2023b) extends it into multi-modal version which encodes the item images by pretrained CLIP image encoder. MMGRec (Liu et al. (024) utilizes Graph RQ- VAE to construct item IDs from both multi- moddl and collaborative information. More- over, IISAN (Fu et al. 2024) propose simple plug-and and-payy architecture using Decoupled PEFT structure and exploiting both intra- and inter-modal adaptation.
METHOD
In this section, we elaborate on the proposed MQL4GRec, a novel approach of transferring rec- ommendation knowledge across different domains and modalities. We first translate item content into a unified quantitative language, which bridge the gaps between different domains and modal- ities. Then, design series of quantitative language generation tasks, and achieve the transfer of recommendation knowledge through pre-training and fine-tuning. The overall framework of the method is illustrated in Figure
3

---

Published as a conference paper at ICLR 2025
Image
Actylic
D.6
ABB
Agmmetric Ieem Generation:
A4
Senpok: Basora: The Line Party
Quaniitaive Langrage Aligooent::
12
B
3
4.6
Pre craining and Fine cuning
Target Sequenee
Prompt Yoken
A
Transfemer lincader
Transformer Decoder
a.
Text
D.
Codebook
Imuge
Figure 2: The overall framework MQLAGRec. We regard the quantizer as translator, convert- ing item content from different domains and modalities into unified quantitative language, thus bridging the gap between them (left). Subsequently, we design a series of quantitative language generation tasks to facilitate the transfer of recommendation knowledge through pre-training and fine-tuning (right).
3.1 QUANTITATIVE LANGUAGE
The original modal content of items is complex, which can affect the efficiency and performance of recommendations (Hua et al.) (023). Therefore, we translate item content from various domains and modalities into a concise and unified quantitative language. In this subsection, we introduce quantitative translator accomplish the aforementioned conversion.
Quantitative Translator. Vector Quantization (VQ) an information compression technique widely utilized across various domains (Van Den Oord et al. 2017 Zeghidour al. 202T). which maps high-dimensional data onto a finite set discrete vectors, known as the codebook. In this paper, we treat the quantizer as a translator that converts complex item content into a concise quan- titative language. Here, the codebook serves as the vocabulary of the quantitative language.
To obtain unified quantitative language, we first employ frozen modal encoder (LLaMA ViT Dosoviiskiy et al., 2020)) to encode item content (text or image), and to obtain the item represen- tation. Further, we take the item representation as input, and train Residual-Quantizze Variational AutoEncoder (RQ-VAF) (Zeghidour et al.) 2021) for generating item tokens. RQ-VAE is multi- level vector quantizer that applies quantization on residuals to generate a tuple codewords (i.e., (left), (left, for an item representation h, RQ-VAE first encodes it into K
item tokens). As shown in Figure
a latent representation z. At each level we have codebook C = { (n k=1 where each codebook vector is a learnable cluster center. The residual quantization process can be represented as:
C4 = arg min ||,, + 11
F++1= o0
(2)
where c; the codeword of the i-th level, r; the residual vector of the i-th level, and Ti = z. As- suming we have L-level codebooks, the quantization representation of z can be obtained according to 2 Σ vi Then & will be used decoder input to reconstruct the item representation h. The loss function can be represented as:
Lrocon = 11 1
(3)
"
Cryyae Σ 1=1
sg [r]]
2i 112 + 1|r, - sg [0]]
(4)
4

---

Published as a conference paper at ICLR 2025
((h) Crecon + Leqvae
(5)
where h is the output of the decoder, sg[*] represents the stop-gradient operator, and [ is loss coefficient. The overall loss is divided into two parts, Lrecon is the reconstruction loss, and Crqvae is the RQ loss used to minimize the distance between codebook vectors and residual vectors.
Items typically encompass content from multiple modalities, representing various aspects of user preferences. In our setup, each item comprises two modalities: text and image. We train quantita- tive translator for each modality, then add prefixes to the codewords from each the two codebooks to form a dictionary. Specifically, for the text quantitative translator, we prepend lowercase letter prefixes to the codewords to obtain Vi = {a_1, 61, 66_.... 2......... d K} J: for the image quantitative transla- tor, prepend uppercase letter prefixes to the codewords to obtain V, = {A_I, 2, ..... D_K). Here, a/A represents the 1-th level codebook, d/ D/D represents the 4-th level codebook, etc. Subse- quently, the dictionary can be represented V {V2, V. With each quantitative translator having LK codewords, the size of our dictionary is 2LK, enabling us to represent a total of K items.
Once the quantitative translators quantitative language. For example, for the item text "Sengoku Basara: The Last Party". ", after encoding it through the text encoder and RQ-VAE, we obtain set of codewords (2, 3, 1, 6). Then, by appending lowercase letters before each number, we can get the text quantitative language of the item <a_ <a_2><b__ 3><c 1><d 6>. Similarly, for the item's image, we can obtain its image quantitative language as <A /></>>B 4><0_ <><<__><666>>
trained,
directly use them to translate new items into
Handling Collisions. Translating item content into quantitative language may lead to item colli- sions, where multiple items possess the same tokens. To address this issue, some methods (Rajput jet al., 2023: Hua et al.] (023) append an additional identifier after the item indices, which may introduce semantically unrelated distributions. LC-Rec (Zheng al. (023) introduces uniform distribution constraint to prevent multiple items from clustering in the same leaf node. However, this method does not completely resolve collisions, such when items have the same modality in- formation or when the number of collisions exceeds the size of the last level codebook, which can lead to inflated performance metries. (More discussion in Appendix E.I
To address the above issue, we reallocate tokens for colliding items based on the distance from the residual vector to the code vectors. Specifically, for N colliding items, we first calculate the distances D E RN LxK between the residual vectors and the code vectors for each level based on di r; - "illzz and sort the distances to obtain the indices I = argsort( D, azis = 2) E N Then, we sort the colliding items based on their minimum distance to the code vectors mintmin (colliding items). Finally, we re- allocate tokens for the sorted colliding items based on I, following these principles: 1) Start from the last level assign the nearest token each item. If collisions occur, assign the next nearest token. 2) If there are insufficient tokens in the last level, for the remaining colliding items, reallocate tokens from the second last level based on distance, and then reallocate tokens from the last level. We repeat this process until all colliding items are handled.
of the last level, i.c., (item1, item....... itemy)
(colldiccooliiing
3.2 QUANTITATIVE LANGUAGE GENERATION TASKS
In this subsection, we design several quantitative language generation tasks with the aim of imbuing quantitative language with more semantic information, thereby transferring prior knowledge the target task, as illustrated in Figure 2 (right). Specifically, we additionally include some special tokens in the dictionary, which can serve as prompts to differentiate the types of tasks.
Next Item Generation. Since our primary goal is to predict the next item, the next item genera- tion task is our main optimization objective. Specifically, each item contains both text and image modalities, so we have two subtasks: 1) Next Text Item Generation; 2) Next Image Item Generation. In this context, the input sequence is the item tokens sequence from the user interaction history, and the output sequence is the target item tokens corresponding to the respective modality. Different modal sequences reflect different aspects of user preferences.
Asymmetric Item Generation. In the next item generation task, the input and output are to- kens of the same modality, and refer to this task as symmetric. To facilitate the interaction
5

---

Published as a conference paper at ICLR 2025
of recommendation knowledge between two modalities, we introduce asymmetric item generation tasks. Here, there are two subtasks: 1) Asymmetric Text Item Generation, where the imput is the image tokens of the interaction history items, and the output is the text tokens of the target item; 2) Asymmetric Image Item Generation, where the input is the text tokens of the interaction his- tory items, and the output is the image tokens of the target item. For example, for the input quence "<* "<*_6><* 6__66>>*=>>> <<<_"><<___>> ______s__ <a_<a__>a fi__>>>b3><>_ 3><c_>><<>> _ecd_l><<aa 6><a 4><b_3><< 1i___-ee 8><d_6> __"6"", in human- understandable language,
be described follows: "Based on the user's text interaction se- quence, please predict the next item's 's image quantitative language: <a <a_2>bb_3 3>cc__3c__________ l><d 6>,
<a_ <a_4><b__> b>>bb_3><c_ 8><d 2 6>".
Quantitative Language Alignment Asymmetric item generation tasks enable the interaction of knowledge between two modalities, but they fall under the category of implicit alignment of the two modalities. We further introduce explicit Quantitative Language Alignment tasks to directly achieve alignment between the text and image quantitative languages of items. Here, we also have two subtasks: 1) Text-to-lmage Alignment; 2) Image-to-Text Alignment. For example, for the (4><a_2><b <b_3><c_l><d
input sequence "<" ">__>><<_13>>>>
in human-understandable language. it can be described as follows: "Please provide the image quantitative language for the
following item: <a aa__>><b_>> >><bb3><c__c $><c_l><d_3>
3.3 TRAINING AND RECOMMENDATION
Training. Quantitative language can be viewed as microcosm natural language. We employ a two-stage paradigm of pre-training and fine-tuning to optimize the model, which is similar to NLG tasks. For pre-training, we utilize the source domain datasets, where the pre-training task consists of two sub-tasks for next item generation. The purpose is transfer recommendation knowledge from the source domains to the target domains. For fine-tuning, we conduct it on the target domain dataset, with tasks encompassing all quantitative language generation tasks. The aim is leverage recommendation knowledge from different modalities to explore users' multifaceted preferences. The tasks mentioned above are conditional language generation tasks performed in sequence-to- sequence manner. We optimize the negative log-likeliiee likeelooooood of the generation target as follows:
Co log Po (Y, X),
(6)
where 0 is the model parameters, X is the input sequence of encoder, and YJ is the j-th token of Y.
Re-ranking for recommendation. There are two sub-tasks in the next item generation task, rep- resenting different user preferences. Although fine-tuning tasks can facilitate the transfer of recom- mendation knowledge between them, there might be some information loss. Therefore, we re-rank items by utilizing the recommendation lists generated from the two sub-tasks. The basic idea is that items appearing in both lists should be ranked higher. Specifically, we first obtain recommendation lists R. and R, for each sub-task through beam search, which include scores for each item. Then, the new score for each item can be formalized as:
(s)(x) (x))222++1 (I)
x E Re, E R., x E Rg
(x)
(x)
x E R.
where s;(x) is the score of item æ in the list R. and i E v).
4 EXPERIMENTS
4.1
EXPERIMENTAL SETTINGS
Datasets. We evaluate the proposed approach on three public real-world benchmarks from the Amazon Product Reviews dataset (Ni et al. 2019), containing reviews and item metadata from May 1996 to October 2018 In particular, we use six categories for pre-training. including "Per Supplies", "Cell Phones and "ccessories". "Automotive". "Tools and Home Improvement". "Toys

---

Published as a conference paper at ICLR 2025
Table I: Performance comparison of different methods on the three datasets. The best and second-
best performances are indicated in bold and underlined font, respectively.
BFRT4Rcc GRRt4Roz
51-Rec
VQ-Rec
MiSSBee
PS-Cin
VIPS
Imprrvv.
HRØi
0.0566
0.0450
0.0039
0.5002
0.023
0.6833
+10.48%
0,9955
0.0856
0,1062
01989
0.8892
Insouments
0.1081
0.1123
0,1257
01361
0.0071
0,1221
NDCGeS
03783
0.8667
0.093
0,0766
00027
0:878
0.3815
+10.77%
1DCG
018857
0,0739
0.0746
0.0859
0,073
0.0891
0.8800
0.070
01882
00950
0.1060
+11.58%
011365
0.0289
0.0172
0.0005
03477
0:042
0.0444
0.0672
HR
0.0817
0.0897
0.001
0.0832
0.0730
0.1038
01021
0:0713
0.704
00894
1.1037
Ans
0.88
0.0933
1220
0.1190
0
0.138
01331
0.0994
0,889
11167
NDCORS
0.0902
0.0902
0,010
0.0583
0.611
0.7522
0069
0:3007
0.0566
0.718
13887
NDCG810
0,0580
0.0575
0.0706
0.0995
0.063
0184
0.0815
0.0662
0.0655
0.0806
0,0950
+2.55%
HR@1
0.0140
0.0115
0.0099
0,0163
0.0136
0.0755
0,001
0.0169
0
00156
0,8203
+1.00%
IRR-5
0.0544
0.0126
0,0557
0.0615
0.621
0.000
0.067
0.0532
0.0400
0.0223
0.0657
Games
BR
0.0895
0,0725
0,9K3
0
0.067
0.1648
0.0834
0.0755
0.3033
NDCGB5
0.0344
0/1270
0
0.0099
0.051
0.0242
0.0355
0.0331
000325
110345
1.8421
+8.2316
NDCGRR0
0.0459
0.0366
0.0461
0.0509
0 0.0468
0.0129
0.0499
0.0454
0.0488
00451
0.85448
-7.66%
Table Ablation study of handling collisions.
Methods
Instruments
Games
HR 69 10
NDCG 10
HR 10
NDCG 10
HR@ 10
NDCG@10
TIGER
0.1221
0.0950
0.1167
0.0806
0,0857
0.0453
TIGER w/o user
0.1216
0.0958
0.1159
0.0810
0.0863
0.0464
Handling Collisions
0.1277
0.0987
0,1
0.0844
0.0885
0.0473
and Games", "Sports and Outdoors". and three categories for sequential recommendation tasks, in- cluding "Musical Instruments", "Arts Crafts and Sewing", "Video Games". We discuss the dataset statistics and pre- preppeeeesing in Appendix [A]
Evaluation Metrics. We use top-k Recall (Recall K) and Normalized Discounted Cumulative Gain (NDCG@ K) with = 1. 5, 10 to evaluate the recommendation performance. Following previ- works (Geng et al. 2022 Hua et al. 2023), we employ the leave-one-ond strategy for evaluation. We perform full ranking evaluation over the entire item set instead of sample-based evaluation. For the generative methods based on beam search, the beam size is uniformly to 20.
4.2 OVERALL PERFORMANCE
In this section, we compare proposed approach for generative recommendation with the follow-
ing sequential recommendation methods (which are described briefly in Appendix B): GRU4Rec
(Hidasi et al. 2015), BERT4Rec (Sun et al. 2019), SASRec (Kang & McAuley (018b). FDSA
Zhang et al. 2019), -Re (Zhou et al. 2020), VQ-Rec (Hou et al. 2023). MISSRec Wang et al.
2023a), P5-CID (Hua et al. 2023). VIP5 (Geng et al.) (202a)), and TIGER (Rajput et al. 2023).
Results are shown in Table] Based on these results, we can find:
For non-generative recommendation methods, MISSRec often achieves better performance in most cases, demonstrating that introducing multimodal information of items can enhance recommenda- tion performance. For generative baseline methods, VIP5 with image information does not achieve good results, which may be due to the modal differences between PLMs and image information. Furthermore, TIGER performs well on the Instruments and Arts datasets but does not exhibit supe- riority on the Games dataset. This may be due to TIGER's lack of auxiliary content information. In contrast, our proposed method introduces recommendation knowledge from different domains and modalities.
Compared to baseline methods, our proposed MQL4GRec achieves the best performance in most cases, especially with significant improvements on the NDCG metric. This superior performance can be attributed to two factors: 1) We translate item content from different domains and modalities into unified quantitative language, breaking down barriers between them; 2) The series of QLG tasks we designed enable the transfer of recommendation knowledge to target tasks through pre- training and fine-tuning methods.
7

---

Published as a conference paper at ICLR 2025
Table 3: Ablation study of various quantitative language generation tasks without pre-training.
Tasks
Instruments
Arts
Games
HR 10
NDCG@10
HR@ 10
NDCG@ 10
HR@10
NDCG@10 10
NIG,
0.1277
0.0987
0.1163
0.0885
0.0473
Text
NIG
0.1275
0.0986
0.1205
0.0877
0.0928
0.0493
+ AIG
0.1279
0.0987
0.1002
0.0529
+ QLA
0.1282
0,0993
0.1293
0,0913
0.1010
0.0531
NIGa
0.1243
0.0968
0.1117
0.0812
0.0881
0.0478
Image
NIG
0.1262
0.0986
0.1158
0,0848
0.9899
0.0487
AIG
0.1299
0.0998
0.1218
0.0878
0,1002
0.0534
+ QQLA
0.1280
0.1001
0.1017
0.0540
4.3 ABLATION STUDY
Handling Collisions. We propose a method based on the distance between the residual vector and the codeword vector to resolve item collisions. To validate the effectiveness of our method, we com- pare it with the collision resolution approach TIGER, which directly adds an item index layer to resolve item collisions, thereby introducing semantically unrelated distribution. The experimental results are shown in Table 2 "TIGER w/o user" refers to the removal of the user ID token from the input sequence, which is also done to facilitate fairer comparison. From the experimental results, can be seen that our method of handling collisions is more rational and effective. Furthermore, results indicate that including user ID token in the input degrades model performance, particularly on the Games dataset. We attribute this to TIGER representing tens of thousands of users with only 2000 tokens, leading to numerous user ID collisions.
Quantitative language generation tasks. We initially assess the effect of different QLG tasks on performance without the use of pre-training, and the results are shown Table (For more detailed results, please refer Appendix
D.I Various tasks include: (1) NIG: the next item generation task introduced in Section 3.2. (2) AIG: the asymmetric item generation task; (3) QLA: the quantitative language alignment task. In this list, tasks without subscripts indicate that two subtasks used simultaneously. "Text" denotes evaluating performance by utilizing the next text item generation subtask (i.e., NIG,); "Image" signifies evaluating performance by utilizing the next image item generation subtask (i.e., NIG2).
The results indicate that several quantitative language generation tasks designed by can signifi- cantly improve performance. Specifically, as the number of tasks increases, the performance of both NIG, and NIG, improves. This indicates that these tasks can enrich the quantitative language by incorporating semantic information and knowledge across different modalities. In summary, con- verting the multimodal content items into a unified quantitative language effectively facilitates the transfer of recommendation knowledge.
Pre-training. We transfer recommendation knowledge from the source domain datasets to the target dataset through pre-training. Here, we employ NIG, evaluate the recommendation perfor- mance, with the results shown in Table (Additional results can be found in Appendix D.2) QLG represents the quantitative language generation tasks without pre- pre-training Specifically, y, the pre- training task labeled "NIG, w/ pre-truining" employs only NIG, from the source domain datasets. On the other hand, the "QLG w/ pre-training" task uses both NIG, and NIG₂-
The results indicate that, under a single modality, pre-training enhances the performance across three downstream datasets, demonstrating that prior knowledge from the source domain can be effectively transferred downstream tasks. Under dual modalities, pre-training significantly improves perfor- mance on the Instruments and Arts datasets; however, it does not yield notable improvement for the Games dataset, potentially due to overfitting. A more intuitive analysis of this phenomenon is provided in Section 4.4 Finally, we re-rank the items according to Equation 7 to generate the final recommendation list.
8

---

Published as a conference paper at ICLR 2025
Table 4: Ablation study of pre-training and quantitative language generation tasks.
Instruments
Arts
Games
Methods
HR@
NDCG@10
HR@ 10
NDCG@10
HR@10
NDCG@ 10
(0) NIG,
0.1277
0.0987
0.1163
0.0844
0.0885
0.0473
(1) QLG
0.1282
0.0993
0.1293
0,0913
0.1010
0.0531
(2) NIC, w/ pre-training
0.1334
0. 1.1043
0.1305
0.0959
0.0950
0.0508
QLG w/ pre-training
0.1362
0.1051
0.1314
0.0944
0.0995
0.0521
(4) MQL4GRec ((3) re-ranking)
0.1375
0.1060
0.1327
0.0950
0,1033
0.0548
NiG; w/ pre traing
QLG w/ pre traininnn
0.140
0.135
0.105
0.135
0.130
0.100
HR@10
HR810
0.125
0.120
0.090
0.120
0.115
0.085
Figure 3: The impact of varying amounts of pre-training datasets recommendation performance.
4.4 FURTHER ANALYSIS
Pre-training datasets. In this subsection, we investigate the impact of varying amounts of pre- training datasets on downstream tasks, and the results are shown in Figure 3 From the results, it can observed that: 1) Exclusively the context of text quantitative language, as number of pre-training datasets increases, the performance of downstream tasks also gradually improves. This suggests that larger numbers pre- training datasets provide more transferable recommenda- tion knowledge. 2) Following pre pre-training with quantitative language under two modalities, fine- tuning shows varying trends across different downstream datasets. Specifically, while increasing the Instruments and Arts datasets, it leads grad- ual decline in performance on the Games dataset. This could indicate either overfitting or significant domain differences between the Games dataset and the source domain datasets.
pre-training datasets enhances performance
Pre-training epochs In this subsection, we investigate the impact of varying the number of pre- training epochs on downstream tasks, with the results displayed Figure 4 From the figure, can be observed that: 1) When pre-training is performed solely with text quantitative language, the performance of downstream tasks gradually increases with the number of pre-training epochs and stabilizes around 25 epochs. 2) When pre-training involves both text and image quantitative languages, the Instruments and Arts datasets reach peak performance early, and further training may impair the transfer of recommendation knowledge. In contrast, for the Games dataset, performance deteriorates as the number of pre- training epochs increases. This suggests that for the Games dataset, recommendation knowledge from different modalities might be more crucial than that from the source domain dataset, and there may be conflicts between the two.
Table 5: Zero-shot capabilities under different number of pre-training datasets.
Instruments
Arts
Games
Numher
HR@10 10
NDCG@10
HR
NDCG@ 10
HR
NDCG@
0.00099
0.00046
0.00113
0.00052
0.00066
0.00031
2
0.00240
0.001337
0.00140
0.00063
0.00109
0.00058
0.00310
0.00170
0.00298
0.00132
0.00054
0.00027
0.00345
0.00171
0.00116
0.00047

---

Published as a conference paper at ICLR 2025
0.14
0.14
HR@lo
0.12
HR@10
0.12
0.10
0.10
0.08
0.08
15
20
25
0
15
20
25
NIG; W/ pre-traing
QLG w/ pre-training
Figure 4: The impact of different pre-training epochs recommendation performance.
4.5 ZERO-SHOT CAPABILITY
We investigate whether models pre-trained on the source domain dataset have zero-shot capabilities, as shown in Table 5 Here, "Number" represents the number of pre-training datasets, with "0" indi- cating model parameters randomly initialized. We use NIG, to evaluate performance. The results demonstrate that pre-trained models exhibit preliminary zero-shot capabilities the Instruments and Arts datasets, although they are still weak. However, this capability is not evident on the Games dataset. We attribute this primarily to the scarcity of pre-training data and the limited parameters of the model, resulting in insufficient generalization. In the future, we aim to delve deeper into this phenomenon.
CONCLUSION
In this paper, we propose novel approach named MQL4GRec, which transforms item content from different domains and modalities into unified quantitative language to facilitate effective transfer of recommendation knowledge. We first train quantitative translator for each modality, converting items into the guantitative language and breaking down the barriers between them. Then, design series of quantitative language generation tasks aiming endowing quantitative lan- guage with rich semantic information and prior knowledge. Finally, we transfer the source domain and multimodal recommendation knowledge the recommendation tasks through pre-training and fine-tuning. Our proposed MQLAGRec achieves superior performance compared to the baseline method. Moreover, MQL4GRec possesses strong scalability and potential it does not rely on traditional item IDs and bridges the gap between different domains and modalities. We believe this represents a significant step towards universal recommendation models.
ACKNOWLEDGMENNTS
This work supported by the National Natural Science Foundation of China (62206141, 62425101, 62332002, 62276277), the National Key Research and Development Pro- gram of China YF02 the Guangdong Basic and Applied Basic Research Foundation (2022B1515120059, and the Shenzhen Science and Technology Program (KQTDD29240722100001066) In addition, we are grateful to Dr. Zhengyu Ma for her support.
REFERENCES
Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. Tallrec: An effective and efficient tuning framework align large language model with recommendation. In Jie Zhang. Li Chen, Shlomo Berkovsky, Min Zhang, Tommaso Di Noia, Justin Basilico, Luiz Piz- zato, and Yang Song (eds.), Proceedings of the 17th ACM Conference RecSys 2023. Singapore, Singapore, September 18-22, 2023, pp. 1007 - ACM, 2023. doi: 10.1 2..144/3604915.360
Recommender Systems,
htR.pttss:t. / org 10 15 111p://...1///1011151..1 3608885
10

---

Published as a conference paper at ICLR 2025
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Πya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiees Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL htttps://proceeedingg neur ips. cc/papper/222/haatt/ 115770d6bfcb499777 8ofb8ac1 42f64a Ao
Abstract. html
Zefeng Cai and Zerui Cai. PEVAE: A hierarchical VAE for personalized explainable recommen- dation. In Enrique Amigó, Pablo Castells, Julio Gonzalo, Ben Carterette, J. Shane Culpepper, and Gabriella Kazai (eds.), SIGIR '22: The 45th International ACM SIGIR Conference on Re- search and Development in Information Retrieval, Madrid, Spain, July 11 15, 2022, pp. 692- doi: 10.144534774995.5522099
702. ACM, 2022.
URL https: wttp::///ooooooo..oo../oooo 1145/
3477495. 3532033
Pedro Dalla Vecchia Chaves, Bruno L Pereira, and Rodrygo LT Santos. Efficient online learning to rank for sequential music recommendation. Proceedings of the ACM Web Conference 2022,
pp. 2442 2450, 2022.
Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference recommender systems, pp. 191-198, 2016.
Zeyu Cui, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. M6-rec: Generative pre- trained language models open-ended recommender systems. CORR, abx/2205.08084, 2022. doi: 10:4850/AARII.2205.00
URL
08084
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04880, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 11999, 2020.
Junchen Fu, Xuri Ge, Xin Xin, Alexandros Karatzoglou, Ioannis Arapakis, Jie Wang, and Joemon M Jose. lisan: Efficiently adapting multimodal representation for sequential recommendation with decoupled peft. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 687-697, 2024.
Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation as language processing (RLP): A unified pretrain, personalized prompt predict paradigm (P5). In Jennifer Golbeck, F. Maxwell Harper, Vanessa Murdock, Michael D). Ekstrand, Bracha Shapira, Justin Basilico, Keld T. Lundgaard, and Even Oldridge (eds.), RecSys '22: Sixteenth ACM Confer- ence on Recommender Systems, Seattle, WA, USA, September 18 23, 2022, pp. 299-319 ACM, URL https:://dot www.:/i...rrr.....0.....0
2022. doi: 10. 10.1144//523227.356667 3546767
/0.
Shijie Geng, Juntao Tan, Shuchang Liu, Zuohui Fu, and Yongfeng Zhang. Vip5: Towards multi- modal foundation models for recommendation. arXiv preprint arXiv:2305.14332. 2023a.
Shijie Geng, Juntao Tan, Shuchang Liu, Zuohui Fu, and Yongfeng Zhang. VIP5: towards mul- timodal foundation models for recommendation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singa- pore, December 6-10, 2023, pp. 9606-9620 Association for Computational Linguistiss, 2023b. doi: 10.18653/V SINNSIIN-EGNGGGNELPAAAA URL http::://00...0////1111163 v1/2023. findi ngs 644
11

---

Published as a conference paper at ICLR 2025
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial networks. CoRR, abs/ ftos//400.2661,,, 2014. URL/ttp:: htttt:///axivi/ http://www..rr//////000..0.
Guibing Guo, Huan Zhou, Bowei Chen, Zhirong Liu, Xiao Xu, Xu Chen, Zhenhua Dong, and Xiuqiang He. IPGAN: generating informative item pairs by adversarial sampling. IEEE Trans. Neural Networks Learn. Syst., 33(2):694-006 2022. doi: 10.1 109/TNNLS 202003.2872200 URL 0228572
https: //doit .rgg/0 1109/TNNI.S
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Ruining He and Julian J. McAuley. VBPR: visual bayesian personalized ranking from implicit feedback. In Dale Schuurmans and Michael P. Wellman (eds.), Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pp. 144- 150. AAAI Press, 2016. doi: 11.00//AAALV3111.00000 1609/ /wt/aaaaii......../.aa...aaaaa.. v30il. 9973
URL Sthhtt::: /tttps://oo.. http///i..g///////
Xiangnan He, Zhankui He, Xiaoyu Du, and Tat-Seng Chua. Adversarial personalized ranking for recommendation. In Kevyn Collins- Tholiisn, Qiaozhu Mei, Brian D. Davison, Yiqun Liu, and Emine Yilmaz (eds.), The 41st International ACM SIGIR Conference on Research & Develop- ment Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08- 12, 2018, pp. 355- doi: 01145/3209998.3209999 URL https /oii org 10. 10.1/11..111//1. 1145/
364. ACM, 2018.
3209978. 3209981
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-based rec- ommendations with recurrent neural networks. arXiv preprint arXiv:1511.0693 2015.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochele,, Mare Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020, URL https: httpt:::://ppooeednnggs neweeedddinnnnn.nnrs..ee.. neurips. cc/ppaper/222//aaeeeeeeeeeeeeee 4cboofec3584af0dd5ee
79ca4b Abstract html
Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. Towards uni- versal sequence representation learning for recommender systems. In Aidong Zhang and Huzefa Rangwala (eds.), KDD '22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14 18, 2022, pp. 585-593 593. ACM, 2022. doi: 10.1145/3534678.3555 35339338... URL https:: //o::///oo../// org/ 10. 1145/ /5155333330 353467334477 3539381
Yupeng Hou, Zhankui He, Julian McAuley, and Wayne Xin Zhao. Learning vector-quantized item representation for transferable sequential recommenders. In Proceedings of the ACM Web Con- ference 2023, pp. 1162-1171, 1171, 2023.
Wenyue Hua, Shuyuan Xu, Yingqiang Ge, and Yongfeng Zhang. How to index item ids for rec- ommendation foundation models. In Qingyao Ai, Yiqin Liu, Alistair Moffat, Xuanjing Huang, Tetsuya Sakai, and Justin Zobel (eds.), Annual International ACM SIGIR Conference on Research and Development in Information Retrieval the Asia Pucific Region, SIGIR-AP 2023, Beijing, China, November 26-28, 2023, pp. 195-204 ACM, 2023. doi: 015553622491.33255553 URL 10. wtt....///32211....... 3625339
httpp::/ddo
Yangqin Jiang, Yuhao Yang, Lianghao Xia, and Chao Huang. Diffkg: Knowledge graph diffusion model for recommendation. In Luz Angelica Caudillo-Mata, Silvio Lattanzi, Andrés Muñoz
Medina, Leman Akoglu, Aristides Gionis, and Sergei Vassilvitskii (eds.), Proceedings of the 17th
ACM International Conference on Web Search and Data Mining, WSDM 2024, Merida, Mexico, March 4-8, 2024, pp. 313 321. ACM, 2024. doi: 10.1 2.1114536668855.66
URL https:
//doi. org. 10. 0. 1 114114/55666 6855 3635850
Wang- Cheng Kang and Julian McAuley. Self-attentive sequential recommendation. In 2018 JEEE international conference on data mining (ICDM), pp. 197-206. IEEE, 201 8a.
12

---

Published as a conference paper at ICLR 2025
Wang-Cheng Kang and Julian J. McAuley. Self-attentive sequential recommendation. In IEEE International Conference on Data Mining, JCDM 2018, Singapore, November 17-20. 2018, pp. 197-206. IEEE Computer Society, 2018b. doi: 10.1109/CDDM.201..0005. URL https://
doi wwt./org11..... 1109/.1000..MMM.. 2018 000355
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14- 16, 2014, Conference Track Proceedings, 2014. http://tt:///xxiii
abs/ 1////312...0 6114
Han Liu, Yinwei Wei, Xuemeng Song, Weili Guan, Yuan-Fang Li, and Liqiang Nie. Mmgrec: Mul- timodal generative recommendation with transformer model. arXiv preprint arXiv:2404.16555, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL /RLLLttps.//opeeeee.ee... wwwp://wwwww.ee//om.mmm.m.m B
Jianmo Ni. Jiacheng Li, and Julian McAuley, Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp. 188-197,11 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Ma- rina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Ma- chine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Ma- mml. mir.
chine Learning Research, pp. 8748 8763. PMLR, 2021. Dzess/v139/aaford222 html
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020a. URL http:
httppeerr/////00 /200 html
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. J. Mach. Learn. Res., 11:140:1-400:7. 2020b. httphhttp://mm org/
papers/v21/20- html
Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan Hulikal Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Q Tran, Jonah Samost, Maciej Kula, Ed H. Chi, and Ma- hesh Sathiamoorthy. Recommender systems with generative retrieval. In Alice Oh, Tris- tan Naumann, Amir Globerson, Kate Saenko. Moritz Hardt, and Sergey Levine (eds.), Ad- vances in Neural Information Processing Systems 36: Annual Conference on Neural Infor- mation Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: httpt::pppppppss..eee wwtpppppppppppppppppppppp..ppe.p www.s/ wttp:///ts/eepee///e/eeeeeeeeeee 20dcab0f 4404655660226616666 htw.://bbea..11122200oaato------ html]
Ilya Shenbin, Anton Alekseev, Elena Tutubalina, Valentin Malykh, and Sergey Nikolenko. Rec- vae: A new variational autoencoder for top-n recommendations with implicit feedback. In James Caverlee, Xia (Ben) Hu, Mounia Lalmas, and Wei Wang (eds.), WSDM '20: The Thirteenth ACM International Conference Web Search and Data Mining, Houston, TX. USA, Febru- ary 3-7, 2020, pp. 7/doi. ridei.org/10. 10. 115/1133111333333333333
ACM, 2020. doi: 10.1145/3361911:331888 URL https:
Zihua Si, Zhongxiang Sun, Jiale Chen, Guozhang Chen, Xiaoxue Zang, Kai Zheng, Yang Song, Xiao Zhang, and Jun Xu. Generative retrieval with semantic tree-structured item identifiers via contrastive learning. CORR, abs/2309.11375. 2023, doi: 10.48550/ARXIV2230011 https://doi http:///ooo.oggg1..o
URL
13

---

Published as a conference paper at ICLR 2025
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequen- tial recommendation with bidirectional encoder representations from transformer. Proceedings of the 28th ACM International Conference Information and Knowledge Management, 2019.
Rui Sun, Xuezhi Cao. Yan Zhao, Junchen Wan, Kun Zhou, Fuzheng Zhang, Zhongyuan Wang, and Kai Zheng. molti--oddal knowledge graphs for recommender systems. In Mathieu d' Aquin, Stefan Dietze, Claudia Hauff, Edward Curry, and Philippe Cudré-Mauroux (eds.), CIKM '20: The 29th ACM International Conference
Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020, pp. 1414. ACM, 2020. doi: 10.1145/3340531.33 3311199771
URL https: /ttp:://doi..oo org 11.00.11450 3340531
0111///11115333331111111 3411947
Wenqi Sun, Ruobing Xie, Shuqing Bian, Wayne Xin Zhao, and Jie Zhou. Universal multi-modal multi-domain pre-trained recommendation. arXiv preprint arXiv:2311 13.0033,,, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023. doi: 10.4850/ARXIV.2300.1155 httpps:/doi. wrgg100. 1.000.4855500 48550/ wrwiv. 2wt.iii..33000 2302. 13971
URL
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Jinpeng Wang, Ziyun Zeng, Yunxiao Wang, Yuting Wang, Xingyu Lu, Tianxiang Li, Jun Yuan, Rui Zhang, Hai-Tao Zheng, and Shu-Tat Xia. Missrec: Pre-training and transferring multi-modal anterest-aware sequence representation for recommendation. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 6548-6557, 2023a.
Qifan Wang, Yinwei Wei, Jianhua Yin, Jianlong Wu, Xuemeng Song, and Liqiang Nie. Dualgnn: Dual graph neural network for multimedia recommendation. IEEE Trans. Multim,, 25:1074- 1084, 2023b. doi: 00.119/TMM.2021..10 3338998.
ottp:ooorg/1.... 1199/MMM
2021 0221.3138299
Wenjie Wang, Yiyan Xu, Fuli Feng, Xinyu Lin, Xiangnan He, and Tat-Seng Chua. Diffusion rec- ommender model. In Hsin-Hsi Chen, Wei-Jou (Edward) Duh, Hen-Hsen Huang. Makoto P. Kato, Josiane Mothe, and Barbara Poblete (eds.), Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023, Taipei, Taiwan, July 23-27, 2023, pp. 832-841, ACM, 2023c. doi: 1111445/35391883591666 URL 3591663
wwg.ooggg1000 wttp://....11111310 1......1155599966 https: http:///oooo..g/1...o
Yidan Wang, Zhaochun Ren, Weiwei Sun, Jiyuan Yang. Zhixiang Liang. Xin Chen, Ruobing Xie, Su Yan, Xu Zhang, Pengjie Ren, Zhumin Chen, and Xin Xin. Enhanced generative recommen- dation via content and collaboration integration. CoRR, abs/2403. 184800 2024. doi: 10.48550/ ARXIV.2403.18.0 18480. htthttss:: http://oc....gg// 0t11//1..4555///40 4855048555//// 1tt.iii.4440.. 2403. 11033.8844 18480
Zhidan Wang, Wenwen Ye, Xu Chen, Wenqiang Zhang. Zhenlei Wang. Lixin Zou, and Weidong Liu. Generative session-based based recommendation. In Frédérique Laforest, Raphaël Troncy, Elena Simperl, Deepak Agarwal, Aristides Gionis, Ivan Herman, and Lionel Médini (eds.). WWW '22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 29, 2022, pp. 2227- URL https: / org 10..
2235. ACM, 2022. doi: 10. 10.1145/3485447735122055
3485447 3512095
Yinwei Wei, Xiang Wang. Liqiang Nie, Xiangnan He, Richang Hong, and Tat-Seng Chua. MMGCN: multi-modal graph convolution network for personalized recommendation of micro- video. In Laurent Amsaleg, Benoit Huet, Martha A. Larson, Guillaume Gravier, Hayley Hung. Chong Wah Ngo, and Wei Tsang Ooi (eds.), Proceedings of the 27th ACM International Con- ference on Multimedia, MM 2019, Nice, France, October 21-25, 2019, pp. 1437-1445. ACM, 2019. doi: 10. 33351034, URL 3351034
org 0. 1145/3343033
Yinwei Wei, Xiang Wang, Liqiang Nic, Xiangnan He, and Tat-Seng Chua. Graph-refined convolu- tional network for multimedia recommendation with implicit feedback. In Chang Wen Chen, Rita
14

---

Published as a conference paper at ICLR 2025
Cucchiara, Kian-Sheng Hua, Guo-Jun Qi, Elisa Ricci, Zhengyou Zhang, and Roger Zimmermann (eds.), MM '20: The 28th ACM International Conference on Multimedia, Virtual Event Seattle, WA, USA, October 12-16, 2020, pp. 3541-3549. ACM, 2020. doi: 10.1145/339417133413566
URL
34111554455
Jing Yi and Zhenzhong Chen. Multi-modal variational graph auto-encoder for recommendation systems. IEEE Trans. Multim. 24: 14:1067-1779 2022. doi: 10.1109TMM.201.3114451 URL
https://doi.ozgg
1109/
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
Jiaqi Zhang, Yu Cheng, Yongxin Ni, Yunzhu Pan, Zheng Yuan, Junchen Fu, Youhua Li, Jie Wang, and Fajie Yuan. Ninerec: A benchmark dataset suite for evaluating transferable recommendation. IEEE Transactions Puttern Analysis and Machine Intelligence, 2024.
Jinghao Zhang, Yanqiao Zhu, Qiang Liu, Shu Wu, Shuhui Wang, and Liang Wang. Mining latent structures for multimedia recommendation. In Heng Tao Shen, Yueting Zhuang, John R. Smith, Yang Yang. Pablo César, Florian Metze, and Balakrishnan Prabhakaran (eds.), MM '21: ACM Multimedia Conference, Virtual Event, China, October 20 24, 2021, pp. 3872-3880. ACM, 2021. doi: 10.1445744740853445559. URL https: / http:/ddi..ooo /ttD:://abi.ozg/00.11152. 01..0/0///0011144400040000000 3475259
Junjie Zhang, Ruobing Xie, Yupeng Hou, Wayne Xin Zhao, Leyu Lin, and Ji-Rong Wen. Rec- ommendation as instruction following: A large language model empowered recommendation doi: I048850/ARXXV.2355.07000 URL https:
approach. CoRR, abs/2305.07001, 2023.
7/doi. oww..org11...m 48w50/arxi........ 2305. 07001
Tingting Zhang, Pengpeng Zhao, Yanchi Liu, Victor S. Sheng, Jiajie Xu, Deqing Wang, Guanfeng Liu, and Xiaofang Zhou. Feature-level deeper self-attention network for sequential recommenda- tion. In International Joint Conference on Artificial Intelligence, 2019.
Bowen Zheng, Yupeng Hou, Hongyu Lu, Yu Chen, Wayne Xin Zhao, and Ji-Rong Wen. Adapting large language models by integrating collaborative semantics for recommendation. arXiv preprint arXiv:2311 1.09049, 2023.
Kun Zhou, Haibo Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, and Ji rong Wen. S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. Proceedings of the 29th ACM International Conference on Information & Knowledge Management, 2020.
15

---

Published as a conference paper at ICLR 2025
A
DATASET STATISTICS
Table 6: Statistics of the preprocessed datasets. "Avg. len" represents the average length of item
sequences.
Datasets
#Users
#ltems
#Interactions
Sparsity
Avg. len
Pet
183697
31986
1571284
99.97%
8.55
Cell
123885
38298
873966
99.98%
7.05
Automotive
105490
39537
845454
99.98%
8.01
Tools
144326
41482
1153959
99.98%
8.00
Toys
135748
47520
1158602
99.98%
8.53
Sports
191920
56395
1504646
99.99%
7.84
Instruments
17112
6250
136226
99.87%
7.96
Arts
22171
9416
174079
99.92%
7.85
Games
42259
13839
373514
99.94%
8.84
To evaluate the performance of the proposed approach, we conduct experiments in the pre-trained source and target domain settings. We use six categories from from the Amazon Product Reviews dataset (Ni et al. 2019) for pre-training, including "Per Supplies", "Cell Phones and Accessories", "Automotive", ", "Tools and Home Improvement", "Toys and Games". "Sports and Outdoors", and three categories for sequential recommendation tasks, including "Musical Instruments", "Arts Crafts and Sewing", "Video Games".
Each item in the dataset is associated with a title, description, and an image. Following previous work (Rajput et al., 2023), we first filter out unpopular users and items with less than five interac- tions. Then, create user behavior sequences based on the chronological order. The maximum item sequence length is uniformly set to 20 to meet all baseline requirements. The statistics of our preprocessed datasets are shown in Table
B BASELINES
We compare the proposed approach with the following baseline methods:
GRU4Rec (Hidasi et al. 2015) introduces Gating Recurrent Unit (GRU) to model user action sequences for session-basee based recommendations.
SASRee (Kang & McAuley 2018b) uses directional self-attentive model to capture item cor-
relations within a sequence.
BERT4Rec (Sun et 2019) employs a bi-directional self-attentive model with the cloze ob- jective for modeling user behavior sequences.
FDSA ZZang et al.
2019 uses a self-attentive model to capture item and feature transition
patterns.
ReeRec Zhou al.
2020) pre-trains sequential models with mutual information maximization to learn the correlations among attributes, items, subsequences, and sequences.
VQ-Rec (Hou et al. 2023 learns vector-quantized item representations for transferable sequen- tial recommenders.
MISSRec (Wang et al. 2023a is multi-modal pre-training and transfer learning framework
for sequential recommendation.
P5-CID (Geng et al. 2022 Hua et al. 2023) organizes multiple recommendation tasks in a text- to-text format and models different tasks uniformly using the T5 model. Here, employ P5 with collaborative indexing as the baseline.
VIPS (Geng et al.) 2023a) is a multimodal foundation model considering visual, textual, and personalization modalities under the P5 recommendation paradigm, to unify various modalities and recommendation tasks.
TIGER (Rajput et al., 2023) adopts the generative retrieval paradigm for sequential recommen- dation and introduces semantic IDD uniquely identify items.
16

---

Published as a conference paper at ICLR 2025
Table 7: Detailed ablation study of various quantitative language generation tasks without pre- training
Modal
Tasks
Instruments
Arts
Games
HR@ 10
NDCG@10 10
HR@ 10
NDCG@ 10
HR@ 10
NDCG@ 10
NIG:
0.1277
0.0987
0.1163
0.0844
0,0885
0,0473
NIG, + QLA
0.1275
0.0986
0.1166
0.0871
0.0465
NIG
0.1275
0.0986
0.1205
0.0877
0.0928
0,0493
NIG + QLA
0.1263
0.0983
0,1204
0.0867
0,0919
0.0492
NIG +
0.1279
0.0987
0.1249
0,1002
0.0529
NIG AIG + QLA
0,1282
0.0993
0.1293
0.0913
0,1010
0,0531
NIG₂
0.1243
0.0968
0,1117
0.0812
0.0881
0.0478
NIG, QLA
0.1237
0.0978
0.1143
0.0826
0.0877
0.0468
Image
NIG
0,1262
0.0986
0.1158
0,0848
0.0899
0.0487
NIG + QLA
0.1265
0.0988
0.1164
0.0849
0.0945
0.0505
NIG
0.1299
0.0998
0,0534
NIG + AIG + QLA
0.1280
0.1001
0.1259
0.0901
0.1017
0.0540
C IMPLEMENTATION DETAILS.
To obtain textual representations, we employ LLaMA to encode the title and description of the item its embedding and use mean pooling to aggregate multiple representations. To obtain visual representations, we utilize CLIP's (Rafford et al., 2021) image branch as encoder to encode the images of items, and we employ ViT-L/14 as the backbone. Both the encoder and decoder of RQ- VAE are implemented as Multi-Layer Perceptrons (MLPs) with ReLU activation functions. The level of codebooks is set to 4, with each level consisting of 256 codebook vectors, and each vector has dimension of 32. The model is optimized using the AdamW optimizer, employing learning rate 0.001 and a batch size of 1024.
the T5 (Raffel al. 2020b) framework to implement our transformer based encoder-decoder architecture. We use layers each for the transformer-basee encoder and decoder models with 6 self-attention heads of dimension 64 in each layer. The MLP and the input dimension was set as 1024 and 128, respectively. The number of prompt tokens for every task is set 4. We employ the AdamW (Loshchilov & Hutter] 2019) optimizer for model optimization, setting the weight decay to 0.01. During pre-training, we utilize a batch size of 4096 with learning rate set 0.001. For alignment tuning, we employ a batch size of 512 with a maximum learning rate of 5c-4, and utilize a cosine scheduler with warm-up to adjust the learning rate.
Following previous work (Rajput et al. 2023).
Our experiments utilize the Tesla V100 GPU. For pretraining, we use four cards, and for fine fine-tuning, we use two cards. Since the model has around 13 million parameters, there is still a substantial amount of GPU memory remaining.
D MORE ABLATION STUDIES
D.1 QUANTITATIVE LANGUAGE GENERATION TASKS.
We have supplemented Table with more detailed ablation experiments of quantitative language generation tasks without pre-training. We study the impact of each task on recommendation perfor- mance through combination of different tasks. We find that: 1) the AIG task always results in significant performance improvement; 2) the QLA task needs to be paired with the AIG task order to achieve better results. This suggests that quantitative language serves as a bridge for knowledge transfer in recommendations, but we need to design appropriate quantitative language tasks.
D.2 PRE- PRE-TRAINING.
We further provide the results using NIG to evaluate the performance of recommendations in Ta- 8 From the results, can be seen that: 1) On the Instruments and Arts datasets, both pre-training and Quantitative Language Generation (QLG) tasks are useful for improving recommendation per- formance. This indicates that quantitative language migrate recommendation knowledge from
17

---

Published as a conference paper at ICLR 2025
Table 8: Detailed ablation study of pre-training and quantitative language generation tasks.
Instruments
Arts
Games
Modal
Methods
HR@ 10
NDCG@
HR@10
NDCG@
HR@10
NDCG@
(0) NIG,
0,1277
0.0987
0,1163
0.0844
0.0885
0.0473
(1) QLG
0.1282
0.0993
0.1293
0.0913
0.1010
0.0531
Text
(2) NIG, w/ pre-training
0.1334
0.1043
0,1305
0.0959
0.0950
0.0508
QLG w/ pre-training
0.1362
0.1051
0.1314
0.0995
0.0521
(4) NIG,
0,1243
0.0968
0.1117
0.0812
0.0881
0.0478
Image
(5) QLG
0,1280
0.1001
0,1259
0.0901
0.1017
0.0540
QLG pre-training
0.1322
0.1029
0.1265
0.0914
0.0987
0.0526
(7) MQI 4Gieec
0.1375
0.1060
0,1327
0.0950
0.1033
0.0548
the source domain and other modalities to the target task. 2) On the Games dataset, QLG with
pre-training impairs performance, which might be due to conflict of recommendation knowl- edge between the source domain and another modality. In the future, we will further explore this phenomenon.
E DISCUSSION
E. HANDLING COLLISIONS.
To address the issue of item collisions, some methods (Rajput et al. 22033 Hua et al. (202)) append an additional identifier to the item indices, which may introduce semantically unrelated distributions. LC-Rec (Zheng et al. 2023) introduces uniform distribution constraint to prevent multiple items from clustering in the same leaf node. Although the LC-Rec achieves better performance in handling collisions compared to previous approaches, has inherent problem: it cannot completely resolve item collisions when item modal content is identical or when the number of collisions exceeds the size of the last level's codebook. This leads another issue: multiple items sharing the same indices results in an unfair comparison of performance.
In contrast, method of dealing with collisions is more rational and essentially solve the aforementioned problems. From the experimental results, our approach achieved similar outcomes LC-Rec, hence we did a dedicated table to list the experimental results.
E.2 LIMITATIONS.
Although method achieves state-of-the-arr performance, there are still some inherent limitations. For example: I) The inference time longer compared to traditional recommendation methods. This is inherent flaw of generative recommendation systems, such methods typically employ beam search and nuto-regressive techniques to generate the next token. 2) Our method requires item content information, and the scenario where item content is missing has yet been studied in the paper. This is an issue that we need to further analyze in our next steps.
18


Reviews:
[{"review_id": "qtdOx6y6SY", "reviewer": "Reviewer_AB8j", "summary": "The paper titled \"Multimodal Quantitative Language for Generative Recommendation\" introduces a novel approach to enhance generative recommendation systems by converting item content from multiple domains and modalities into a unified \"quantitative language\". This methodology seeks to bridge the gap between the generalized linguistic knowledge of pre-trained language models (PLMs) and the specialized needs of recommendation systems. The authors developed a new framework, MQL4GRec, which employs \"quantitative translators\" to convert textual and visual item data into a shared vocabulary. This shared language is then enriched with semantic information through various generation tasks to enable effective knowledge transfer from multimodal data to recommendation systems.\n\nThe paper's main contribution lies in its innovative method of integrating multimodal data to improve recommendation performance significantly, surpassing baseline methods by notable margins in terms of the NDCG metric across multiple datasets. Furthermore, the framework introduces a potential shift towards more universal recommendation systems that do not rely on traditional item IDs, thereby addressing common challenges like improving scalability and transferability across different domains.", "questions": "1. How do different training operations, such as collision handling and quantization, impact the computational time and resource requirements of the model?\n\n2. What methods or algorithms are employed to map these quantitative vectors back to specific items in the dataset?\n\n3. Why does the model not use multimodal information simultaneously to predict the next click item? What are the challenges or limitations that prevent this integration?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The approach to transform diverse item content from different domains and modalities into a unified quantitative language is highly innovative. This integration allows for a more robust and versatile recommendation system that can handle varied inputs effectively.\n\n2. The paper introduces a well-structured design of pre-training and fine-tuning tasks that includes not only generative prediction but also alignment tasks, enhancing the robustness of the method. This comprehensive approach allows the model to effectively leverage both generative capabilities and alignment strategies to improve overall recommendation accuracy.\n\n3. The proposed framework significantly outperforms existing models on several benchmark datasets, particularly in terms of the NDCG metric. This suggests that the method is not only theoretically sound but also practically effective.", "weaknesses": "1. The paper does not discuss the impact of various training operations on the algorithm's time complexity, such as the handling of collisions. This oversight might leave questions about the scalability and efficiency of the proposed method in practical applications.\n\n2. The methodology section lacks a clear explanation of how the generated quantitative vectors are used to retrieve corresponding items during the next item generation task. This omission could lead to ambiguity regarding the operational specifics of the model.\n\n3. The model does not utilize multimodal information simultaneously to predict the next click item, which limits the paper's innovativeness and potential for expansion. Leveraging such multimodal data more effectively could enhance the model's predictive performance and applicability in more complex scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z6n3uSP6MY", "reviewer": "Reviewer_vNzn", "summary": "MQL4GRec introduces a novel approach for generative recommendation by transforming multimodal content from different domains into a unified \"quantitative language,\" facilitating cross-domain knowledge transfer in recommendation tasks. The method uses quantitative translators for text and image content, building a shared vocabulary to encode semantic information across modalities. A series of language generation tasks further enriches this vocabulary, enhancing the model's capacity to represent multi-faceted user preferences. Experimental results demonstrate notable performance improvements over baseline models on key metrics across multiple datasets, showcasing MQL4GRec's scalability and potential in multimodal recommendation.\n\n\n\nHowever, its innovation appears limited, closely resembling existing methods like GenRet and MMGRec in both tokenization approach and generative structure.", "questions": "- User ID Collision: With only 2000 tokens representing a large user base, there is a potential for ID collisions, which may lead to inaccuracies in recommendation results in real-world applications.\n   - Domain Adaptability: The model performs poorly on certain domain-specific datasets, such as the Games dataset, suggesting limitations in domain transferability.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Innovation: The proposed MQL4GRec method translates content from different modalities into a unified “quantitative language,” enabling cross-domain and cross-modal recommendation knowledge transfer. This approach addresses limitations in handling multimodal data in existing generative recommendation models.\n\n\n2. Superior Performance: Experimental results on multiple public datasets demonstrate that MQL4GRec outperforms baseline methods on key metrics such as NDCG.\n\n\n\n3. Open-Source Availability: The paper provides a fully accessible code repository, facilitating reproducibility and further research within the community.", "weaknesses": "1. Similarity to Existing Tokenization Approaches\n- The unified vocabulary for multimodal information closely resembles the generative retrieval tokenization approach found in \"Learning to Tokenize for Generative Retrieval,\" [1]  particularly the \"GenRet\" model, which uses discrete auto-encoding for compact identifiers. This \"multimodal codebook\" seems to be an adaptation of single-modality tokenization, relying on established techniques like RQ-VAE and offering only incremental improvements without substantial architectural or performance innovation.\n- The motivational structure and initial figures in MQL4GRec are closely aligned with those in GenRet, which may diminish the perceived originality of the proposed approach.\n\n\n\n2. Lack of Comparative Analysis\n-  MQL4GRec’s generative approach appears heavily inspired by MMGRec [2], which also employs Graph RQ-VAE for multimodal representation through user-item interactions, raising concerns about the uniqueness of MQL4GRec’s contributions. \n- The paper does not clearly distinguish MQL4GRec's advancements over MMGRec, especially in terms of multimodal token-based representations. A more thorough comparison is needed to establish any unique contributions beyond MMGRec’s existing framework.\n\n\n[1] Sun, W., Yan, L., Chen, Z., Wang, S., Zhu, H., Ren, P., ... & Ren, Z. (2024). Learning to tokenize for generative retrieval. Advances in Neural Information Processing Systems, 36.\n\n[2 ] Liu, H., Wei, Y., Song, X., Guan, W., Li, Y. F., & Nie, L. (2024). MMGRec: Multimodal Generative Recommendation with Transformer Model. arXiv preprint arXiv:2404.16555.\n\nThe paper does not clearly distinguish MQL4GRec's advancements over MMGRec, especially in terms of multimodal token-based representations. A more thorough comparison is needed to establish any unique contributions beyond MMGRec’s existing framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TUrh6Kc6kz", "reviewer": "Reviewer_RLim", "summary": "This paper presents a novel multimodal generative recommendation method, MQL4GRec, which facilitates the effective transfer of recommendation knowledge by converting item content from different domains and modalities into a unified quantitative language. Specifically, MQL4GRec introduces a unified quantitative language representation to handle multimodal content, including text and images. Additionally, a series of quantitative language generation tasks are designed to enrich the semantic representation. Extensive experiments across three datasets show that this method significantly outperforms baseline approaches.", "questions": "* Figure 3 shows that the performance on the Games dataset slightly declines as the amount of pre-training data increases. Does this suggest that the pre-training strategy has limitations when applied to domains with substantial cross-domain differences?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* High Innovation: This work is the first to propose using a unified quantitative language to address knowledge transfer in multimodal recommendation, which is highly valuable for enhancing the generalization ability of recommendation systems.\n\n* Thorough Experimental Validation: The paper conducts extensive experiments across three datasets, showcasing not only the overall performance of the method but also analyzing the role of individual components through ablation studies, indicating rigorous experimental design.", "weaknesses": "* The model's high complexity poses significant computational and storage demands, which could lead to considerable costs in real-world deployment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oIoElpU7ea", "reviewer": "Reviewer_rVKM", "summary": "This paper introduces a novel approach, MQL4GRec, designed to convert item content from diverse domains and modalities into a unified quantitative language.", "questions": "1.As an efficient framework, I am interested in understanding the training costs of MQL4GRec compared to other baselines. Specifically, how does it perform in terms of training time, VRAM usage, and inference time?\n\n2.In the field of recommender systems, there is a lack of widely recognized pre-trained models. Could MQL4GRec potentially serve as a foundation model for other downstream tasks?\n\nMy concerns are addressed by the authors, therefore, I will raise my scores from 6 to 8.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 4, "strengths": "1. The proposed concept of a multimodal quantitative language, together with the design of quantitative language generation tasks, represents a novel and innovative advancement in the field of generative recommendation\n\n2.  The architecture design is elegant, presenting the idea in a straightforward way, yet experiments demonstrate its strong effectiveness. I personally appreciate this type of work and believe it can make a meaningful impact in the field of generative recommendation.\n\n3. The availability of the code significantly enhances reproducibility.", "weaknesses": "1.The proposed framework is rooted in the generative recommendation paradigm and aligns with a preprint in a similar research direction [1]. However, it still represents a valuable contribution in my view, even if not groundbreaking.\n\n2.The authors should consider testing the statistical significance of MQL4GRec results.\n\n3.There remains a limitation in zero-shot capability, which is a known challenge in the field of recommendation.\n\n4.To enhance the comprehensiveness of the \"Multi-modal Recommendation\" section in the related work, the authors could consider including more recent state-of-the-art multimodal recommender system papers, such as [2,3]. The field of multimodal codebooks from other communities should also be included in the related work section to clarify the distinctions between the proposed MQL approach and existing methods, such as those in [4, 5]\n\n\n[1] Liu, Han, et al. \"MMGRec: Multimodal Generative Recommendation with Transformer Model.\" arXiv preprint arXiv:2404.16555 (2024).\n\n[2] Fu, Junchen, et al. \"IISAN: Efficiently adapting multimodal representation for sequential recommendation with decoupled PEFT.\" *Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval*. 2024.\n\n[3] Liu, Han, et al. \"MMGRec: Multimodal Generative Recommendation with Transformer Model.\" *arXiv preprint arXiv:2404.16555* (2024).\n\n[4] Lan, Zhibin, et al. \"Exploring better text image translation with multimodal codebook.\" arXiv preprint arXiv:2305.17415 (2023).\n\n[5] Duan, Jiali, et al. \"Multi-modal alignment using representation codebook.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)