Recommender Systems with Generative Retrieval
Authors: Shashank Rajput
Nikhil Mehta
Anima Singh
Raghunandan Hulikal Keshavan
Trung Vu
Lukasz Heldt
Lichan Hong
Yi Tay
Vinh Q. Tran
Jonah Samost
Maciej Kula
Ed H. Chi
Maheswaran Sathiamoorthy
Venue: NeurIPS 2023 poster (2023)
Keywords: Recommender Systems
Generative Retrieval
Vector Quantization
TL;DR: We propose a Generative Retrieval framework for Recommender Systems based on Semantic ID representation of items. We achieve a new SOTA recommendation performance on multiple datasets with the proposed framework.

Abstract: Modern recommender systems perform large-scale retrieval by embedding queries and item candidates in the same unified space, followed by approximate nearest neighbor search to select top candidates given a query embedding. In this paper, we propose a novel generative retrieval approach, where the retrieval model autoregressively decodes the identifiers of the target candidates. To that end, we create semantically meaningful tuple of codewords to serve as a Semantic ID for each item. Given Semantic IDs for items in a user session, a Transformer-based sequence-to-sequence model is trained to predict the Semantic ID of the next item that the user will interact with. We show that recommender systems trained with the proposed paradigm significantly outperform the current SOTA models on various datasets. In addition, we show that incorporating Semantic IDs into the sequence-to-sequence model enhances its ability to generalize, as evidenced by the improved retrieval performance observed for items with no prior interaction history.

Recommender Systems with Generative Retrieval
Shashank Rajput* University of Visconsin-Madison
Nikhil Mehta" Google DeepMind
Anima Singh Google DeepMind
Raghunandan Keshavan
Trung Google
Lukasz Heldt
Lichan Hong Google DeepMind
Google
Google
Yi Tay Google DeepMind
Vinh Q. Tran Google
Jonah Samost
Maciej Kula Google DeepMind
Google
Ed H. Chi Google DeepMind
Maheswaran Sathiamoorthy Google DeepMind
Abstract
Modern recommender systems perform large-scale retrieval by embedding queries and item candidates in the same unified space, followed by approximate nearest neighbor search to select top candidates given a query embedding. this paper, we propose novel generative retrieval approach, where the retrieval model au- toregressively decodes the identifiers the target candidates. To that end, we create semantically meaningful tuple of codewords to serve Semantic ID for each item. Given Semantic IDs for items in a user session, Transformer-based sequence-to seeuuuec-t--seueeneee model is trained to predict the Semantic ID of the next item that the user will interact with. We show that recommender systems trained with the proposed paradigm significantly outperform the current SOTA models various datasets. In addition, we show that incorporating Semantic IDs into the sequence-to-see so-euuence model enhances its ability generalize, evidenced by the improved retrieval performance observed for items with no prior interaction history.
1 Introduction
Recommender systems help users discover content of interest and are ubiquitous various recommendation domains such videos 4 43 9,, apps R products [6. g)
User History
Next
Brandy Abomic hom
Domiiii (Atomii itom 515
Semantic
and music 118 19] Modern recommender strategy,
Lookup
systems adopt a retrieve-and-runs where set of viable candidates are lected in the retrieval stage, which are then ranked using ranker model. Since the ranker model works only on the candidates it receives, it is desired that the retrieval stage emits highly relevant candidates.
(5, 23,23.85 86)
15.25.22
(5.25.
Figure 1: Overview of the Transformer Index for GEnerative Recommenders (TIGER) framework. With TIGER, sequential recommendation is expressed generative retrieval task by representing each item as tuple of discrete semantic tokens.
Equal contribution. Work done when SR was at Google. Comespondence to sajput.shashank1 11 @ Degmail.com nikhilmehta wowggee....ooooomoommmoooo nlogn googleccoo.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

There are standard and well-established methods for building retrieval models. Matrix factoriza- tion [19] learns query and candidate embeddings in the same space. In order better capture the non-linearities in the data, dual-encoder architectures 39 (i.e., one tower for the query and another for the candidate) employing inner-product to embed the query and candidate embeddings in the same space have become popular in recent years. To use these models during inference, an index that stores the embeddings for all items is created using the candidate tower. For a given query, its embedding is obtained using the query tower, and an Approximate Nearest Neighbors (ANN) algorithm is used for retrieval. In recent years, the dual encoders architectures have also been extended for sequential recommendations interactions.
24
32
44
that explicitly take into account the order of user-item
We propose a new paradigm of building generative retrieval models for sequential recommendation. Instead of traditional query-candidate matching approaches, our method uses an end-to-end generative model that predicts the candidate IDss directly. We propose to leverage the Transformer [6 memory (parameters) as an end-to-end index for retrieval in recommendation systems, reminiscent of Tay et al.
34 that used Transformer memory for document retrieval. We refer to our method as Transformer Index for GEnerative Recommenders (TIGER). A high-level overview of TIGER is shown in Figure TIGER is uniquely characterized by novel semantic representation of items called "Semantic ID" a sequence of tokens derived from each item's 's content information. Concretely, given an item's s text features,
pre- tre-trained text encoder (e.g.. SentenceTS (27] to generate dense content embeddings. A quantization scheme is then applied on the embedding of an item to form a set of ordered tokens/codewords, which we refer to the Semantic ID of the item. Ultimately, these Semantic IDs are used to train the Transformer model on the sequential recommendation task.
Representing items as a sequence of semantic tokens has many advantages. Training the transformer memory on semantically meaningful data allows knowledge sharing across similar items. This allows us to dispense away with the atomic and random item Ids that have been previously used [33 as item features in recommendation models. With semantic token representations for items, the model is less prone to the inherent feedback loop I 26 39] in recommendation systems, allowing the model to generalize to newly added items to the corpus. Furthermore, using a sequence of tokens for item representation helps alleviate the challenges associated with the scale of the item corpus; the number of items that can be represented using tokens is the product of the cardinality of cach token in the sequence. Typically, the item corpus size can be in the order of billions and learning unique embedding for each item can be memory-intension be adopted reduce the item representation space, in this work, show that using semantically meaningful ful tokens for item representation is an appealing alternative. The main contributions of this work are summarized below:
While random bessing--bseed techniques [16]
1. We propose TIGER, a novel generative retrieval-based recommendation framework that assigns Semantic IDs to each item, and trains retrieval model to predict the Semantic ID of an item that a given user may engage with.
2. We show that TIGER outperforms existing SOTA recommender systems on multiple datasets as measured by recall and NDCG metrics.
3. We find that this new paradigm of generative retrieval leads to two additional capabilities in sequential recommender systems: 1. Ability to recommend new and infrequent items, thus improving cold-start recommendations, and 2. Ability to generate diverse recommendations using a tunable parameter.
In Section 2 we provide a brief literature survey of recommender systems, generative retrieval, and the Semantic ID generation techniques we use in this paper. In Section 3 we explain our proposed framework, and outline the various techniques we use for Semantic generation. We present the result of our experiments in Section 4 and conclude the paper Section
Paper Overview.
2
Related Work
Sequential Recommenders. Using deep sequential models in recommender systems has devel- oped into rich literature. GRU4REC [I]] was the first to use GRU based RNNs for sequential recommendations. Li al. (24 proposed Neural Attentive Session- based Recommendation (NARM), where attention mechanism along with GRU layer is used to track long term intent of the user.

---

bom 64
Bemn 00 26, 55)
t.5
1,56
Contest
Description Categories
Bidirectional Transformer Encoder
Transformer Decoder
LD
T-6
1,26
1,78
Iarm 2013
barn 515
Item Content Information
Content Encodor
Quantssation
25,
Ihem Interaction Hatorry User
(a) Semantic ID generation items using (b) Transformer based encoder-decoder setup building quantization of content embeddings.
saauence-to-seqeeee model used for generative retrieval.
Figure 2: overview the modeling approach used TIGER.
AttRec 41 proposed by Zhang et al. used self-attention mechanism to model the user's intent ensured by modeling user-item affinity with metric
in the current session, and personalization learning. Concurrently, Kang et al. also proposed SASRec [77, which used self-attention similar to decoder-only transformer models. Inspired by the success of masked language modeling in language tasks, BERT4Rec (32 and Tee [6] utilize transformer models with masking strate-
goes beyond just masking by pre-training on four self- supervised tasks improve data representation. The models described above learn nigh-dimensional embedding for each item and perform an ANN in Maximum Inner Product Search (MIPS) space to predict the next item. contrast, our proposed technique, TIGER, uses Generative Retrieval to directly predict the Semantic ID of the next item.
gies for sequential recommendation tasks. 3 -Rc
P5 fine-tunes pre-trained large language models for multi-task recommender systems. The P5 model relies on the LLM tokenizer (SentencePiece tokenizer (29)) to generate tokens from randomly- assigned item IDs. Whereas, we use Semantic ID representation of items thay are learned based on the content information of the items. In our experiments (Table 2). we demonstrate that recommendation systems based on Semantic ID representation of items yield much better results than using random codes.
Semantic IDs. Hou et al. proposed VQ-Rec 12 to generate "codes" (analogous to Semantic IDs) using content information for item representation. However, their focus on building transferable recommender systems, and do not use the codes in generative manner for retrieval. While they also use product guantization [I5] to generate the codes, we use EQE to generate Semantic IDs, which leads to hierarchical representation of items (Section 4.2 In concurrent work Singh et al. 31 show that hierarchical Semantic IDs can be used to replace item IDs for ranking models in large scale recommender systems improves model generalization.
Generative Retrieval. While techniques for learning search indices have been proposed in the past 200, generative retrieval is a recently developed approach for document retrieval, where the task is to return set of relevant documents from a database. Some examples include GENRE [5], DSI [34] NCI 3 and CGR [22]. A more detailed coverage of the related work is in Appendix A To the best of our knowledge, we are the first propose generative retrieval for recommendation systems using
Semantic ID) representation of items.
3 Proposed Framework
Our proposed framework consists of two stages:
1. Semantic ID generation using content features. This involves encoding the item content features embedding vectors and quantizing the embedding into a tuple of semantic codewords. The resulting tuple of codewords is referred to as the item's Semantic ID.
2. Training generative recommender system Semantic IDs. A Transformer model is trained the sequential recommendation task using sequences of Semantic IDs.

---

Residual Quantization
codaback
codebook
codabook
Entraddngg
DNN Encoder
DNN Decoder
representation
Semantic codes
Figure 3: RQ- VAE: In the figure, the vector output by the DNN Encoder, say vo (represented by the blue bar), fed the quantizer, which works iteratively. First, the closest vector ro is found in the first level codebook. Let this closest vector be Ecg (represented by the red bar). Then, the residual error is computed as = Bay- This is fed into the sccond level of the quantizer, and the process is repeated: The closest vector to is found in the second level, say es, (represented by the green bar), and then the second level residual error is computed as r2 = ri- c. Then, the process repeated for third time ra. The semantic codes are computed as the indices Eog Eeg and Eog their respective codebooks. In the example shown the figure, this results in the code (7, 4.
3.1 Semantic ID Generation
In this section, we describe the Semantic ID generation process for the items in the recommendation corpus. We assume that each item has associated content features that capture useful semantic information (e.g. titles or descriptions or images). Moreover, we assume that we have access to pre-trained content encoder to generate a semantic embedding a: E Rd For example, general-purpost pre-trained text encoders such as Sentence-T5 [27] and BERT 7 can be used to convert an item' 's text features to obtain semantic embedding. The semantic embeddings are then quantized to generate Semantic ID for each item. Figure 2a gives high-level overview of the process.
We define Semantic ID to be tuple of codewords of length m. Each codeword in the tuple comes from different codebook. The number of items that the Semantic IDs can represent uniquely is thus equal to the product of the codebook sizes. While different techniques to generate Semantic IDs result in the IDs having different semantic properties, we want them to at least have the following property: Similar items (items with similar content features whose semantic embeddlings are close) should have overlapping Semantic IDs. For example, an item with Semantic (10, 21,35) should be more similar to one with Semantic ID (10, 21, 40). than item with ID (10, 23, 32). Next, discuss the quantization schemes which we use for Semantic ID generation.
RQ-VAE for Semantic IDs. Residual- Quantizze Variational AutoEncoder (RQ-VAE) 40 is multi-level vector quantizer that applies quantization on residuals generate a tuple of codewords (aka Semantic IDs). The Autoencoder is jointly trained by updating the quantization codebook and the DNN encoder-decoder parameters. Fig. illlssratee the process of generating Semantic IDs through residual quantization.
RQ-VAE first encodes the input via encoder & learn latent representation z E(xr). At the zero-th level (d = 0), the initial residual is simply defined as ro == z. At each level d, we have a codebook C2 := {ex {= where K is the codebook size. Then, ro is quantized by mapping nearest embedding from that level's codebook. The index of the closest embedding ec, at d = 0, i.c. c arg min, ||ro 1) represents the zero-th codeword. For the next level d the residual is defined as == To Cc. Then, similar to the zero-th level, the code for the first level is computed by finding the embedding in the codebook for the first level which is nearest to T1. This process is repeated recursively m times to get tuple of m codewords that represent the Semantic ID. This recursive approach approximates the input from a coarse-to-fine granularity. Note that we chose to use separate codebook of size K for each of the m levels, instead of using single, mk mK-sizee -ized codebook. This was done because the norm of residuals tends to decrease with increasing levels, hence allowing for different granularities for different levels.
k=
the

---

Once we have the Semantic ID (co..... Cm-1), a quantized representation of z is computed as 2 = Ed=o Ec.- Then Σ is passed to the decoder, which tries to recreate the input æ using 2. The RQ-VAE loss is defined as [(x) := Lreeon + Lupaes where Lreson := | a|I?, and Bl|r: - sg(e,,]ll.. Here & is the output of the decoder, and sg is the stop-gradient operation 35 This loss jointly trains the encoder, decoder, and the codebook.
Layvee Eg=o [|s//r]]
Ec. I| Ec.]
As proposed in [4)], prevent RQ- VAE from codebook collapse, where most of the input gets mapped to only a few codebook vectors, we use k-means clustering- based initialization for the codebook. Specifically, we apply the k-means algorithm on the first training batch and use the centroids as initialization.
Other alternatives for quantization. A simple alternative to generating Semantic IDs is to use Locality Sensitive Hashing (LSH). We perform an ablation study in Subsection 4.2 where we find that RQ-VAE indeed works better than LSH. Another option is to use k-means clustering hicrarchically [4]. but it loses semantic meaning between different clusters (37. We also tried VQ-VAE, and while performs similarly to VAE for generating the candidates during retrieval, it loses the hierarchical nature of the IDs which confers many new capabilities that are discussed in Section 4.3
Handling Collisions. Depending on the distribution of semantic embeddings, the choice of codebook size, and the length of codewords, semantic collisions can occur (i.e., multiple items can map to the same Semantic ID). To remove the collisions, we append an extra token at the end of the ordered semantic codes to make them unique. For example, two items share the Semantic ID (12, 24, 52), append additional tokens to differentiate them, representing the two items as (12, 24, 52, 0) and (12, 24, 52, 1). To detect collisions, we maintain a lookup table that maps Semantic IDs to corresponding items. Note that collision detection and fixing is done only once after the RQ-VAE model is trained. Furthermore, since Semantic IDs are integer tuples, the lookup table is efficient in terms of storage in comparison high dimensional embeddings.
3.2 Generative Retrieval with Semantic IDs
We construct item sequences for every user by sorting chronologically the items they have interacted with. Then, given a sequence of the form (itemı.... item,). the recommender system's task is to pre- dict the next item 1tem,+11- We propose a generative approach that directly predicts the Semantic ID of the next item. Formally, let (c.u...., -.mmm1)) be the m-length Semantic ID for item,. Then, we con- vert the item sequence to the sequence (C1.0:.... C1,m-1, C2.0...., -2.m-1.- The srquence-to-sequenee model then trained to predict the Semantic ID of item,+1. which is (Cn+1.0. ... Ca+1,m-1)) 1). Given the generative nature of our framework, it is possible that generated Semantic ID from the decoder does not match an item in the recommendation corpus. However, as we show in appendix (Fig. () the probability of such an event occurring is low. We further discuss how such events can be handled in appendix
Cn.D. mn.m-1).
4 Experiments
Datasets. We evaluate the proposed framework on three public real-world benchmarks from the Amazon Product Reviews dataset [10], containing user reviews and item metadata from May 1996 to July 2014. In particular, we use three categories of the Amazon Product Reviews dataset for the sequential recommendation task: "Beauty", "Sports and Outdoors", and "Toys and Games". We discuss the dataset statistics and pre-processing in Appendix
Evaluation Metrics. We use top-k Recall (Recall K) and Normalized Discounted Cumulative Gain (NDCG@K) with K 5, 10 to evaluate the recommendation performance.
RQ-VAE Implementation Details. As discussed in section 3.1 RQ-VAE is used to quantize the semantic embedding of item. We use the pre-trained Sentence-T5 27 model to obtain the semantic embedding of each item in the dataset. In particular, we use item's 's content features such as title, price, brand, and category to construct sentence, which is then passed to the pre-trained Sentence-TS model to obtain the item's semantic embedding of 768 dimension.
The VAE model consists of three components: DNN encoder that encodes the input semantic embedding into a latent representation, residual quantizer which outputs a quantized representation, and DNN decoder that decodes the quantized representation back to the semantic input embedding
5

---

HoT Product
Dath
Dateggoy
Mobeus go femovar
Pun
5cn
Frgganee Som
5cm
Orgruntt
Probubility
Ce
Probabiitt
6.8
(a) The ground-truth category distribution for all the items in the dataset colored by the value the first codeword e1
(b) The category distributions for items having the Semantic ID (CI, *, *), where eι 31,2,3,4). The categories are color- coded based the second semantic token c2-
Figure 4: Qualitative study of RQ-VA (Q-VAE Semantic IDs (C1, C2, Cs, c4) the Amazon Beauty dataset. We show that the ground truth categories are distributed across different Semantic tokens. Moreover, the RQVAE semantic IDs form hierarchy of items, where the first semantic token (c1) corresponds to coarse-level category, while second/third semantic token (02/c3) correspond to fine-grained categorics.
space. The encoder has three intermediate layers of size 512, 256 and 128 with ReLU activation, with final latent representation dimension of 32. To quantize this representation, three levels of residual quantization is done. For each level, codebook of cardinality 256 is maintained, where B 0.25.
each vector in the codebook has dimension of 32. When computing the total loss, The RQ-VAF model is trained for 20k epochs to ensure high codebook usage (2 80%). We use Adagrad optimizer with learning rate of 0.4 and a batch size of 1024. Upon training, we use the learned encoder and the quantization component to generate 3-tuple Semantic ID for each item. To avoid multiple items being mapped to the same Semantic ID, we add a unique 4th code for items that share the same first three codewords, i.e. two items associated with a tuple (7, 1, 4) are assigned (7, 1, 0) and (7, 1, 4, respectively (if there are no collisions, we still assign 0 as the fourth codeword). This results in a unique Semantic ID of length 4 for each item in the recommendation corpus.
Sequence-to-Sequeene Model Implementation Details. We use the open-sourced T5X frame- work (28) to implement our transformer based encoder-decoder architecture. To allow the model to process the input for the sequential recommendation task, the vocabulary of the sequence-to-ssquencc model contains the tokens for each semantic codeword. In particular, the vocabulary contains 1024 (256 x 4) tokens to represent items in the corpus. In addition to the semantic codewords for items, we add user-specific tokens to the vocabulary. To keep the vocabulary size limited, we only add 2000 tokens for user IDs. We use the Hashing Trick (38 to map the raw user ID to one of the 2000 user ID tokens. We construct the input sequence the user Id token followed by the sequence of Semantic ID tokens corresponding to given user's 's item interaction history. We found that adding user ID to the input, allows the model to personalize the items retrieved.
We use layers each for the transformer-basee encoder and decoder models with self-attention heads of dimension 64 in each layer. We used the ReLUU activation function for all the layers. The MLP and the input dimension was set as 1024 and 128, respectively. We used a dropout of 0. Overall, the model has around 13 million parameters. We train this model for 200k steps for the "Beauty" and "Sports and Outdoors" dataset. Due to the smaller size of the "Toys and Games" dataset, it is trained only for 100k steps. We use batch size of 256. The learning rate 0.01 for the first 10k steps and then follows an inverse square root decay schedule.
4.1 Performance on Sequential Recommendation
In this section, we compare our proposed framework for generative retrieval with the following sequential recommendation methods (which are described briefly Appendix [B: GRU4Rec [T].
Caser (33. HGN (25). SASRec [17], BERT4Rec [32) FDSA 42]. S Reec [4), and P5 (8). Notably

---

Table 1: Performance comparison on sequential recommendation. The last row depicts % improvement with
TIGER relative to the best baseline. Bold (underline) are used to denote the best (second-bess metric.
Sports Outdoors
Beauty
Toys and Games
Methods
Recall 6#5
NDCG
Recall
NDCG
Recall 655
NDCG 69.5
Recall @10
NDCG 10
Recall 68.5
NDCG
Recall
NDCG
@
@010
P5
0,0061
0.0041
0,0055
0.0052
0.0163
0.0107
0.0254
0.0136
0.0070
0.0050
0.0121
0.0066
Caser
33
0.0116
0.0072
0.0194
0.0097
0.0205
0.0131
0.0347
0.0176
0.0166
0.0107
0.0270
0.0141
HGN
0.0189
0.0120
0.0313
0.0159
0.0325
0.0206
0.0512
0.0266
0.0321
0,0221
0.0497
0.0277
GRU4Rcc
0.0129
0.0086
0.0204
0,0110
0.0164
0.0099
0.0283
0.0137
0.0097
0.0059
0,0176
0.0084
BERT4Rec
32
0.0115
0.0075
0.0191
0.0099
0.0203
0.0124
0.01447
0.0170
0.0116
0.0071
0.0203
0.0099
FDSA (2
0.0182
0.0122
0.0288
0,0156
0.0267
0.0163
0.0407
0.0208
0.0228
0.0140
0,0381
0.0189
SASRec
17
0,0233
0,0154
0.0350
0.0192
0.0387
0.0249
0,0605
0,0318
0,01463
0.0306
0.0675
0.0374
S--Ree
[44
0.0251
0.0161
0.0385
0.0204
0.0387
0.0244
0.0647
0.0327
0.0443
0.0294
0.0700
0.0376
TIGER [Ours]
0.0264
0.0181
0.0400
0.0225
0.0454
0.0321
0.0648
0,0384
0.0521
0.0371
0.0712
+5.22%
+12.55%
+3.90%
+10.29%
+17.31%
+29.04%
+0.15%
+17,43%
+12.53%
+21.24%
+1.71%
+14.97%
all the baselines (except P5), learn a high-dimensional vector space using dual encoder, where the user's past item interactions and the candidate items are encoded as high-dimensional representation and Maximum Inner Product Search (MIPS) is used to retrieve the next candidate item that the user will potentially interact with. In contrast, our novel generative retrieval framework directly predicts the item's Semantic ID token-by-token using sequence-to-sequenee model.
Recommendation Performance. We perform an extensive analysis of our proposed TIGER on the sequential recommendation task and compare against the baselines above. The results for all baselines, except P5, are taken from the publicly accessible results made available by Zhou et al. [4]. For P5, we use the source code made available by the authors. However, for a fair comparison, we updated the data pre-processing method to be consistent with the other baselines and our method. We provide further details related to our changes in Appendix D
The results are shown in Table [ We observe that TIGER consistently outperforms the existing baselines"] F particular, TIGER performs considerably better on the Beauty benchmark compared to the second-best baseline with up to 29% improvement in NDCG 5 compared to SASRec and 17.3% improvement in Recall @5 compared Ree. Similarly on the Toys and Games dataset, TIGER is 21 % and 15% better in NDCG@5 5 and NDCG@ 10, respectively.
We see significant improvement across all the three benchmarks that we considered. In
4.2 Item Representation
In this section, we analyze several important characteristics of RQ- VAE Semantic IDs. In particular, we first perform a qualitative analysis to observe the hierarchical nature of Semantic IDs. Next, evaluate the importance of our design choice of using RQ-VAE for quantization by contrasting the performance with alternative hashing-basse quantization method. Finally, we perform an ablation to study the importance of using Semantic IDs by comparing TIGER with a sequence-to-sequence model that uses Random ID for item representation.
Qualitative Analysis. We analyze the RQ-VAE Semantic IDs learned for the Amazon Beauty dataset in Figure 4 For exposition, we set the number of RQ- levels as 3 with codebook size of 4, 16, and 256 respectively, i.e. for a given Semantic ID (c1, C2, 03) of an item, 0- S C1 < 3, 0 I 02 I 15 and 0 S C3 S 255. In Figure 4a we annotate each item's 's category using C₁ to visualize c₁ -specciii categories in the overall category distribution of the dataset. As shown in Figure 4a ei captures the high-level category of the item. For instance, C1 = 3 contains most of the products related to "Hair". Similarly, majority of items with e = 1 are "Makeup" and "Skin" products for face, lips and eyes.
We also visualize the hierarchical nature of RQ-VAE Semantic IDs by fixing Ci and visualizing the category distribution for all possible values of 02 Fig.44b We again found that the second codeword 02 further categorizes the high-level semantics captured with e into fine-grained categories. The hierarchical nature of Semantic IDs learned by RQ- opens a wide-array of new capabilities which are discussed in Section 4.3 As opposed existing recommendation systems that learn item embeddings based on random atomic IDs, TIGER uses Semantic IDs where semantically similar
Phurs://github.c http::/lgihuh..cm/aaHHWaaaaa www.///aiiiaaa////////22
We show in Table 9 that the standard error in the metrics for TIGER is insignificant.
7

---

Table 2: Ablation study for different ID generation techniques for generative retrieval. We show that RQ-VAE Semantic (SID) perform significantly better compared to hashing SIDs and Random IDs.
Sports Outdoors
Beauty
Toys Games
Methods
Recall 65
NDCG
Recall
NDCG
Recall
NDCG
Recall 6:0
NDCG
Recall
NDCG
Recall @10
NDCG
(85
@0
@10
@55
85
@10
(85
@00
Random ID
0.007
0.005
0.0116
0.0063
0.0296
0.0205
0,0434
0.0250
0.0362
0.0270
0.0448
0.0298
L.SH SID
0.0215
0.0146
0.0321
0.0180
0.0379
0.0333
0.0309
0.0412
0.0299
0.0566
0.0349
RQ-VAL VAE SID
0,0264
0.0181
0,0400
0.0225
0.0454
0.0321
0.0648
0.0384
0.0521
0.0371
0.0712
0.0432
0.34
0.10
0.12
0.08
Rocell@10
0.30
Overall
Utseen
0.08
0.06
0.04
Oomal Unseon
Spit
0.02
0.00
10
0.4
0.5
K
Epsilon (e)
(a) Recall@K vs. K. (E 0.1).
(b) Recall @ 10 vs.
Figure 5: Performance in the cold-start retrieval setting.
items have overlapping codewords, which allows the model to effectively share knowledge from semantically similar items in the dataset.
Hashing RQ-VAE Semantic IDs. We study the importance of RQ-VAE in our framework by comparing RQ- VAE against Locality Sensitive Hashing (LSH) [4 2 for Semantic ID generation. LSH is popular hashing technique that can be easily adapted to work for our setting. To generate LSH Semantic IDs, we use h random hyperplanes wi...., Wh, to perform a random projection of the embedding vector and compute the following binary vector: (lwi) æ>-0>-..>
".]aa>> a2-0). This vector converted into an integer code as co Σ 2' -- m>0 This process is repeated m times using an independent set of random hyperplanes, resulting in m codewords (co, CI,..., Cm-1), which we refer to as the LSH Semantic ID.
In Table [ we compare the performance of LSH Semantic ID with our proposed RQ- VAE Semantic ID. In this experiment, for LSH Semantic IDs, we used h = 8 random hyperplanes and set m = 4 to ensure comparable cardinality with the RQ-VAE. The parameters for the hyperplanes are randomly sampled from standard normal distribution, which ensures that the hyperplanes are spherically symmetric. Our results show that RQ-VAE consistently outperforms L.SH. This illustrates that learning Semantic IDs via non-linear, Deep Neural Network (DNN) architecture yields better quantization than using random projections, given the same content-based semantic embedding.
Random ID vs. Semantic ID. We also compare the importance of Semantic IDs in our generative retrieval recommender system. In particular, we compare randomly generated IDs with the Semantic IDs. To generate the Random ID baseline, we assign m random codewords to each item. A Random ID of length m for an item is simply (cI...., Cm.) where o; is sampled uniformly at random from 21, K}. We set m 4, and K = 255 for the Random ID baseline to make the cardinality similar to RQ- VAE Semantic IDs. comparison of Random IDD against RAEV and LSH Semantic IDs is shown in Table 2 We see that Semantic IDs consistently outperform Random ID baseline, highlighting the importance of leveraging content-base semantic information.
4.3 New Capabilities
We describe two new capabilities that directly follow from our proposed generative retrieval frame- work, namely cold-start recommendations and recommendation diversity. We refer to these capabil- ities as "new" since existing sequential recommendation models (See the baselines in section 4.1 cannot be directly used to satisfy these real-world use cases. These capabilities result from a synergy between RQ- VAE based Semantic IDs and the generative retrieval approach of our framework. We discuss how TIGER is used in these settings in the following sections.
8

---

Cold-Start Recommendation. In this section, we study the cold-start recommendation capability of our proposed framework. Due to the fast-changing nature of the real-world recommendation corpus, new items are constantly introduced. Since newly added items lack user impressions in the training corpus, existing recommendation models that use a random atomic II) for item representation fail to retrieve new items as potential candidates. In contrast, the TIGER framework can easily perform cold-start recommendations since it leverages item semantics when predicting the next item.
For this analysis, we consider the Beauty dataset from Amazon Reviews, To simulate newly added items, we remove 5% of test items from the training data split. We refer to these removed items as unseen items. Removing the items from the training split ensures there is no data leakage with respect to the unseen items. As before, we use Semantic ID of length 4 to represent the items, where the first 3 tokens are generated using RQ- VAE and the 4th token is used to ensure a unique ID exists for all the seen items. We train the RQ-VA VAE quantizer and the suneenceto-sequenc model on the training split. Once trained, we use the RAE model to generate the Semantic IDss for all the items in the dataset, including any unseen items in the item corpus.
Given Semantic ID (cI, C2 Ca, 64) predicted by the model, we retrieve the seen item having the same corresponding ID. Note that by definition, each Semantic ID predicted by the model can match at most one item in the training dataset. Additionally, unseen items having the same first three semantic tokens, i.e. (c1, C22 03) are included to the list of retrieved candidates. Finally, when retrieving a set of top-K candidates, we introduce hyperparameter e which specifies the maximum proportion of unseen items chosen by our framework.
We compare the performance of TIGER with the k-Nearest Neighbors (KNN) approach the cold-start recommendation setting in Fig. S For KNN, we use the semantic representation space to perform the yearest-neighhon search. We refer to the KNN-based baseline as Semantic KNN. Fig. [5a] shows that our framework with c = 0.1 consistently outperforms Semantic_KKa for all Recall K metries. In Fig. [5b] provide comparison between method and Seman- tic_ tic_KNN for various values of e. For all settings of e 2 0.1, our method outperforms the baseline.
Recommendation diversity. While Recall and NDCG are the primary metrics used to evaluate a recommendation system, diversity of predictions is another critical objective of interest. A recommender system with poor diversity can be detrimental to the long-term engagement of users. Here, we discuss how our generative retrieval framework can be used
Table 3: The entropy of the category distribution predicted by the model for the Beauty dataset. A higher entropy corresponds more diverse items predicted by the model.
Temperature
Entropy 10
Entropy @20
Entropy
T=
0.76
1.14
1.70
T= T=20
1.14
1.52
2.06
1.38
1.76
2.28
to predict diverse items. We show that temperature-based sampling during the decoding process can be effectively used control the diversity of model predictions. While temperature-based sampling can be applied to any existing recommendation model, TIGER allows sampling across various levels of hierarchy owing the properties of RQ-VAE Semantic IDs. For instance, sampling the first token of the Semantic ID allows retrieving items from coarse-level categories, while sampling a token from second/third token allows sampling items within category.
Table 4: Recommendation diversity with temperature- decoding.
Target Canegory
Most-cammon Categories top-10 predicted
2.0
Styling Products
Styling Products
Styling Products, Hair Styling Tools Face
Tools Nail
Tools
Tools Makeup
Makeup Nails
Makeup Nails
Makeup Nuils, Skin Hands & Nails, Tools Nail
Skin byee
Skin leye
Hair Relaxess, Skin Face. Hair Styling Products, Skin Byes
Makeup Face
Tools Makeup MakkhusMaakuup Face
Tools Makeup Brushes, Makeup Face.Skin Face. Makeup Sees. Hair Styling Tools Skin Face. Hair Loss Products, Hair Stampoos, Hair Scalp Treatments, Hair Conditioness
Hair Loos Produces
Hair Loss Products, Skin Facee Skin Body
We quantitatively measure the diversity of predictions using Entropy @ K metric, where the entropy
calculated for the distribution of the ground-truth categories of the top-K items predicted by the model. We report the Entropy @ K for various temperature values in Table 3 We observe that temperature-sampling in the decoding stage can be effectively used increase the diversity in the ground-truth categories of the items. We also perform qualitative analysis in Table 4

---

Table 5: Recall and NDCG metrics for different number layers.
Number of Layers
Recall @5
NDCG@5
Recall @ 10
NDCG@ 10
0.04499
0.03062
0.06699
0.03768
0.0454
0.0321
0.0648
0.0384
0.04633
0.03206
0.06596
0.03834
4.4 Ablation Study
We measure the effect of varying the number of layers the sequence-to-sequene model Table We see that the metrics improve slightly as we make the network bigger. We also measure the effect of providing user information, the results for which are provided Table & in the Appendix.
4.5 Invalid IDs
Since the model decodes the codewords of the target Semantic ID autoregressively, it is possible that the model may predict invalid IDs (i.e., IDs that do not map to any item in the recommendation dataset). In our experiments, cardinality of 256 (i.e., codebook size = 256 for each level). The number of possible IDs spanned by this combination items in the datasets we consider is 10K-20K (See Table 6). Even though the number of valid IDs is only fraction of all complete ID space, observe that the model almost always predicts valid IDs. We visualize the fraction of invalid IDs produced by TIGER as a function of the number of retrieved items K in Figure 6 For top-10 10 predictions, the fraction of invalid IDs varies from 0.1% 1.6% for the three datasets. To counter the effect of invalid IDs and to always get top- 10 valid IDs, we can increase the beam size and filter the invalid IDs.
used semantic IDs of length with each codeword having
256*, which is approximately trillion. On the other hand, the number of
is important to note that, despite generating invalid IDs, TIGER achieves state-of-the-art perfor- mance when compared to other popular methods used for sequential recommendations. One extension to handle invalid tokens could be to do prefix matching when invalid tokens are generated by the model. Prefix matching of Semantic IDs would allow retrieving items that have similar semantic meaning as the tokens generated by the model. Given the hierarchical nature of our RQ-VAE tokens, prefix matching can be thought of as model predicting item category as opposed the item index. Note that such an extension could improve the recall/NDCG metrics even further. We leave such an extension as a future work.
Conclusion
This paper proposes novel paradigm, called TIGER, retrieve candidates in recommender systems using a generative model. Underpinning this method a novel semantic ID representation for items, which uses a hierarchical quantizer (RQ-VAE) content embeddings generate tokens that form the semantic IDs. Our framework provides results in a model that can be used to train and serve without creating index the transformer memory acts semantic index for items. We note that the cardinality of our embedding table does not grow linearly with the cardinality of item space, which works in our favor compared systems that need create large embedding tables during training or generate an index for every single item. Through experiments on three datasets, we show that our model can achieve SOTA retrieval performance, while generalizing to new and unseen items.
References
[1] Himan Abdollahpouri, Masoud Mansoury, Robin Burke, and Bamshad Mobasher. The unfair- ness of popularity bias in recommendation. arXiv preprint 1rXiv:1907.13286, 2019.
[2] Moses Charikar. Similarity estimation techniques from rounding algorithms. Proceedings of the thiry-fourth annual ACM symposium Theory of computing, pages 380-388. 2002.
[3] Heng-Tze Cheng, Levent Koe, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for
10

---

recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems, pages 7-10, 2016.
[4] Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommenda- tions. In Proceedings of the 10th ACM conference on recommender systems, pages 191-198, 2016.
[5] Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. Autoregressive entity retrieval. arXiv preprint arXiv:2010.00 00904, 2020.
[6] Gabriel de Souza Pereira Moreira, Sara Rabhi, Jeong Min Lee, Ronay Ak, and Even Oldridge. Transformers4rec: Bridging the gap between nip and sequential//ession--on based recommendation. In Fifteenth ACM Conference on Recommender Systems, pages 143-153, 2021.
[7] Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[8] Shijic Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5).
arXiv preprint arXiv:2203. 1 13366, 2022.
[9] Carlos A Gomez-Uribe and Neil Hunt. The netflix recommender system: Algorithms, business value, and innovation. ACM Transactions Management Information Systems (TMIS), 6(4):1- 19, 2015.
[10] Ruining He and Julian McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web, pages 507-517, 2016.
[11] Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-based recommendations with recurrent neural networks. arXiv preprint
2015.
[12) Yupeng Hou, Zhankui He, Julian McAuley, and Wayne Xin Zhao. Learning vector-quantized item representation for transferable sequential recommenders. arXiv preprint arXiv:2210.12311 2022.
[13] Piotr Indyk and Rajeev Motwani. Approximate nearest neighbors: towards removing the curse of dimensionality. In Proceedings of the thirtieth annual ACM symposium on Theory of computing, pages 604-613, 1998.
[14] Piotr Indyk, Rajeev Motwani, Prabhakar Raghavan, and Santosh Vempala. Locality hashing in multidimensional spaces. In Proceedings of the twenty-ninth annual ACM symposium on Theory of computing, pages 618 1-65, 1997.
[15] Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence, 33(1):117-128, 2010.
[16) Wang-Cheng Kang, Derek Zhiyuan Cheng, Tiansheng Yao, Xinyang Yi, Ting Chen, Lichan Hong, and Ed H Chi. Learning to embed categorical features without embedding tables for recommendation. arXiv preprint wrXiv:2010.10784 2020.
[17] Wang- Cheng- Kang and Julian McAuley. Self-attentive sequential recommendation. In 2018 IEEE international conference on data mining (ICDM), pages 197-206. IEFF, 2018.
[18] Dongmoon Kim, Kun-su Kim, Kyo-Hyun Park, Jee-Hyong Lee, and Keon Myung Lee. A music recommendation system with dynamic k-means clustering algorithm. In Sixth international conference on machine learning and applications (ICMLA 2007), pages 399-403. IEEE, 2007.
[19] Ychuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recom- mender systems. Computer, 42(8):30-37, 2009.
[20] Tim Kraska, Alex Beutel, Ed H Chi, Jeffrey Dean, and Neoklis Polyzotis. The case for learned index structures. In Proceedings of the 2018 international conference on management of data, pages 489-504, 2018.
11

---

[21] Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11523-11532 2022.
[22] Hyunji Lee, Jaeyoung Kim, Hoyeon Chang, Hanseok Oh, Sohee Yang, Vlad Karpukhin, Yi Lu, and Minjoon Seo. Contextualized generative retrieval. arXiv preprint arXiv:2210.0000 12.002600 2022.
[23] Hyunji Lee, Sohee Yang, Hanseok Oh, and Minjoon Sco. Generative retrieval for long sequences. arXiv preprint arXiv:2204. 13596, 2022.
[24] Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. Neural attentive session-based recommendation. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pages 1419-1428, 2017.
[25] Chen Ma, Peng Kang, and Xue Liu. Hierarchical gating networks for sequential recommen- dation. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 825-833, 2019.
[26] Masoud Mansoury, Himan Abdollahpouri, Mykola Pechenizkiy, Bamshad Mobasher, and Robin Burke. Feedback loop and bias amplification in recommender systems, 2020.
[27] Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Findings of the Association for Computational Linguistics: ACL. 2022, pages 1864-1874, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[28] Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor. Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Mare van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin Sheeard,, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. Scaling up models and data with t5x and seqio. arXiv preprint 2rXiv:2203..7110 2022.
[29] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv: 150::09009999 2015.
[30] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics.
[31] Anima Singh, Trung Vu, Raghumandan Keshavan, Nikhil Mehta, Xinyang Yi, Lichan Hong, Lukasz Heldt, Li Wei, Ed Chi, and Maheswaran Sathiamoorthy. Better generalization with semantic ids: A case study in ranking for recommendations. arXiv preprint arXiv22306.08121, 2023.
[32] Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer. In Pro- ceedings of the 28th ACM international conference on information and knowledge management, pages 1441-1450, 2019.
[33] Jiaxi Tang and Ke Wang. Personalized top-n sequential recommendation via convolutional sequence embedding. In Proceedings of the eleventh ACM international conference web search and data mining, pages 565-573, 2018.
[34] Yi Tay, Vinh Q Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, et al. Transformer memory as a differentiable search index. arXiv preprint ar xrXiv:2202.0699) 2022.
[35] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
12

---

[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[37] Yujing Wang, Yingyan Hou, Haonan Wang, Ziming Miao, Shibin Wu, Hao Sun, Qi Chen, Yuqing Xia, Chengmin Chi, Guoshuai Zhao, et al. A neural corpus indexer for document retrieval. arXiv preprint S.K.:::200.027444. 2022.
[38] Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola, and Josh Attenberg. Feature hashing for large scale multitask learning. In Proceedings of the 26th annual international conference on machine learning, pages 1113-1120, 2009.
[39] Xinyang Yi, Ji Yang, Lichan Hong, Derek Zhiyuan Cheng, Lukasz Heldt, Aditee Kumthekar, Zhe Zhao, Li Wei, and Ed Chi. Sampling-bias Serpllnng-------eoe saae--bii-cooone conrecore neural modeling for large corpus item recommendations. In Proceedings of the 13th ACM Conference on Recommender Systems, pages 269-277, 2019.
[40] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. CoRR, abs/2107 07.113 2021.
[41] Shuai Zhang, Yi Tay, Lina Yao, and Aixin Sun. Next item recommendation with self-attention. arXiv preprint arXiv: 008806644, 2018.
[42] Tingting Zhang, Pengpeng Zhao, Yanchi Liu, Victor S Sheng, Jiajie Xu, Deqing Wang, Guan- feng Liu, and Xiaofang Zhou. Feature-level deeper self-attention network for sequential recommendation. In IJCAL, pages 4320-4326, 2019.
[43] Zhe Zhao, Lichan Hong, Li Wei, Jilin Chen, Aniruddh Nath, Shawn Andrews, Aditee Kumthekar, Maheswaran Sathiamorrthy, Xinyang Yi, and Ed Chi. Recommending what video to watch next: multitask ranking system. Proceedings of the 13th ACM Conference on Recommender Systems, pages 43-51, 2019.
[44] Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang. Fuzheng Zhang, Zhongyuan Wang, and Ji- Ji-Rong Wen. S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 1902, 2020.
13

---

A Related Work (cont.)
Generative Retrieval Document retrieval traditionally involved training a 2-tower model which mapped both queries and documents to the same high-dimensional vector space, followed by perform- ing an ANN or MIPS for the query over all the documents to return the closest ones. This technique presents some disadvantages like having a large embedding table [22 23 Generative retrieval is recently proposed technique that aims to fix some of the issues of the traditional approach by producing token by token either the title, name, or the document id string of the document. Cao et al. [5] proposed GENRE for entity retrieval, which used a wunsformer-basee architecture to return, token-by-token, the name of the entity referenced to in a given query. Tay et al. for document retrieval, which was the first system to assign structured semantic DocIDs to each document. Then given query, the model autoregresiively returned the Docll) of the document loken-by-token. The DSI work marks a paradigm shift in IR to generative retrieval approaches and the first successful application of an end-to-end Transformer for retrieval applications. Subsequently, Lee et al. [23] show that generative document retrieval is useful even the multi-hop setting, where complex query cannot be answered directly by single document, and hence their model generates intermediate queries, in a thain-off-thouhh manner, to ultimately generate the output for the complex query. Wang et al. [37] supplement the hierarchical k-means clustering based semantic DocIDs of Tay et al. 34 by proposing a new decoder architecture that specifically takes into account the prefixes in semantic DocIDs. In CGR [22], the authors propose way to take advantage of both the bi-encoder technique and the generative retrieval technique, by allowing the decoder, of their secodder-dcooder-bseee model, learn separate contextualized embeddings which store information about documents intrinsically. To the best of our knowledge, we are the first to use generative for retrieval models.
(44 proposed DSI
Semantic IDs created using an auto-encoder (RQ-VAE 40 21)
Vector Quantization. We refer to Vector Quantization as the process of converting a high- dimensional vector into low-dimensional tuple of codewords. One of the most straightforward techniques uses hierarchical clustering, such as the one used in [34), where clusters created in a particular iteration further partitioned into sub-clusters in the next iteration. An alternative popular approach is Vector-Quantiaa Variational AutoEncoder (VQ-VAE), which was introduced (35) as way to encode natural images into a sequence of codes. The technique works by first passing the input vector (or image) through an encoder that reduces the dimensionality. The smaller dimensional vector is partitioned and each partition quantized separately, thus resulting in a sequence of codes: one code per partition. These codes are then used by a decoder to recreate the original vector (or image).
RQ-VAE 40 21 applics residual quantization to the output of the encoder of VQ-VAE to achieve
lower reconstruction error. We discuss this technique in more detail in Subsection 3.1 Locality Sensitive Hashing (LSH) [14 [3] is a popular technique used for clustering and approximate nearest neighbor search. The partieular version that use in this paper for clustering is SimHash (2) which uses random hyperplanes to create binary vectors which serve as hashes of the items. Because it has low computational complexity and is scalable [3], we use this baseline technique for vector quantization.
B Baselines
Below we briefly describe each of the baselines with which we compare TIGER
is the first RNN-based approach that uses a customized GRU for the sequential recommendation task.
GRU4Rec
I|]
Caser (33) uses CNN architecture for capturing high-order Markov Chains by applying horizon- tal and vertical convolutional operations for sequential recommendation.
HGN 25: Hierarchical Gating Network captures the long-term and short-term user interests via a new gating architecture.
SASRec [17 : Self-Attentive Sequential Recommendation uses causal mask Transformer to model a user's s sequential interactions.
BERT4Rec (32] BERT4Rec addresses the limitations of uni-directional architectures by using a bi-directional self-attention Transformer for the recommendation task.
14

---

FDSA 442:
Feature-level Deeper Self-Attentie Network incorporates item features in addition the item embeddings as part of the input sequence in the Transformers.
S*-Ree 44: Supe-Suped Learning for Sequential Recommendation proposes pre-training
bi-directional Transformer on self-supervision tasks to improve the sequential recommendation.
P5 g P5 recent method that uses pretrained Large Language Model (L.LM) to unify different recommendation tasks in single model.
C Dataset Statisties
Table 6: Dataset statisties for the three real-world benchmarks.
Dataset
Users
# Items
Sequence Length
Mean
Median
Beauty Sports and Outdoors
22,363
12,101
8.87
6
35,598
18,357
8.32
6
Toys and Games
19,412
11,924
8.63
6
We use three public benchmarks from the Amazon Product Reviews dataset 10], containing user reviews and item metadata from May 1996 to July 2014. We use three categories of the Amazon Product Reviews dataset for the sequential recommendation task: "Beauty", "Sports and Outdoors", and "Toys and Games". [ the statistics of the datasets. We use users' review history to create item sequences sorted by timestamp and filter out users with less than 5 reviews. Following the standard evaluation protocol [17 8) we use the leave-one-out strategy for evaluation. For each item sequence, the last item is used for testing, the item before the last is used for validation, and the rest is used for training. During training, limit the number of items in user's history to 20.
D Modifications to the P5 data preprocessing
Table 7: Results for P5[8 with the standard pre pre-processing.
Sports and Outdoors
Beouty
Toys and Games
Methods
NDCGGS
NDCG
Recalle
NDCG010
Reculle
NDCG@5
1ecal1:10
0.0051
auDOss
0L0022
0,0107
0.0254
0.01
0.0070
0,0050
0.0121
0.0066
0.0076
0.008S
0.035
0.255
0,0298
0.033
0,0235
0.015
The P5 source code Paaeeeprepppooooo ppree-prccesss the Amazon dataset first create sessions for each user, containing chronologically ordered list of items that the user reviewed. After creating these sessions, the original item IDs from the dataset are remapped to integers 1, 3,.. [ Hence, the first item in the first session gets id of '1', the second item, if not seen before, gets an id of '2', and so on. Notably, this rrepproesssing scheme applied before creating training and testing splits. This results in the creation of sequential dataset where many sequences are of the form a,, + 1, + 2, ..... Given that P5 uses Sentence Piece tokenizer [30] (See Section 4. in (8). the test and train items in user session may share the sub-word and can lead to information leakage during inference.
To resolve the leakage issue, instead of assigning sequentially increasing integer ids items, we assigned random integer IDs, and then created splits for training and evaluation. The rest of the code for P5 was kept identical the source code provided in the paper. The results for this dataset are reported in Table as the row 'P5'. We also implemented version of P5 ourselves from scratch, and train the model on only sequential recommendation task. The results for our implementation are depicted as 'P5-ours'. We were also able to verify in our P5 implementation that using consecutive integer sequences for the item IDs helped us get equivalent or better metrics than those reported in P5.
15

---

Table The effect of providing user information to the recommender system
@
NDCG@5
Recall @ 10
NDCG@
No information
0.04458
0.0302 0.0321
0.06479 0.0648
0.0367
With user id (reported in the paper)
0. 0.0454
0.0384
Table 9: The mean and stand error of the metrics for different dataset (computed using runs with
different random seeds)
Datasets
Recall@S
NDCG@5
Recall @ 10
NDCG@ 10
Beauty
0.0441 = 0.00069
0.0309 = 0.00062
0.0642 = 0.00092
0.0374 = 0.00061
Sports and Outdoors
0.0278 + 0.00069
0.0189 + 0.00043
0.0419 + 0.0010
0.0234 1 0.00048
Toys and Games
0.0518 + 0.00064
0.0375 + 0.00039
0.0698 + 0.0013
0.0433 + 0.00047
0.3
1.5
0.30
10s 196)
2.21
2.20
Irvelee
3.50
0.40
083
0.0
2838
(a) Sports and Outdoors
(b) Beauty
(c) Toys and Games
Figure 6: Percentage of invalid IDs when generating Semantic IDs using Beam search for various values of K. As shown, 0.3% 6% of the IDs invalid when retrieving the top-20 items.
E
Discussion
Effects of Semantic ID length and codebook size. We tried varying the Semantic ID length and codebook size, such having ID consisting of 6 codewords each from a codebook of size 64. We noticed that the recommendation metrics for TIGER were robust to these changes. However, note that the input sequence length increases with longer IDs (i.c., more codewords per item ID), which makes the computation more expensive for our transformer- based eqquence-t--squeene model.
Scalability. To test the scalability of Semantic IDs, we ran the following experiment: We combined all the three datasets and generated Semantic IDs for the entire set of items from the three datasets. Then, we used these Semantic IDs for the recommendation task on the Beauty dataset. We compare the results from this experiment with the original experiment where the Semantic IDs are generated only from the Beauty dataset. The results are provided in Table I0 We see that there is only a small decrease in performance here.
Inference cost. Despite the remarkable success of our model on the sequential recommendation task, we note that our model can be more computationally expensive than ANN-based models during inference due to the use of beam search for autoregressive decoding. We emphasize that optimizing the computational efficiency of TIGER was not the main objective of this work. Instead, our work opens up new area of research: Recommender Systems based on Generative Retrieval. As part of future work, we will consider ways to make the model smaller or explore other ways of improving inference efficiency.
Memory cost of lookup tables. We maintain two lookup hash tables for TIGER: an Item ID to Semantic ID table and a Semantic ID to Item ID table. Note that both of these tables are generated only once and then frozen: they are generated after the training of the RQ-VAE-basee Semantic ID generation model, and after that, they are frozen for the training of the requence-to-sequeme
'hupps//githhub.c 'htrp://giiuu..comjyyyyye http:/ghiihbcoo///yyiigggn/////e
16

---

Table 10: Testing scalability by generating the Semantic IDs on the combined dataset generating the Semantic IDs on only the Beauty dataset.
R
NDCG@5
Recall@ 10
NDCG@
Semantic ID [Combined datasets] Semantic ID [Amazon Beauty]
0.04355
0.3047 0.0321
0.06314 0.0648
0.03676
0.0454
0.0384
transformer model. Each Semantic ID consists of a tuple of 4 integers, each of which are stored in bits, hence totalling 32 bits per item. Each item is represented by an Item ID, stored as bit integer. Thus, the size of each lookup table will be of the order of 64N bits, where N is the number of items in the dataset.
Memory cost of embedding tables. TIGER uses much smaller embedding tables compared to traditional recommender systems. This is because where traditional recommender systems store embedding for each item, TIGER only stores an embedding for each semantic codeword. In our experiments, used codewords each of cardinality 256 for Semantic ID representation, resulting in 1024 (256x4) embeddings. For traditional recommender systems, the number of embeddings is N, where N is the number of items in the dataset. In our experiments, N ranged from 10K to 20K depending on the dataset. Hence, the memory cost of TIGER's embedding table is 1024d, where is the dimension of the embedding, whereas the memory cost for embedding lookup tables in traditional recommendation sy systems is Nd.
17


Reviews:
[{"review_id": "YP2USPAoU3", "reviewer": "Reviewer_VYRN", "summary": "The paper introduces TIGER (Transformer Index for Generative Recommenders), a novel generative retrieval model for sequential recommender systems. This innovative approach involves generating Semantic IDs—sequences of semantic tokens derived from an item's text features. A hierarchical quantizer (RQ-VAE) is employed on content embeddings to formulate these tokens, thereby allowing the model to predict the Semantic IDs for generative retrieval. The authors claim that TIGER outperforms current state-of-the-art models, while offering a new solution for cold-start recommendations and increasing item diversity by adjusting the temperature parameters. Furthermore, it showcases impressive scalability and the capacity to generalize to novel and unseen items.", "questions": "Please refer to the weaknesses section. ", "limitations": "Please refer to the weaknesses section. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and technically solidity, effectively combining novel concepts such as Differentiable Search Index (DSI) using Transformer memory and a hierarchical quantizer, RQ-VAE, for generative retrieval in sequential recommendation task.\n\n2. The idea driving the work is both logical and compelling, providing a fresh perspective on generative retrieval tasks in the context of recommender systems.\n\n3. Empirical results show that TIGER not only outperforms the SOTA baselines on the SR task but also exhibits an ability to generalize to unseen items and enhance item recommendation diversity.\n", "weaknesses": "1. The paper could benefit from more detailed information about the text encoder's input, given that quantization is a crucial part of Semantic ID generation. It's unclear whether all components depicted in Figure 2(a) are utilized. Additionally, handling missing information/data in datasets, such as the Amazon Beauty dataset, needs addressing (E.g., in the Amazon Beauty dataset, approximately 50% of the brand information is missing.). The authors should explain how they preprocessed the data. \n\n2. The paper's treatment of related work and baseline comparisons raises some concerns. Specifically, the performance results of the P5 model appear notably lower than what's presented in its original paper. The authors have explained the data reprocessing using a random item ID, but a comparison using the more updated benchmark in [2] - which also uses a random item ID - could offer a more balanced analysis. It would be insightful to understand the reasons behind the observed performance degradation of the P5 model in authors’ setting.\n\n3. The proposed solution to collision problems, appending an extra token at the end of the ordered semantic codes, raises questions about scalability. For instance, if ten items share the Semantic ID (12,24,52), would the model need to add two extra tokens and maintain a new lookup table?\n\n4. The experimental design for cold-start recommendations in section 4.3 could be revisited. The removal of 5% of test items from the training data split might change the original training sequences, potentially leading to shorter sequences and artificially enhancing recall results in the cold-start retrieval setting.\n\n5. A comparison with additional ID encoding methods, similar to those presented in [2], could strengthen the argument for RQ-VAE's superiority and effectiveness.\n\n6. While the authors state that inference cost is not their main contribution, a rough comparison of TIGER's inference cost to LSH and NN-based methods would help to understand this method's trade-offs.\n\n\nReferences.\n[1] OpenP5: Benchmarking Foundation Models for Recommendation(https://arxiv.org/pdf/2306.11134.pdf)\n[2] How to Index Item IDs for Recommendation Foundation Models. (https://arxiv.org/abs/2305.06569)\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and effort in reviewing our paper and providing valuable comments. We are glad that the reviewer feels that our paper is technically solid, combines novel ideas, and provides a fresh perspective on generative retrieval tasks in the context of recommender systems. \n\nPlease find our responses to your concerns below:\n\n> The paper could benefit from more detailed information about the text encoder's input, given that quantization is a crucial part of Semantic ID generation. It's unclear whether all components depicted in Figure 2(a) are utilized. Additionally, handling missing information/data in datasets, such as the Amazon Beauty dataset, needs addressing (E.g., in the Amazon Beauty dataset, approximately 50% of the brand information is missing.). The authors should explain how they preprocessed the data.\n\nIn our implementation, we used the item id, title, brand, categories, price and the description of the items (whenever the attributes were available) to generate the Semantic IDs. The prompt provided to the pre-trained text encoder is the following: \"Item ID: <>. Title: <>. Brand: <>. Categories: <>. Description: <>. Price: <>\". If any of these attributes is not available, we simply skip that attribute.\nPlease note that this information is only used in the Semantic ID generation; these attributes are not used when training the Transformer on the sequential recommendation task. We will clarify this in the final version of the paper.\n\n> The paper's treatment of related work and baseline comparisons raises some concerns. Specifically, the performance results of the P5 model appear notably lower than what's presented in its original paper. The authors have explained the data reprocessing using a random item ID, but a comparison using the more updated benchmark in [2] - which also uses a random item ID - could offer a more balanced analysis. It would be insightful to understand the reasons behind the observed performance degradation of the P5 model in authors’ setting.\n\nThank you for bringing the new paper [2] to our attention. While we are not able to implement the new indexing techniques proposed in [2] within the time frame of the rebuttal period, we would like to explain again the reason for the disparity in the results for P5 in our paper vs the original P5 paper: We suspect that there exist data leakage from the preprocessing stage in the P5 paper’s evaluation. We explain details in the supplementary (Please see section D). In particular, in the official P5 implementation the item-ids are assigned sequentially during preprocessing of the user session. For instance, if a given user has interacted with the following items: Item_A, Item_B, Item_C, Item_D, Item_E, item_F, during preprocessing the assigned ids can be 20, 21, 22, 23, 24, 25. More importantly, this assignment in P5 is carried out before creating train/validation/test splits. The training, validation and testing splits for this example are as follows:\n\nTraining split: Input: 20, 21, 22. Target item: 23\n\nValidation split: Input: 20, 21, 22, 23. Target item: 24\n\nTesting split: 20, 21, 22, 23, 24; Target item: 25\n\nSince the P5 model uses Sentence Piece Tokenizer[3] for tokenization, the item ids are split into subwords (e.g. 21 in the input is split as “2” and “1”). Therefore, it is likely that same subwords occur in the input item sequence and the target item leading to data leakage.\nTo fix this, we used the original P5 code (available on GitHub) and retrained it on the dataset without doing sequential item id assignment to avoid the aforementioned leakage, keeping all the other training hyperparams the same. The result is presented in Appendix D, Table 2\n\n> The proposed solution to collision problems, appending an extra token at the end of the ordered semantic codes, raises questions about scalability. For instance, if ten items share the Semantic ID (12,24,52), would the model need to add two extra tokens and maintain a new lookup table?\n\nFor scaling the Semantic IDs wrt items, we can increase the cardinality of the codebook used in RQ-VAE. (In our experiments, we used a cardinality of 256 which worked well for all the benchmarks). Please note that theoretically the number of items that can be uniquely represented with Semantic IDs grow  exponentially with codebook cardinality. For instance, three tokens with 256 cardinality can potentially represent $256^3$ items. However, since we use item content features to learn Semantic ID, there may exist collisions when two items have high content similarity. We found that appending a single unique token to the learned Semantic ID works surprisingly well in avoiding collisions.\n\n> The experimental design for cold-start recommendations in section 4.3 could be revisited. The removal of 5% of test items from the training data split might change the original training sequences, potentially leading to shorter sequences and artificially enhancing recall results in the cold-start retrieval setting.\n\nWhile we agree that the distribution can change, it does not change drastically as shown below, and hence the results presented in the paper still provide some indication of cold-start performance in the original distribution. Below we provide the mean and median for user sequence length for the standard and the cold-start experiment recommendation setting considered in the paper:\n\nStatistics for the number of items in a user session:\n\nStandard design: Mean: 8.87 & Median: 6\n\nCold-start design: Mean 8.53 & Median: 6\n\n> A comparison with additional ID encoding methods, similar to those presented in [2], could strengthen the argument for RQ-VAE's superiority and effectiveness.\n\nWe thank the reviewer for pointing us to this important line of new work recently published. We will incorporate some of those ID encoding methods in future work.\n\n[3] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units."}, {"review_id": "gSod4G814b", "reviewer": "Reviewer_mHfk", "summary": "This paper proposes a generative retrieval for an efficient recommendation, where the retrieval model autoregressively decodes the identifiers of the target candidates. The evaluation results show the proposed method can perform better than competing baselines on various dataset. ", "questions": "see the attachments. ", "limitations": "N/A", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "1.  Generative retrieval-based recommender model with item features\n2.  Extensive evaluation show the superiority of the proposed algorithm", "weaknesses": "1. Limited contributions. This paper claims the first to propose generative retrieval for recommendation systems using Semantic ID representation of items. However, this is not true. RQ-VAE is also simply used for obtaining the semantic ids, and no improvements are made in the paper.\n\n2. The similar idea is mainly used for efficient recommendation, but no efficiency study is performed. \n\n3. The settings of some baselines are not reported. I know that SASRec could be significantly improved using some tricks. It is not clear whether these tricks are used in the experiments. \n\n4. The comparison is not fair. The proposed algorithm contains side information, but some baselines do not take them into account. \n\n5. The experiments are only performed on the Amazon datasets. It is not sufficient to prove the effectiveness of the proposed algorithms. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time in reviewing our paper and providing feedback. We are glad to know that the reviewer found our evaluation extensive. Please find our responses to your concerns below:\n> Limited contributions. This paper claims the first to propose generative retrieval for recommendation systems using Semantic ID representation of items. However, this is not true. RQ-VAE is also simply used for obtaining the semantic ids, and no improvements are made in the paper.\n\nIt would be helpful for us if you could please provide a link to the prior work that performs generative recommendation using semantic IDs. As for RQ-VAE, please note that we do not claim that we improve upon RQ-VAE, but rather we use it as a part of our generative recommender system.\n>The similar idea is mainly used for efficient recommendation, but no efficiency study is performed.\n\nWe are unaware of any prior work that uses a similar idea for efficient recommendation, it would be helpful if you could please provide a link to the prior work that uses a similar idea for recommendation. We would like to state again that the focus of this paper was not on efficient recommendation, but rather proposing a new paradigm for recommender systems.\n>The settings of some baselines are not reported.  I know that SASRec could be significantly improved using some tricks. It is not clear whether these tricks are used in the experiments\n \nIn our paper, the results for all baselines are taken from the publicly accessible results available at https://github.com/aHuiWang/CIKM2020-S3Rec (we have also mentioned this in the paper). We are unaware of the tricks that can be used to improve SASRec, and it would be very helpful if you could point us to any information regarding that.\n>The comparison is not fair. The proposed algorithm contains side information, but some baselines do not take them into account.\n\n We agree that some baselines do not use side information, however, please note that 1) it is not straightforward how to incorporate side information into those baselines, 2) we only use side-information like title, price, brand, etc. that are available for most datasets, and 3) the side information is only used in the generation of Semantic IDs and not used for training the Transformer architecture on the sequential recommendation task. We will add a discussion regarding this in the discussion section of our paper.\n"}, {"review_id": "8iz63L5kmU", "reviewer": "Reviewer_5DSM", "summary": "The paper proposes TIGER, a generative retrieval approach to sequential recommendation. More specifically, TIGER proposes to use semantic IDs of a product (based off RQ-VAE quantization of sentence-T5 product feature embeddings) instead of typically used atomic item-IDs in the sequential recommendation pipeline. Empirical analysis on 3 Amazon subsets show promising results compared to itemID-only and ID+text based existing recommenders.", "questions": "- It seems the text+ID based recommenders are not consistent in what product features are being used. E.g., S3-Rec seems to use category and brand information whereas FDSA seems to use category, brand, and description. Is the improvement in TIGER just because of using extra feature signals like title and price?\n- Unclear which two semantic IDs are “closer”? I had to assume that since a residual quantization is being performed, being close to the first quanta holds more importance than the last.", "limitations": "I will use this space to provide some (hopefully constructive) suggestions for this paper:\n\n- Best to outline in Table 1 that S3-Rec and FDSA use item-text features whereas all others are ID-only recommenders.\n- It would be interesting to see the performance on the unprocessed Amazon datasets since arbitrary preprocessing of user-data has been noted to interfere with true performance [1].\n- It would be interesting to see a TIGER variant where only three-length semantic IDs are used (i.e., non-unique items would be allowed) as it would become a text-only recommender.\n- Complementing the zero-shot analysis in the paper with an analysis of TIGER’s performance over the user/item coldness spectrum (in comparison with other models)\n- Although released after this submission to NeurIPS, [2] might be a good reference to compare TIGER with (this comparison does not affect my review).\n\n[1] Sachdeva, N., Wu, C. J., & McAuley, J. On sampling collaborative filtering datasets. WSDM ‘22.\n\n[2] Li, J., Wang, M., Li, J., Fu, J., Shen, X., Shang, J., & McAuley, J. Text Is All You Need: Learning Language Representations for Sequential Recommendation. arXiv preprint arXiv:2305.13731.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Interesting design to use quantized embeddings of items (semantic IDs)\n- Successful application (i.e., outperforming atomic ID-based recommenders) of the generative retrieval paradigm for sequential recommendation\n- State of the art results on medium-sized academic datasets", "weaknesses": "- No ablations, e.g., effect of number of codebooks, prepending user-ID, codebook size, Transformer architecture, etc.\n- No significance tests while reporting ranking results", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and effort in reviewing our paper. We are glad that the reviewer liked the idea of Semantic IDs and the generative retrieval paradigm for sequential recommendation. Please find our responses to your concerns below:\n\n> No ablations, e.g., effect of number of codebooks, prepending user-ID, codebook size, Transformer architecture, etc.\n>No significance tests while reporting ranking results\n\nThank you for these suggestions. We have done some ablation experiments using different Semantic ID generation techniques (RQ-VAE vs LSH vs Random). We will add some more ablation experiments and compute the significance tests (variance and standard deviation) as you have suggested. Note that since setting up and running these experiments will take some time, we will keep adding the results during the discussion period.\n\n>It seems the text+ID based recommenders are not consistent in what product features are being used. E.g., S3-Rec seems to use category and brand information whereas FDSA seems to use category, brand, and description. Is the improvement in TIGER just because of using extra feature signals like title and price?\n\nWe would like to clarify that we do not use additional product features when training the Transformer model on the sequential recommendation task (similar to S3-Rec and FDSA baselines). We only use product features to generate Semantic IDs which are used as alternatives to randomly assigned item ids in traditional recommendation systems. We attribute the improvement in recommendation performance in TIGER to the Transformer architecture’s ability to act as a retrieval index. In our experiments, we show the Semantic Ids are an essential ingredient to allow Transformer architecture to perform well on the recommendation task.\n\n>Unclear which two semantic IDs are “closer”? I had to assume that since a residual quantization is being performed, being close to the first quanta holds more importance than the last.\n\nYes, the reviewer’s interpretation is correct. For the semantic IDs generated using RQ-VAE, the first codeword holds more importance, and the subsequent codewords work at increasingly finer granularities. For examples, if three items have semantic ids (1, 4, 10), (1, 4, 15) and (1, 3, 8), then all the three items are similar (since they all belong to ‘category/cluster’ 1 on the first level), but item #1 and #2 are even more similar since they also belong to the same ‘category/cluster’ on the second level. On the other hand, for items with semantic ids  (1, 4, 10), (3, 5, 10) and (1, 3, 8), items #1 and #3 would be more closely related than items #1 and #2.\n\nLimitations:\n> Best to outline in Table 1 that S3-Rec and FDSA use item-text features whereas all others are ID-only recommenders.\n\n Thanks for pointing this out, we will add this to our paper\n>It would be interesting to see the performance on the unprocessed Amazon datasets since arbitrary preprocessing of user-data has been noted to interfere with true performance [1].\n\nWe only do two filtering steps in our preprocessing: \nRemove sequences with length less than 5. This is because we reserve the last two items in each sequence as the validation and the test samples respectively. Hence, with sequences of length less than 5, we will end up with a training sequence of length less than 3, that is, a training sequence of length 1 or 2. Thus, we remove sequences of length less than 5. This is also done in prior literature.\nTruncate sequences at length 20. We mainly did this because this is done in prior literature.\n"}, {"review_id": "FD9PrCzEac", "reviewer": "Reviewer_aLHH", "summary": "The paper is trying to approach sequential recommendation task using semantic IDs. By incorporating semantic IDs, the paper shows that the retrieval performance can be improved. ", "questions": "Besides the questions in the weakness section above, another question is \n1. Semantic ID idea seems useful, but would it be nicer if we do not need to use semantic embeddings as middle layer and directly do generative AI on word tokens?", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The ablation study conducted is good. For example, it compares RQ-VAE against LSH, and random.\n2. The proposed method and framework is intuitive and makes sense.\n3. The conducted experiment seems extensive and the proposed method is shown to outperform than non-semantic-id method like S3-Rec.", "weaknesses": "1. The paper title seems to emphasize on Generative Retrieval rather than on ranking, if that is the case, in the experiment section, shouldn't we care more about recall than NDCG? Why proposed method seems to improve NDCG a lot but much less so on recall metrics?\n2. The novelty of the proposed method seems limited, the main idea seems to overlap with paper \"Learning Vector-Quantized Item Representation for Transferable Sequential Recommenders\", and probably should be considered as incremental value added to the previous published paper. Although the method emphasized on generative AI, it is still generative on the item semantic ID level, it would be more significant if the generative mechanism is on the word tokens, in my opinion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and effort in reviewing our paper and providing constructive feedback. We are glad to know that the reviewer found the experiments extensive, and that the proposed technique is intuitive. Please find our responses to your concerns below:\n\n> The paper title seems to emphasize on Generative Retrieval rather than on ranking, if that is the case, in the experiment section, shouldn't we care more about recall than NDCG? Why proposed method seems to improve NDCG a lot but much less so on recall metrics?\n\nThank you for pointing this out. Our goal for this paper was to develop a generative retrieval system for recommender systems. NDCG is a useful metric in retrieval that can be used to evaluate how good the order of the retrieved list is. The improved NDCG metrics of our method indicates that our retrieval works very well in getting relevant items in the top of the list.\n\n\n> The novelty of the proposed method seems limited, the main idea seems to overlap with paper \"Learning Vector-Quantized Item Representation for Transferable Sequential Recommenders\", and probably should be considered as incremental value added to the previous published paper.\n\nWe consider \"Learning Vector-Quantized Item Representation for Transferable Sequential Recommenders\" as a concurrent work to our own, and have cited it as such. Please see lines 91-95 to see the differences between that and our paper.\n\n>Semantic ID idea seems useful, but would it be nicer if we do not need to use semantic embeddings as middle layer and directly do generative AI on word tokens?\n\nWe would like to clarify how the current framework is generative: Similar to how a sequence of words from a generative LLM would describe the semantics of an item that may or may not exist (for example “blue nike men’s running shoes, size 11, limited edition”), similarly the Semantic ID describes the semantics of an item. Hence, when our model generates a Semantic ID, it essentially generates the semantics of a product (similar to how an LLM based generative recommender model would generate sequence of words describing the item). While using word tokens directly to represent items is an interesting idea, we leave this exploration as a future research direction.\n"}, {"review_id": "ob0IJLhPQw", "reviewer": "Reviewer_5EdZ", "summary": "This paper proposes a generative retrieval approach wherein the model autoregressively decodes the identifiers, called Semantic ID, for the target candidate. They propose a method to generate the semantic ID for items which is a tuple of codewords which are trained from the embeddings of the text component of the items obtained using a standard sentence BERT type of model. Finally, a seq2seq model is trained to predict the semantic ids for the next item given the current item. The authors show that the proposed technique outperforms SOTA techniques on some retrieval datasets. They also claim that the semantic ID trick enhances the generalization property of the seq2seq model especially for cold start cases.", "questions": "I would like to see large code sets for items. This would make the system truly scalable. Otherwise, constraining embeddings to 3 codewords seems a step in the opposite direction which will work with very small item sets in controlled environments. \n\nI would also ask the authors to perform comparative experiments with P5, where the parameters do not need to be comparable, but the best performance obtained should be the primary aim. ", "limitations": "NA", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "Originality: The paper seems to present a novel idea which is original enough. The items are represented in a novel semantic ID sequence form which is then used in a standard seq2seq framework.\n\nClarity: I think the paper is clear enough to understand and reproduce.\n\n", "weaknesses": "The method itself is and its larger claim of generative in nature is the primary concern I have with the paper. The proposed method tries to generate semantic id sequence for items already seen by the system. Any novel combination of semantic ID will lead to an item which is not part of the retrieval system. This leads to the authors constraining the semantic IDs to just 3 code words and then an ad hoc 4th code word to introduce uniqueness in the codes. This part seems to suit small, well controlled item sets, but the scaling out of such a system to billions of items seems not clearly explained or experimented with. \n\nThe experimental results seem not consistent with the published paper, especially for P5 paper. The numbers reported in the paper are higher than this paper which are not shown as is, but the authors implement their own P5 and compare results which are way different than the published results.\n\nThe other comparative results are all copied over from the benchmark mentioned in the footnote ttps://github.com/aHuiWang/CIKM2020-S3Rec. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and effort in reviewing our paper and providing valuable feedback. We are glad to know that the reviewer found our idea novel, and the paper easy to understand and reproduce. Please find our responses to your questions below:\n\n> I would like to see large code sets for items. This would make the system truly scalable. Otherwise, constraining embeddings to 3 codewords seems a step in the opposite direction which will work with very small item sets in controlled environments.\n\nThank you for the suggestion. Scaling with Semantic IDs can be easily achieved by increasing the size/cardinality of the codebook used in the RQ-VAE model. In our experiments, we used 256 as the codebook size. Note that theoretically the number of items that can be uniquely represented with Semantic IDs grow exponentially with codebook cardinality. For instance, three tokens with 256 cardinality can potentially represent $256^3$ items. However, since Semantic IDs are generated using content features, multiple items may have the same Semantic ID (referred to as a collision) when the items have similar content features. This can be easily mitigated by adding an additional token that uniquely identifies items that undergo collisions. (See Ln 168-176 for details).\n\n>I would also ask the authors to perform comparative experiments with P5, where the parameters do not need to be comparable, but the best performance obtained should be the primary aim.\n\nWe have explained in Appendix D the reason for not directly taking the numbers reported in the P5 paper. We suspect that there exists data leakage in the model input and expected target item due to the preprocessing done in the P5 implementation. In particular, in the official P5 implementation the item-ids are assigned sequentially during preprocessing of the user session. For instance, if a given user has interacted with the following items: Item_A, Item_B, Item_C, Item_D, Item_E, item_F, during preprocessing the assigned ids can be 20, 21, 22, 23, 24, 25. More importantly, this assignment in P5 is carried out before creating train/validation/test splits. The training, validation and testing splits for this example are as follows:\n\nTraining split: Input: 20, 21, 22. Target item: 23\n\nValidation split: Input: 20, 21, 22, 23. Target item: 24\n\nTesting split: 20, 21, 22, 23, 24; Target item: 25\n\nSince the P5 model uses Sentence Piece Tokenizer[1], the item ids are split into subwords (e.g. 21 is split as “2” and “1”). Therefore, it is likely that same subwords occur in the input and target items leading to data leakage. To fix this, we used the original P5 code (available on GitHub) and retrained it on the dataset without doing sequential item id assignment to avoid the aforementioned leakage, keeping all the other training hyperparams the same. The result of this is presented in Appendix D, Table 2. We also implemented our own version of P5 with random ids and those results are also presented in Appendix D, Table 2.\n"}]

Author rebuttal:

Meta-review: The paper "A Generative Recommender System with Semantic Identifiers" introduces a new approach to recommender systems that addresses the problem of cold start. The proposed approach uses a generative model to predict identifiers for recommended items based on semantically meaningful codewords. This allows the recommender system to recommend items that are relevant to the user's interests, even if the user has not interacted with the system before.

The paper is well-written and the proposed approach is novel and interesting. The authors provide a thorough evaluation of the proposed approach, and the results show that it outperforms state-of-the-art methods on several benchmark datasets.

The only potential limitation of the paper is that the applicability and performance of the proposed approach at scale have not been demonstrated as raised by 5EdZ. However, the authors argue that the Semantic ID approach is scalable and that it could be applied to real-world recommender systems providing a reference where it has been done.

Overall, I believe that the paper makes a significant contribution to the field of recommender systems. The proposed approach is novel and effective, and it has the potential to improve the performance of recommender systems in a variety of settings. I recommend that the paper be accepted for publication.

Reviewer mHfk still has some valid concerns, in particular about the possibility to tune more some baselines and using more data sources, but I think the discussion clarified  the positioning and overall I like the idea of this work.
Decision: Accept (poster)