Title: Recommender Systems with Generative Retrieval
Venue: NeurIPS 2023
Accepted: yes
Keywords: Recommender Systems

Generative Retrieval

Vector Quantization
Abstract: Modern recommender systems perform large-scale retrieval by embedding queries and item candidates in the same unified space, followed by approximate nearest neighbor search to select top candidates given a query embedding. In this paper, we propose a novel generative retrieval approach, where the retrieval model autoregressively decodes the identifiers of the target candidates. To that end, we create semantically meaningful tuple of codewords to serve as a Semantic ID for each item. Given Semantic IDs for items in a user session, a Transformer-based sequence-to-sequence model is trained to predict the Semantic ID of the next item that the user will interact with. We show that recommender systems trained with the proposed paradigm significantly outperform the current SOTA models on various datasets. In addition, we show that incorporating Semantic IDs into the sequence-to-sequence model enhances its ability to generalize, as evidenced by the improved retrieval performance observed for items with no prior interaction history.

Recommender Systems with Generative Retrieval

Abstract

Modern recommender systems perform large-scale retrieval by embedding queries and item candidates in the same unified space, followed by approximate nearest neighbor search to select top candidates given a query embedding. In this paper, we propose a novel generative retrieval approach, where the retrieval model autoregressively decodes the identifiers of the target candidates. To that end, we create semantically meaningful tuple of codewords to serve as a Semantic ID for each item. Given Semantic IDs for items in a user session, a Transformer-based sequence-to-sequence model is trained to predict the Semantic ID of the next item that the user will interact with. We show that recommender systems trained with the proposed paradigm significantly outperform the current SOTA models on various datasets. In addition, we show that incorporating Semantic IDs into the sequence-to-sequence model enhances its ability to generalize, as evidenced by the improved retrieval performance observed for items with no prior interaction history.

1 Introduction

Recommender systems help users discover content of interest and are ubiquitous in various recommendation domains such as videos [4, 43, 9], apps [3], products [6, 8], and music [18, 19]. Modern recommender systems adopt a retrieve-and-rank strategy, where a set of viable candidates are selected in the retrieval stage, which are then ranked using a ranker model. Since the ranker model works only on the candidates it receives, it is desired that the retrieval stage emits highly relevant candidates.

Orange shoes, Brand X

Red shoes, Brand Y

Orange shoes, Brand Y

Semantic ID

Generator

(5, 23, 55)

(5, 25, 78)

(5, 25, 55)

Item

Lookup

Generative

Retrieval

User History

Next item

There are standard and well-established methods for building retrieval models. Matrix factorization [19] learns query and candidate embeddings in the same space. In order to better capture the non-linearities in the data, dual-encoder architectures [39] (i.e., one tower for the query and another for the candidate) employing inner-product to embed the query and candidate embeddings in the same space have become popular in recent years. To use these models during inference, an index that stores the embeddings for all items is created using the candidate tower. For a given query, its embedding is obtained using the query tower, and an Approximate Nearest Neighbors (ANN) algorithm is used for retrieval. In recent years, the dual encoders architectures have also been extended for sequential recommendations [11, 24, 41, 17, 32, 6, 44] that explicitly take into account the order of user-item interactions.

We propose a new paradigm of building generative retrieval models for sequential recommendation. Instead of traditional query-candidate matching approaches, our method uses an end-to-end generative model that predicts the candidate IDs directly. We propose to leverage the Transformer [36] memory (parameters) as an end-to-end index for retrieval in recommendation systems, reminiscent of Tay et al. [34] that used Transformer memory for document retrieval. We refer to our method as Transformer Index for GEnerative Recommenders (TIGER). A high-level overview of TIGER is shown in Figure 1. TIGER is uniquely characterized by a novel semantic representation of items called "Semantic ID" - a sequence of tokens derived from each item's content information. Concretely, given an item's text features, we use a pre-trained text encoder (e.g., SentenceT5 [27]) to generate dense content embeddings. A quantization scheme is then applied on the embedding of an item to form a set of ordered tokens/codewords, which we refer to as the Semantic ID of the item. Ultimately, these Semantic IDs are used to train the Transformer model on the sequential recommendation task.

Representing items as a sequence of semantic tokens has many advantages. Training the transformer memory on semantically meaningful data allows knowledge sharing across similar items. This allows us to dispense away with the atomic and random item Ids that have been previously used [33, 42, 11, 8] as item features in recommendation models. With semantic token representations for items, the model is less prone to the inherent feedback loop [1, 26, 39] in recommendation systems, allowing the model to generalize to newly added items to the corpus. Furthermore, using a sequence of tokens for item representation helps alleviate the challenges associated with the scale of the item corpus; the number of items that can be represented using tokens is the product of the cardinality of each token in the sequence. Typically, the item corpus size can be in the order of billions and learning a unique embedding for each item can be memory-intensive. While random hashing-based techniques [16] can be adopted to reduce the item representation space, in this work, we show that using semantically meaningful tokens for item representation is an appealing alternative. The main contributions of this work are summarized below:

We propose TIGER, a novel generative retrieval-based recommendation framework that assigns Semantic IDs to each item, and trains a retrieval model to predict the Semantic ID of an item that a given user may engage with.

We show that TIGER outperforms existing SOTA recommender systems on multiple datasets as measured by recall and NDCG metrics.

We find that this new paradigm of generative retrieval leads to two additional capabilities in sequential recommender systems: 1. Ability to recommend new and infrequent items, thus improving cold-start recommendations, and 2. Ability to generate diverse recommendations using a tunable parameter.

Paper Overview. In Section 2, we provide a brief literature survey of recommender systems, generative retrieval, and the Semantic ID generation techniques we use in this paper. In Section 3, we explain our proposed framework, and outline the various techniques we use for Semantic ID generation. We present the result of our experiments in Section 4, and conclude the paper in Section 5.

2 Related Work

Sequential Recommenders. Using deep sequential models in recommender systems has developed into a rich literature. GRU4REC [11] was the first to use GRU based RNNs for sequential recommendations. Li et al. [24] proposed Neural Attentive Session-based Recommendation (NARM), where an attention mechanism along with a GRU layer is used to track long term intent of the user.

Bidirectional Transformer Encoder

User_5

t_u5

Item 515

Sem. ID = (5, 25, 78)

Item 233

Sem. ID = (5, 23, 55)

t_23

t_55

t_5

Tokens

Transformer Decoder

<BOS>

Item 64

Sem. ID = (5, 25, 55)

Item Interaction History of User 5

t_25

t_78

t_5

Encoded

Context

t_25

t_5

t_5

<EOS>

t_55

t_25

t_55

Next Item

(a) Semantic ID generation for items using quantization of content embeddings.

Neurips Version (b) Transformer based encoder-decoder setup for building the sequence-to-sequence model used for generative retrieval.

AttRec [41] proposed by Zhang et al. used self-attention mechanism to model the user's intent in the current session, and personalization is ensured by modeling user-item affinity with metric learning. Concurrently, Kang et al. also proposed SASRec [17], which used self-attention similar to decoder-only transformer models. Inspired by the success of masked language modeling in language tasks, BERT4Rec [32] and Transformers4Rec [6] utilize transformer models with masking strategies for sequential recommendation tasks. S 3 -Rec [44] goes beyond just masking by pre-training on four self-supervised tasks to improve data representation. The models described above learn a high-dimensional embedding for each item and perform an ANN in a Maximum Inner Product Search (MIPS) space to predict the next item. In contrast, our proposed technique, TIGER, uses Generative Retrieval to directly predict the Semantic ID of the next item.

P5 [8] fine-tunes a pre-trained large language models for multi-task recommender systems. The P5 model relies on the LLM tokenizer (SentencePiece tokenizer [29]) to generate tokens from randomlyassigned item IDs. Whereas, we use Semantic ID representation of items thay are learned based on the content information of the items. In our experiments (Table 2), we demonstrate that recommendation systems based on Semantic ID representation of items yield much better results than using random codes.

Semantic IDs. Hou et al. proposed VQ-Rec [12] to generate 'codes' (analogous to Semantic IDs) using content information for item representation. However, their focus is on building transferable recommender systems, and do not use the codes in a generative manner for retrieval. While they also use product quantization [15] to generate the codes, we use RQ-V AE to generate Semantic IDs, which leads to hierarchical representation of items (Section 4.2). In a concurrent work to us, Singh et al. [31] show that hierarchical Semantic IDs can be used to replace item IDs for ranking models in large scale recommender systems improves model generalization.

Generative Retrieval. While techniques for learning search indices have been proposed in the past [20], generative retrieval is a recently developed approach for document retrieval, where the task is to return a set of relevant documents from a database. Some examples include GENRE [5], DSI [34], NCI [37], and CGR [22]. A more detailed coverage of the related work is in Appendix A. To the best of our knowledge, we are the first to propose generative retrieval for recommendation systems using Semantic ID representation of items.

3 Proposed Framework

Our proposed framework consists of two stages:

Semantic ID generation using content features. This involves encoding the item content features to embedding vectors and quantizing the embedding into a tuple of semantic codewords. The resulting tuple of codewords is referred to as the item's Semantic ID.

Training a generative recommender system on Semantic IDs. A Transformer model is trained on the sequential recommendation task using sequences of Semantic IDs.

Embedding

+

+

=

DNN

Decoder

7 ,

1  ,

4)

(

0

1

2

3

4

5

6

7

codebook_1

0

1

2

3

4

5

6

7

codebook_2

0

1

2

3

4

5

6

7

codebook_3

Semantic codes

Quantized representation

d=2

=

-

3

2

d=1

=

-

2

1

d=0

=

-

1

Embedding

DNN

Encoder

Residual Quantization

3.1 Semantic ID Generation

In this section, we describe the Semantic ID generation process for the items in the recommendation corpus. We assume that each item has associated content features that capture useful semantic information ( e.g. titles or descriptions or images). Moreover, we assume that we have access to a pre-trained content encoder to generate a semantic embedding x ∈ R d . For example, general-purpose pre-trained text encoders such as Sentence-T5 [27] and BERT [7] can be used to convert an item's text features to obtain a semantic embedding. The semantic embeddings are then quantized to generate a Semantic ID for each item. Figure 2a gives a high-level overview of the process.

We define a Semantic ID to be a tuple of codewords of length m . Each codeword in the tuple comes from a different codebook. The number of items that the Semantic IDs can represent uniquely is thus equal to the product of the codebook sizes. While different techniques to generate Semantic IDs result in the IDs having different semantic properties, we want them to at least have the following property: Similar items (items with similar content features or whose semantic embeddings are close) should have overlapping Semantic IDs. For example, an item with Semantic ID (10 , 21 , 35) should be more similar to one with Semantic ID (10 , 21 , 40) , than an item with ID (10 , 23 , 32) . Next, we discuss the quantization schemes which we use for Semantic ID generation.

RQ-VAE for Semantic IDs. Residual-Quantized Variational AutoEncoder (RQ-VAE) [40] is a multi-level vector quantizer that applies quantization on residuals to generate a tuple of codewords (aka Semantic IDs). The Autoencoder is jointly trained by updating the quantization codebook and the DNN encoder-decoder parameters. Fig. 3 illustrates the process of generating Semantic IDs through residual quantization.

RQ-VAE first encodes the input x via an encoder E to learn a latent representation z := E ( x ) . At the zero-th level ( d = 0 ), the initial residual is simply defined as r 0 := z . At each level d , we have a codebook C d := { e k } K k =1 , where K is the codebook size. Then, r 0 is quantized by mapping it to the nearest embedding from that level's codebook. The index of the closest embedding e c d at d = 0 , i.e., c 0 = arg min i ‖ r 0 -e k ‖ , represents the zero-th codeword. For the next level d = 1 , the residual is defined as r 1 := r 0 -e c 0 . Then, similar to the zero-th level, the code for the first level is computed by finding the embedding in the codebook for the first level which is nearest to r 1 . This process is repeated recursively m times to get a tuple of m codewords that represent the Semantic ID. This recursive approach approximates the input from a coarse-to-fine granularity. Note that we chose to use a separate codebook of size K for each of the m levels, instead of using a single, mK -sized codebook. This was done because the norm of residuals tends to decrease with increasing levels, hence allowing for different granularities for different levels.

Once we have the Semantic ID ( c 0 , . . . , c m -1 ) , a quantized representation of z is computed as ̂ z := ∑ m -1 d =0 e c i . Then ̂ z is passed to the decoder, which tries to recreate the input x using ̂ z . The RQ-VAE loss is defined as L ( x ) := L recon + L rqvae, where L recon := ‖ x -̂ x ‖ 2 , and L rqvae := ∑ m -1 d =0 ‖ sg [ r i ] -e c i ‖ 2 + β ‖ r i -sg [ e c i ] ‖ 2 . Here ̂ x is the output of the decoder, and sg is the stop-gradient operation [35]. This loss jointly trains the encoder, decoder, and the codebook.

As proposed in [40], to prevent RQ-VAE from a codebook collapse, where most of the input gets mapped to only a few codebook vectors, we use k-means clustering-based initialization for the codebook. Specifically, we apply the k-means algorithm on the first training batch and use the centroids as initialization.

Other alternatives for quantization. A simple alternative to generating Semantic IDs is to use Locality Sensitive Hashing (LSH). We perform an ablation study in Subsection 4.2 where we find that RQ-VAE indeed works better than LSH. Another option is to use k-means clustering hierarchically [34], but it loses semantic meaning between different clusters [37]. We also tried VQ-VAE, and while it performs similarly to RQ-VAE for generating the candidates during retrieval, it loses the hierarchical nature of the IDs which confers many new capabilities that are discussed in Section 4.3.

Handling Collisions. Depending on the distribution of semantic embeddings, the choice of codebook size, and the length of codewords, semantic collisions can occur ( i.e. , multiple items can map to the same Semantic ID). To remove the collisions, we append an extra token at the end of the ordered semantic codes to make them unique. For example, if two items share the Semantic ID (12 , 24 , 52) , we append additional tokens to differentiate them, representing the two items as (12 , 24 , 52 , 0) and (12 , 24 , 52 , 1) . To detect collisions, we maintain a lookup table that maps Semantic IDs to corresponding items. Note that collision detection and fixing is done only once after the RQ-V AE model is trained. Furthermore, since Semantic IDs are integer tuples, the lookup table is efficient in terms of storage in comparison to high dimensional embeddings.

3.2 Generative Retrieval with Semantic IDs

We construct item sequences for every user by sorting chronologically the items they have interacted with. Then, given a sequence of the form ( item 1 , . . . , item n ) , the recommender system's task is to predict the next item item n +1 . Wepropose a generative approach that directly predicts the Semantic ID of the next item. Formally, let ( c i, 0 , . . . , c i,m -1 ) be the m -length Semantic ID for item i . Then, we convert the item sequence to the sequence ( c 1 , 0 , . . . , c 1 ,m -1 , c 2 , 0 , . . . , c 2 ,m -1 , . . . , c n, 0 , . . . , c n,m -1 ) . The sequence-to-sequence model is then trained to predict the Semantic ID of item n +1 , which is ( c n +1 , 0 , . . . , c n +1 ,m -1 ) . Given the generative nature of our framework, it is possible that a generated Semantic ID from the decoder does not match an item in the recommendation corpus. However, as we show in appendix (Fig. 6) the probability of such an event occurring is low. We further discuss how such events can be handled in appendix E.

4 Experiments

Datasets. We evaluate the proposed framework on three public real-world benchmarks from the Amazon Product Reviews dataset [10], containing user reviews and item metadata from May 1996 to July 2014. In particular, we use three categories of the Amazon Product Reviews dataset for the sequential recommendation task: 'Beauty', 'Sports and Outdoors', and 'Toys and Games'. We discuss the dataset statistics and pre-processing in Appendix C.

Evaluation Metrics. We use top-k Recall (Recall@K) and Normalized Discounted Cumulative Gain (NDCG@K) with K = 5 , 10 to evaluate the recommendation performance.

RQ-VAE Implementation Details. As discussed in section 3.1, RQ-VAE is used to quantize the semantic embedding of an item. We use the pre-trained Sentence-T5 [27] model to obtain the semantic embedding of each item in the dataset. In particular, we use item's content features such as title, price, brand, and category to construct a sentence, which is then passed to the pre-trained Sentence-T5 model to obtain the item's semantic embedding of 768 dimension.

The RQ-VAE model consists of three components: a DNN encoder that encodes the input semantic embedding into a latent representation, residual quantizer which outputs a quantized representation, and a DNN decoder that decodes the quantized representation back to the semantic input embedding

Hair Treatments

Hair Conditione

Hair Shampoos

Hair Products

Hair Products

Hair Color

Hair Tools

Hair Sets

Hair

SemanticID

(2,**)

(1.)

(3,**)

(0, *, *)

Category

Hair Shampoo & Condi

Bath Bathing Accesso

Hair & Scalp Treatme

Hair Styling Product

Hair Conditioners

Hair Styling Tools

Hair Shampoos

83

Semantic ID

(3, 6, *)

(3. 5. *)

(3,10, *)

(3, 9, *)

(3, 8, *)

(3. 7., *)

(3, 14, *)

(3. 13, *)

(3, 12, *)

(3. 11, *)

(3, 15, *)

Category

Fragrance

Skin Body

三833

28三

(0. 11. )

Semantic ID

(0. 9, *)

(0. 8, *)

(0. 7, *)

(0. 4, *)

(0. 13, *)

(0, 12, *)

(0, 10, *)

(0, 14, *)

(0, 15. *)

Bath Treatments

Skin Body

Hair Color

(3. 2, *)

(3.1. )

(3, 0, *)

(0, 0, *)

Category

Skin Face

Hair Loss Products

Skin Hands & Nails

Makeup Face

Skin Nails

0.00

0.02

0.04

Probability

0.06

0.10

0.12

0.00

0.020.040.060.080.100.120.140.16

Probability

Tools &

Makeup Remover

Bath Cleansers

Makeup Eyes

Makeup Sets

Makeup Lips

Tools Cases

Skin Eyes

Bath

Category

Makeup Remover

Makeup Face

Skin Body

Skin Eyes

Skin Face

Semantic ID

(1. 5. )

(1. 15, *)

(1, 9, *)

(1, 8, *)

(1. 7, *)

(1, 6, *)

(1. 4. *)

(1, 12, *)

(1.11. *)

(1. 10, *)

一

(1, 13, *)

(1. 14, *)

ategory

Bath Scrubs & Body T

Bath Bathing Accesso

Tools Bags & Cases

Skin Hands & Nails

Skin Sets & Kits

Bath Cleansers

Makeup Sets

Makeup Lips

Skin Body

Skin Face

Bath

Semantic ID

(2.8, *)

(2. 9. *)

(2, 10,*)

(2,12,)

(2, 11. *)

(2, 15, *)

(2, 14, *)

(2.13. *)

Skin

0.00

0.02

0.04

0.06

0.08

Makeup Eyes

Skin

0.00

0.01

0.02

0.030.040.050.06

Probability

0.070.08

(1, 0, *)

Tools Cotton & Swabs

Makeup Remover

Makeup Palettes

Makeup Eyes

Makeup Face

0.00

0.02

0.04

Probability

90'0

0.08

(2. 5, *)

(2.2, )

(2, 4. *)

(2, 3., *)

(2, 1., *)

(2, 0, *)

0.10

Probability

space. The encoder has three intermediate layers of size 512, 256 and 128 with ReLU activation, with a final latent representation dimension of 32 . To quantize this representation, three levels of residual quantization is done. For each level, a codebook of cardinality 256 is maintained, where each vector in the codebook has a dimension of 32 . When computing the total loss, we use β = 0 . 25 . The RQ-VAE model is trained for 20k epochs to ensure high codebook usage ( ≥ 80% ). We use Adagrad optimizer with a learning rate of 0.4 and a batch size of 1024. Upon training, we use the learned encoder and the quantization component to generate a 3-tuple Semantic ID for each item. To avoid multiple items being mapped to the same Semantic ID, we add a unique 4 th code for items that share the same first three codewords, i.e. two items associated with a tuple (7, 1, 4) are assigned (7, 1, 4, 0) and (7, 1, 4, 1) respectively (if there are no collisions, we still assign 0 as the fourth codeword). This results in a unique Semantic ID of length 4 for each item in the recommendation corpus.

Sequence-to-Sequence Model Implementation Details. We use the open-sourced T5X framework [28] to implement our transformer based encoder-decoder architecture. To allow the model to process the input for the sequential recommendation task, the vocabulary of the sequence-to-sequence model contains the tokens for each semantic codeword. In particular, the vocabulary contains 1024 ( 256 × 4 ) tokens to represent items in the corpus. In addition to the semantic codewords for items, we add user-specific tokens to the vocabulary. To keep the vocabulary size limited, we only add 2000 tokens for user IDs. We use the Hashing Trick [38] to map the raw user ID to one of the 2000 user ID tokens. We construct the input sequence as the user Id token followed by the sequence of Semantic ID tokens corresponding to a given user's item interaction history. We found that adding user ID to the input, allows the model to personalize the items retrieved.

We use 4 layers each for the transformer-based encoder and decoder models with 6 self-attention heads of dimension 64 in each layer. We used the ReLU activation function for all the layers. The MLP and the input dimension was set as 1024 and 128, respectively. We used a dropout of 0.1. Overall, the model has around 13 million parameters. We train this model for 200k steps for the 'Beauty' and 'Sports and Outdoors' dataset. Due to the smaller size of the 'Toys and Games' dataset, it is trained only for 100k steps. We use a batch size of 256. The learning rate is 0 . 01 for the first 10k steps and then follows an inverse square root decay schedule.

4.1 Performance on Sequential Recommendation

In this section, we compare our proposed framework for generative retrieval with the following sequential recommendation methods (which are described briefly in Appendix B): GRU4Rec [11], Caser [33], HGN [25], SASRec [17], BERT4Rec [32], FDSA [42], S 3 -Rec [44], and P5 [8]. Notably

all the baselines (except P5), learn a high-dimensional vector space using dual encoder, where the user's past item interactions and the candidate items are encoded as a high-dimensional representation and Maximum Inner Product Search (MIPS) is used to retrieve the next candidate item that the user will potentially interact with. In contrast, our novel generative retrieval framework directly predicts the item's Semantic ID token-by-token using a sequence-to-sequence model.

Recommendation Performance. We perform an extensive analysis of our proposed TIGER on the sequential recommendation task and compare against the baselines above. The results for all baselines, except P5, are taken from the publicly accessible results 3 made available by Zhou et al. [44]. For P5, we use the source code made available by the authors. However, for a fair comparison, we updated the data pre-processing method to be consistent with the other baselines and our method. We provide further details related to our changes in Appendix D.

The results are shown in Table 1. We observe that TIGER consistently outperforms the existing baselines 4 . We see significant improvement across all the three benchmarks that we considered. In particular, TIGER performs considerably better on the Beauty benchmark compared to the second-best baseline with up to 29% improvement in NDCG@5 compared to SASRec and 17.3% improvement in Recall@5 compared to S 3 -Rec. Similarly on the Toys and Games dataset, TIGER is 21% and 15% better in NDCG@5 and NDCG@10, respectively.

4.2 Item Representation

In this section, we analyze several important characteristics of RQ-V AE Semantic IDs. In particular, we first perform a qualitative analysis to observe the hierarchical nature of Semantic IDs. Next, we evaluate the importance of our design choice of using RQ-VAE for quantization by contrasting the performance with an alternative hashing-based quantization method. Finally, we perform an ablation to study the importance of using Semantic IDs by comparing TIGER with a sequence-to-sequence model that uses Random ID for item representation.

Qualitative Analysis. We analyze the RQ-VAE Semantic IDs learned for the Amazon Beauty dataset in Figure 4. For exposition, we set the number of RQ-VAE levels as 3 with a codebook size of 4, 16, and 256 respectively, i.e. for a given Semantic ID ( c 1 , c 2 , c 3 ) of an item, 0 ≤ c 1 ≤ 3 , 0 ≤ c 2 ≤ 15 and 0 ≤ c 3 ≤ 255 . In Figure 4a, we annotate each item's category using c 1 to visualize c 1 -specific categories in the overall category distribution of the dataset. As shown in Figure 4a, c 1 captures the high-level category of the item. For instance, c 1 = 3 contains most of the products related to 'Hair'. Similarly, majority of items with c 1 = 1 are 'Makeup' and 'Skin' products for face, lips and eyes.

We also visualize the hierarchical nature of RQ-VAE Semantic IDs by fixing c 1 and visualizing the category distribution for all possible values of c 2 in Fig. 4b. We again found that the second codeword c 2 further categorizes the high-level semantics captured with c 1 into fine-grained categories. The hierarchical nature of Semantic IDs learned by RQ-VAE opens a wide-array of new capabilities which are discussed in Section 4.3. As opposed to existing recommendation systems that learn item embeddings based on random atomic IDs, TIGER uses Semantic IDs where semantically similar

0.14

Model

Ours

0.12

Semantic_KNN

ItemSplit

0.10

Overall

1

?

0.08

Unseen

call

0.06

0.04

0.02

0.00

0.0

0.2

0.4

0.6

0.8

1.0

Epsilon(ε)

0.12

0.10

K

0.08

0.06

Model

0.04

Semantic_KNN

0.02

Overall

一-

Unseen

0.00

0

10

20

30

40

50

K

glyph[epsilon1]

(a) Recall@K vs. K, (

= 0

.

1

).

(b) Recall@10 vs. glyph[epsilon1] .

items have overlapping codewords, which allows the model to effectively share knowledge from semantically similar items in the dataset.

Hashing vs. RQ-VAE Semantic IDs. We study the importance of RQ-VAE in our framework by comparing RQ-VAE against Locality Sensitive Hashing (LSH) [14, 13, 2] for Semantic ID generation. LSH is a popular hashing technique that can be easily adapted to work for our setting. To generate LSH Semantic IDs, we use h random hyperplanes w 1 , . . . , w h to perform a random projection of the embedding vector x and compute the following binary vector: (1 w glyph[latticetop] 1 x > 0 , . . . , 1 w glyph[latticetop] h x > 0 ) . This vector is converted into an integer code as c 0 = ∑ h i =1 2 i -1 1 w glyph[latticetop] i x > 0 . This process is repeated m times using an independent set of random hyperplanes, resulting in m codewords ( c 0 , c 1 , . . . , c m -1 ) , which we refer to as the LSH Semantic ID.

In Table 2, we compare the performance of LSH Semantic ID with our proposed RQ-VAE Semantic ID. In this experiment, for LSH Semantic IDs, we used h = 8 random hyperplanes and set m = 4 to ensure comparable cardinality with the RQ-VAE. The parameters for the hyperplanes are randomly sampled from a standard normal distribution, which ensures that the hyperplanes are spherically symmetric. Our results show that RQ-VAE consistently outperforms LSH. This illustrates that learning Semantic IDs via a non-linear, Deep Neural Network (DNN) architecture yields better quantization than using random projections, given the same content-based semantic embedding.

Random ID vs. Semantic ID. We also compare the importance of Semantic IDs in our generative retrieval recommender system. In particular, we compare randomly generated IDs with the Semantic IDs. To generate the Random ID baseline, we assign m random codewords to each item. A Random ID of length m for an item is simply ( c 1 , . . . , c m ) , where c i is sampled uniformly at random from { 1 , 2 , . . . , K } . We set m = 4 , and K = 255 for the Random ID baseline to make the cardinality similar to RQ-VAE Semantic IDs. A comparison of Random ID against RQ-VAE and LSH Semantic IDs is shown in Table 2. We see that Semantic IDs consistently outperform Random ID baseline, highlighting the importance of leveraging content-based semantic information.

4.3 New Capabilities

We describe two new capabilities that directly follow from our proposed generative retrieval framework, namely cold-start recommendations and recommendation diversity. We refer to these capabilities as 'new' since existing sequential recommendation models (See the baselines in section 4.1) cannot be directly used to satisfy these real-world use cases. These capabilities result from a synergy between RQ-VAE based Semantic IDs and the generative retrieval approach of our framework. We discuss how TIGER is used in these settings in the following sections.

Cold-Start Recommendation. In this section, we study the cold-start recommendation capability of our proposed framework. Due to the fast-changing nature of the real-world recommendation corpus, new items are constantly introduced. Since newly added items lack user impressions in the training corpus, existing recommendation models that use a random atomic ID for item representation fail to retrieve new items as potential candidates. In contrast, the TIGER framework can easily perform cold-start recommendations since it leverages item semantics when predicting the next item.

For this analysis, we consider the Beauty dataset from Amazon Reviews. To simulate newly added items, we remove 5% of test items from the training data split. We refer to these removed items as unseen items . Removing the items from the training split ensures there is no data leakage with respect to the unseen items. As before, we use Semantic ID of length 4 to represent the items, where the first 3 tokens are generated using RQ-VAE and the 4 th token is used to ensure a unique ID exists for all the seen items. We train the RQ-VAE quantizer and the sequence-to-sequence model on the training split. Once trained, we use the RQ-VAE model to generate the Semantic IDs for all the items in the dataset, including any unseen items in the item corpus.

Given a Semantic ID ( c 1 , c 2 , c 3 , c 4 ) predicted by the model, we retrieve the seen item having the same corresponding ID. Note that by definition, each Semantic ID predicted by the model can match at most one item in the training dataset. Additionally, unseen items having the same first three semantic tokens, i.e. ( c 1 , c 2 , c 3 ) are included to the list of retrieved candidates. Finally, when retrieving a set of top-K candidates, we introduce a hyperparameter glyph[epsilon1] which specifies the maximum proportion of unseen items chosen by our framework.

We compare the performance of TIGER with the k-Nearest Neighbors (KNN) approach on the cold-start recommendation setting in Fig. 5. For KNN, we use the semantic representation space to perform the nearest-neighbor search. We refer to the KNN-based baseline as Semantic_KNN. Fig. 5a shows that our framework with glyph[epsilon1] = 0 . 1 consistently outperforms Semantic_KNN for all Recall@K metrics. In Fig. 5b, we provide a comparison between our method and Seman-tic_KNN for various values of glyph[epsilon1] . For all settings of glyph[epsilon1] ≥ 0 . 1 , our method outperforms the baseline.

Recommendation diversity. While Recall and NDCG are the primary metrics used to evaluate a recommendation system, diversity of predictions is another critical objective of interest. A recommender system with poor diversity can be detrimental to the long-term engagement of users. Here, we discuss how our generative retrieval framework can be used

to predict diverse items. We show that temperature-based sampling during the decoding process can be effectively used to control the diversity of model predictions. While temperature-based sampling can be applied to any existing recommendation model, TIGER allows sampling across various levels of hierarchy owing to the properties of RQ-V AE Semantic IDs. For instance, sampling the first token of the Semantic ID allows retrieving items from coarse-level categories, while sampling a token from second/third token allows sampling items within a category.

We quantitatively measure the diversity of predictions using Entropy@K metric, where the entropy is calculated for the distribution of the ground-truth categories of the top-K items predicted by the model. We report the Entropy@K for various temperature values in Table 3. We observe that temperature-sampling in the decoding stage can be effectively used to increase the diversity in the ground-truth categories of the items. We also perform a qualitative analysis in Table 4.

4.4 Ablation Study

We measure the effect of varying the number of layers in the sequence-to-sequence model in Table 5. We see that the metrics improve slightly as we make the network bigger. We also measure the effect of providing user information, the results for which are provided in Table 8 in the Appendix.

4.5 Invalid IDs

Since the model decodes the codewords of the target Semantic ID autoregressively, it is possible that the model may predict invalid IDs (i.e., IDs that do not map to any item in the recommendation dataset). In our experiments, we used semantic IDs of length 4 with each codeword having a cardinality of 256 (i.e., codebook size = 256 for each level). The number of possible IDs spanned by this combination = 256 4 , which is approximately 4 trillion. On the other hand, the number of items in the datasets we consider is 10K-20K (See Table 6). Even though the number of valid IDs is only a fraction of all complete ID space, we observe that the model almost always predicts valid IDs. We visualize the fraction of invalid IDs produced by TIGER as a function of the number of retrieved items K in Figure 6. For top-10 predictions, the fraction of invalid IDs varies from ∼ 0 . 1% -1 . 6% for the three datasets. To counter the effect of invalid IDs and to always get top-10 valid IDs, we can increase the beam size and filter the invalid IDs.

It is important to note that, despite generating invalid IDs, TIGER achieves state-of-the-art performance when compared to other popular methods used for sequential recommendations. One extension to handle invalid tokens could be to do prefix matching when invalid tokens are generated by the model. Prefix matching of Semantic IDs would allow retrieving items that have similar semantic meaning as the tokens generated by the model. Given the hierarchical nature of our RQ-V AE tokens, prefix matching can be thought of as model predicting item category as opposed to the item index. Note that such an extension could improve the recall/NDCG metrics even further. We leave such an extension as a future work.

5 Conclusion

This paper proposes a novel paradigm, called TIGER, to retrieve candidates in recommender systems using a generative model. Underpinning this method is a novel semantic ID representation for items, which uses a hierarchical quantizer (RQ-VAE) on content embeddings to generate tokens that form the semantic IDs. Our framework provides results in a model that can be used to train and serve without creating an index - the transformer memory acts as a semantic index for items. We note that the cardinality of our embedding table does not grow linearly with the cardinality of item space, which works in our favor compared to systems that need to create large embedding tables during training or generate an index for every single item. Through experiments on three datasets, we show that our model can achieve SOTA retrieval performance, while generalizing to new and unseen items.

References

Himan Abdollahpouri, Masoud Mansoury, Robin Burke, and Bamshad Mobasher. The unfairness of popularity bias in recommendation. arXiv preprint arXiv:1907.13286 , 2019.

Moses S Charikar. Similarity estimation techniques from rounding algorithms. In Proceedings of the thiry-fourth annual ACM symposium on Theory of computing , pages 380-388, 2002.

Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for

recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems , pages 7-10, 2016.

Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems , pages 191-198, 2016.

Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. Autoregressive entity retrieval. arXiv preprint arXiv:2010.00904 , 2020.

Gabriel de Souza Pereira Moreira, Sara Rabhi, Jeong Min Lee, Ronay Ak, and Even Oldridge. Transformers4rec: Bridging the gap between nlp and sequential/session-based recommendation. In Fifteenth ACM Conference on Recommender Systems , pages 143-153, 2021.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). arXiv preprint arXiv:2203.13366 , 2022.

Carlos A Gomez-Uribe and Neil Hunt. The netflix recommender system: Algorithms, business value, and innovation. ACM Transactions on Management Information Systems (TMIS) , 6(4):119, 2015.

Ruining He and Julian McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web , pages 507-517, 2016.

Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-based recommendations with recurrent neural networks. arXiv preprint arXiv:1511.06939 , 2015.

Yupeng Hou, Zhankui He, Julian McAuley, and Wayne Xin Zhao. Learning vector-quantized item representation for transferable sequential recommenders. arXiv preprint arXiv:2210.12316 , 2022.

Piotr Indyk and Rajeev Motwani. Approximate nearest neighbors: towards removing the curse of dimensionality. In Proceedings of the thirtieth annual ACM symposium on Theory of computing , pages 604-613, 1998.

Piotr Indyk, Rajeev Motwani, Prabhakar Raghavan, and Santosh Vempala. Locality-preserving hashing in multidimensional spaces. In Proceedings of the twenty-ninth annual ACM symposium on Theory of computing , pages 618-625, 1997.

Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence , 33(1):117-128, 2010.

Wang-Cheng Kang, Derek Zhiyuan Cheng, Tiansheng Yao, Xinyang Yi, Ting Chen, Lichan Hong, and Ed H Chi. Learning to embed categorical features without embedding tables for recommendation. arXiv preprint arXiv:2010.10784 , 2020.

Wang-Cheng Kang and Julian McAuley. Self-attentive sequential recommendation. In 2018 IEEE international conference on data mining (ICDM) , pages 197-206. IEEE, 2018.

Dongmoon Kim, Kun-su Kim, Kyo-Hyun Park, Jee-Hyong Lee, and Keon Myung Lee. A music recommendation system with a dynamic k-means clustering algorithm. In Sixth international conference on machine learning and applications (ICMLA 2007) , pages 399-403. IEEE, 2007.

Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer , 42(8):30-37, 2009.

Tim Kraska, Alex Beutel, Ed H Chi, Jeffrey Dean, and Neoklis Polyzotis. The case for learned index structures. In Proceedings of the 2018 international conference on management of data , pages 489-504, 2018.

Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11523-11532, 2022.

Hyunji Lee, Jaeyoung Kim, Hoyeon Chang, Hanseok Oh, Sohee Yang, Vlad Karpukhin, Yi Lu, and Minjoon Seo. Contextualized generative retrieval. arXiv preprint arXiv:2210.02068 , 2022.

Hyunji Lee, Sohee Yang, Hanseok Oh, and Minjoon Seo. Generative retrieval for long sequences. arXiv preprint arXiv:2204.13596 , 2022.

Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. Neural attentive session-based recommendation. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management , pages 1419-1428, 2017.

Chen Ma, Peng Kang, and Xue Liu. Hierarchical gating networks for sequential recommendation. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pages 825-833, 2019.

Masoud Mansoury, Himan Abdollahpouri, Mykola Pechenizkiy, Bamshad Mobasher, and Robin Burke. Feedback loop and bias amplification in recommender systems, 2020.

Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Findings of the Association for Computational Linguistics: ACL 2022 , pages 1864-1874, Dublin, Ireland, May 2022. Association for Computational Linguistics.

Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. Scaling up models and data with t5x and seqio . arXiv preprint arXiv:2203.17189 , 2022.

Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909 , 2015.

Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics.

Anima Singh, Trung Vu, Raghunandan Keshavan, Nikhil Mehta, Xinyang Yi, Lichan Hong, Lukasz Heldt, Li Wei, Ed Chi, and Maheswaran Sathiamoorthy. Better generalization with semantic ids: A case study in ranking for recommendations. arXiv preprint arXiv:2306.08121 , 2023.

Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management , pages 1441-1450, 2019.

Jiaxi Tang and Ke Wang. Personalized top-n sequential recommendation via convolutional sequence embedding. In Proceedings of the eleventh ACM international conference on web search and data mining , pages 565-573, 2018.

Yi Tay, Vinh Q Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, et al. Transformer memory as a differentiable search index. arXiv preprint arXiv:2202.06991 , 2022.

Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems , 30, 2017.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Yujing Wang, Yingyan Hou, Haonan Wang, Ziming Miao, Shibin Wu, Hao Sun, Qi Chen, Yuqing Xia, Chengmin Chi, Guoshuai Zhao, et al. A neural corpus indexer for document retrieval. arXiv preprint arXiv:2206.02743 , 2022.

Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola, and Josh Attenberg. Feature hashing for large scale multitask learning. In Proceedings of the 26th annual international conference on machine learning , pages 1113-1120, 2009.

Xinyang Yi, Ji Yang, Lichan Hong, Derek Zhiyuan Cheng, Lukasz Heldt, Aditee Kumthekar, Zhe Zhao, Li Wei, and Ed Chi. Sampling-bias-corrected neural modeling for large corpus item recommendations. In Proceedings of the 13th ACM Conference on Recommender Systems , pages 269-277, 2019.

Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. CoRR , abs/2107.03312, 2021.

Shuai Zhang, Yi Tay, Lina Yao, and Aixin Sun. Next item recommendation with self-attention. arXiv preprint arXiv:1808.06414 , 2018.

Tingting Zhang, Pengpeng Zhao, Yanchi Liu, Victor S Sheng, Jiajie Xu, Deqing Wang, Guanfeng Liu, and Xiaofang Zhou. Feature-level deeper self-attention network for sequential recommendation. In IJCAI , pages 4320-4326, 2019.

Zhe Zhao, Lichan Hong, Li Wei, Jilin Chen, Aniruddh Nath, Shawn Andrews, Aditee Kumthekar, Maheswaran Sathiamoorthy, Xinyang Yi, and Ed Chi. Recommending what video to watch next: a multitask ranking system. In Proceedings of the 13th ACM Conference on Recommender Systems , pages 43-51, 2019.

Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, and Ji-Rong Wen. S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management , pages 1893-1902, 2020.

A Related Work (cont.)

Generative Retrieval Document retrieval traditionally involved training a 2-tower model which mapped both queries and documents to the same high-dimensional vector space, followed by performing an ANN or MIPS for the query over all the documents to return the closest ones. This technique presents some disadvantages like having a large embedding table [22, 23]. Generative retrieval is a recently proposed technique that aims to fix some of the issues of the traditional approach by producing token by token either the title, name, or the document id string of the document. Cao et al. [5] proposed GENRE for entity retrieval, which used a transformer-based architecture to return, token-by-token, the name of the entity referenced to in a given query. Tay et al. [34] proposed DSI for document retrieval, which was the first system to assign structured semantic DocIDs to each document. Then given a query, the model autoregressively returned the DocID of the document token-by-token. The DSI work marks a paradigm shift in IR to generative retrieval approaches and is the first successful application of an end-to-end Transformer for retrieval applications. Subsequently, Lee et al. [23] show that generative document retrieval is useful even in the multi-hop setting, where a complex query cannot be answered directly by a single document, and hence their model generates intermediate queries, in a chain-of-thought manner, to ultimately generate the output for the complex query. Wang et al. [37] supplement the hierarchical k -means clustering based semantic DocIDs of Tay et al. [34] by proposing a new decoder architecture that specifically takes into account the prefixes in semantic DocIDs. In CGR [22], the authors propose a way to take advantage of both the bi-encoder technique and the generative retrieval technique, by allowing the decoder, of their encoder-decoder-based model, to learn separate contextualized embeddings which store information about documents intrinsically. To the best of our knowledge, we are the first to use generative Semantic IDs created using an auto-encoder (RQ-VAE [40, 21]) for retrieval models.

Vector Quantization. We refer to Vector Quantization as the process of converting a highdimensional vector into a low-dimensional tuple of codewords. One of the most straightforward techniques uses hierarchical clustering, such as the one used in [34], where clusters created in a particular iteration are further partitioned into sub-clusters in the next iteration. An alternative popular approach is Vector-Quantized Variational AutoEncoder (VQ-VAE), which was introduced in [35] as a way to encode natural images into a sequence of codes. The technique works by first passing the input vector (or image) through an encoder that reduces the dimensionality. The smaller dimensional vector is partitioned and each partition is quantized separately, thus resulting in a sequence of codes: one code per partition. These codes are then used by a decoder to recreate the original vector (or image).

RQ-VAE [40, 21] applies residual quantization to the output of the encoder of VQ-VAE to achieve a lower reconstruction error. We discuss this technique in more detail in Subsection 3.1. Locality Sensitive Hashing (LSH) [14, 13] is a popular technique used for clustering and approximate nearest neighbor search. The particular version that we use in this paper for clustering is SimHash [2], which uses random hyperplanes to create binary vectors which serve as hashes of the items. Because it has low computational complexity and is scalable [13], we use this as a baseline technique for vector quantization.

B Baselines

Below we briefly describe each of the baselines with which we compare TIGER

GRU4Rec [11] is the first RNN-based approach that uses a customized GRU for the sequential recommendation task.

Caser [33] uses a CNN architecture for capturing high-order Markov Chains by applying horizontal and vertical convolutional operations for sequential recommendation.

HGN [25]: Hierarchical Gating Network captures the long-term and short-term user interests via a new gating architecture.

SASRec [17]: Self-Attentive Sequential Recommendation uses a causal mask Transformer to model a user's sequential interactions.

BERT4Rec [32]: BERT4Rec addresses the limitations of uni-directional architectures by using a bi-directional self-attention Transformer for the recommendation task.

FDSA [42]: Feature-level Deeper Self-Attention Network incorporates item features in addition to the item embeddings as part of the input sequence in the Transformers.

S 3 -Rec [44]: Self-Supervised Learning for Sequential Recommendation proposes pre-training a bi-directional Transformer on self-supervision tasks to improve the sequential recommendation.

P5 [8]: P5 is a recent method that uses a pretrained Large Language Model (LLM) to unify different recommendation tasks in a single model.

C Dataset Statistics

We use three public benchmarks from the Amazon Product Reviews dataset [10], containing user reviews and item metadata from May 1996 to July 2014. We use three categories of the Amazon Product Reviews dataset for the sequential recommendation task: 'Beauty', 'Sports and Outdoors', and 'Toys and Games'. Table 6 summarizes the statistics of the datasets. We use users' review history to create item sequences sorted by timestamp and filter out users with less than 5 reviews. Following the standard evaluation protocol [17, 8], we use the leave-one-out strategy for evaluation. For each item sequence, the last item is used for testing, the item before the last is used for validation, and the rest is used for training. During training, we limit the number of items in a user's history to 20.

D Modifications to the P5 data preprocessing

The P5 source code 5 pre-processes the Amazon dataset to first create sessions for each user, containing chronologically ordered list of items that the user reviewed. After creating these sessions, the original item IDs from the dataset are remapped to integers 1 , 2 , 3 , . . . 6 . Hence, the first item in the first session gets an id of '1', the second item, if not seen before, gets an id of '2', and so on. Notably, this pre-processing scheme is applied before creating training and testing splits. This results in the creation of a sequential dataset where many sequences are of the form a, a +1 , a +2 , . . . . Given that P5 uses Sentence Piece tokenizer [30] (See Section 4.1 in [8]), the test and train items in a user session may share the sub-word and can lead to information leakage during inference.

To resolve the leakage issue, instead of assigning sequentially increasing integer ids to items, we assigned random integer IDs, and then created splits for training and evaluation. The rest of the code for P5 was kept identical to the source code provided in the paper. The results for this dataset are reported in Table 7 as the row 'P5'. We also implemented a version of P5 ourselves from scratch, and train the model on only sequential recommendation task. The results for our implementation are depicted as 'P5-ours'. We were also able to verify in our P5 implementation that using consecutive integer sequences for the item IDs helped us get equivalent or better metrics than those reported in P5.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

K

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Invalid IDs (%)

(a) Sports and Outdoors

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

K

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Invalid IDs (%)

(b) Beauty

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

K

0

1

2

3

4

5

6

Invalid IDs (%)

(c) Toys and Games

Figure 6: Percentage of invalid IDs when generating Semantic IDs using Beam search for various values of K . As shown, ∼ 0 . 3% -6% of the IDs are invalid when retrieving the top-20 items.

E Discussion

Effects of Semantic ID length and codebook size. We tried varying the Semantic ID length and codebook size, such as having an ID consisting of 6 codewords each from a codebook of size 64 . We noticed that the recommendation metrics for TIGER were robust to these changes. However, note that the input sequence length increases with longer IDs (i.e., more codewords per item ID), which makes the computation more expensive for our transformer-based sequence-to-sequence model.

Scalability. To test the scalability of Semantic IDs, we ran the following experiment: We combined all the three datasets and generated Semantic IDs for the entire set of items from the three datasets. Then, we used these Semantic IDs for the recommendation task on the Beauty dataset. We compare the results from this experiment with the original experiment where the Semantic IDs are generated only from the Beauty dataset. The results are provided in Table 10. We see that there is only a small decrease in performance here.

Inference cost. Despite the remarkable success of our model on the sequential recommendation task, we note that our model can be more computationally expensive than ANN-based models during inference due to the use of beam search for autoregressive decoding. We emphasize that optimizing the computational efficiency of TIGER was not the main objective of this work. Instead, our work opens up a new area of research: Recommender Systems based on Generative Retrieval . As part of future work, we will consider ways to make the model smaller or explore other ways of improving the inference efficiency.

Memory cost of lookup tables. We maintain two lookup hash tables for TIGER: an Item ID to Semantic ID table and a Semantic ID to Item ID table. Note that both of these tables are generated only once and then frozen: they are generated after the training of the RQ-VAE-based Semantic ID generation model, and after that, they are frozen for the training of the sequence-to-sequence

transformer model. Each Semantic ID consists of a tuple of 4 integers, each of which are stored in 8 bits, hence totalling to 32 bits per item. Each item is represented by an Item ID, stored as a 32 bit integer. Thus, the size of each lookup table will be of the order of 64 N bits, where N is the number of items in the dataset.

Memory cost of embedding tables . TIGER uses much smaller embedding tables compared to traditional recommender systems. This is because where traditional recommender systems store an embedding for each item, TIGER only stores an embedding for each semantic codeword. In our experiments, we used 4 codewords each of cardinality 256 for Semantic ID representation, resulting in 1024 (256 × 4) embeddings. For traditional recommender systems, the number of embeddings is N , where N is the number of items in the dataset. In our experiments, N ranged from 10K to 20K depending on the dataset. Hence, the memory cost of TIGER's embedding table is 1024 d , where d is the dimension of the embedding, whereas the memory cost for embedding lookup tables in traditional recommendation systems is Nd .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a generative retrieval approach wherein the model autoregressively decodes the identifiers, called Semantic ID, for the target candidate. They propose a method to generate the semantic ID for items which is a tuple of codewords which are trained from the embeddings of the text component of the items obtained using a standard sentence BERT type of model. Finally, a seq2seq model is trained to predict the semantic ids for the next item given the current item. The authors show that the proposed technique outperforms SOTA techniques on some retrieval datasets. They also claim that the semantic ID trick enhances the generalization property of the seq2seq model especially for cold start cases.\n\nSTRENGTHS:\nOriginality: The paper seems to present a novel idea which is original enough. The items are represented in a novel semantic ID sequence form which is then used in a standard seq2seq framework.\n\nClarity: I think the paper is clear enough to understand and reproduce.\n\nWEAKNESSES:\nThe method itself is and its larger claim of generative in nature is the primary concern I have with the paper. The proposed method tries to generate semantic id sequence for items already seen by the system. Any novel combination of semantic ID will lead to an item which is not part of the retrieval system. This leads to the authors constraining the semantic IDs to just 3 code words and then an ad hoc 4th code word to introduce uniqueness in the codes. This part seems to suit small, well controlled item sets, but the scaling out of such a system to billions of items seems not clearly explained or experimented with. \n\nThe experimental results seem not consistent with the published paper, especially for P5 paper. The numbers reported in the paper are higher than this paper which are not shown as is, but the authors implement their own P5 and compare results which are way different than the published results.\n\nThe other comparative results are all copied over from the benchmark mentioned in the footnote ttps://github.com/aHuiWang/CIKM2020-S3Rec.\n\nQUESTIONS:\nI would like to see large code sets for items. This would make the system truly scalable. Otherwise, constraining embeddings to 3 codewords seems a step in the opposite direction which will work with very small item sets in controlled environments. \n\nI would also ask the authors to perform comparative experiments with P5, where the parameters do not need to be comparable, but the best performance obtained should be the primary aim.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

comments: SUMMARY:
The paper is trying to approach sequential recommendation task using semantic IDs. By incorporating semantic IDs, the paper shows that the retrieval performance can be improved.

STRENGTHS:
1. The ablation study conducted is good. For example, it compares RQ-VAE against LSH, and random.
2. The proposed method and framework is intuitive and makes sense.
3. The conducted experiment seems extensive and the proposed method is shown to outperform than non-semantic-id method like S3-Rec.

WEAKNESSES:
1. The paper title seems to emphasize on Generative Retrieval rather than on ranking, if that is the case, in the experiment section, shouldn't we care more about recall than NDCG? Why proposed method seems to improve NDCG a lot but much less so on recall metrics?
2. The novelty of the proposed method seems limited, the main idea seems to overlap with paper "Learning Vector-Quantized Item Representation for Transferable Sequential Recommenders", and probably should be considered as incremental value added to the previous published paper. Although the method emphasized on generative AI, it is still generative on the item semantic ID level, it would be more significant if the generative mechanism is on the word tokens, in my opinion.

QUESTIONS:
Besides the questions in the weakness section above, another question is 
1. Semantic ID idea seems useful, but would it be nicer if we do not need to use semantic embeddings as middle layer and directly do generative AI on word tokens?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes TIGER, a generative retrieval approach to sequential recommendation. More specifically, TIGER proposes to use semantic IDs of a product (based off RQ-VAE quantization of sentence-T5 product feature embeddings) instead of typically used atomic item-IDs in the sequential recommendation pipeline. Empirical analysis on 3 Amazon subsets show promising results compared to itemID-only and ID+text based existing recommenders.\n\nSTRENGTHS:\n- Interesting design to use quantized embeddings of items (semantic IDs)\n- Successful application (i.e., outperforming atomic ID-based recommenders) of the generative retrieval paradigm for sequential recommendation\n- State of the art results on medium-sized academic datasets\n\nWEAKNESSES:\n- No ablations, e.g., effect of number of codebooks, prepending user-ID, codebook size, Transformer architecture, etc.\n- No significance tests while reporting ranking results\n\nQUESTIONS:\n- It seems the text+ID based recommenders are not consistent in what product features are being used. E.g., S3-Rec seems to use category and brand information whereas FDSA seems to use category, brand, and description. Is the improvement in TIGER just because of using extra feature signals like title and price?\n- Unclear which two semantic IDs are “closer”? I had to assume that since a residual quantization is being performed, being close to the first quanta holds more importance than the last.\n\nLIMITATIONS:\nI will use this space to provide some (hopefully constructive) suggestions for this paper:\n\n- Best to outline in Table 1 that S3-Rec and FDSA use item-text features whereas all others are ID-only recommenders.\n- It would be interesting to see the performance on the unprocessed Amazon datasets since arbitrary preprocessing of user-data has been noted to interfere with true performance [1].\n- It would be interesting to see a TIGER variant where only three-length semantic IDs are used (i.e., non-unique items would be allowed) as it would become a text-only recommender.\n- Complementing the zero-shot analysis in the paper with an analysis of TIGER’s performance over the user/item coldness spectrum (in comparison with other models)\n- Although released after this submission to NeurIPS, [2] might be a good reference to compare TIGER with (this comparison does not affect my review).\n\n[1] Sachdeva, N., Wu, C. J., & McAuley, J. On sampling collaborative filtering datasets. WSDM ‘22.\n\n[2] Li, J., Wang, M., Li, J., Fu, J., Shen, X., Shang, J., & McAuley, J. Text Is All You Need: Learning Language Representations for Sequential Recommendation. arXiv preprint arXiv:2305.13731.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes a generative retrieval for an efficient recommendation, where the retrieval model autoregressively decodes the identifiers of the target candidates. The evaluation results show the proposed method can perform better than competing baselines on various dataset.

STRENGTHS:
1.  Generative retrieval-based recommender model with item features
2.  Extensive evaluation show the superiority of the proposed algorithm

WEAKNESSES:
1. Limited contributions. This paper claims the first to propose generative retrieval for recommendation systems using Semantic ID representation of items. However, this is not true. RQ-VAE is also simply used for obtaining the semantic ids, and no improvements are made in the paper.

2. The similar idea is mainly used for efficient recommendation, but no efficiency study is performed. 

3. The settings of some baselines are not reported. I know that SASRec could be significantly improved using some tricks. It is not clear whether these tricks are used in the experiments. 

4. The comparison is not fair. The proposed algorithm contains side information, but some baselines do not take them into account. 

5. The experiments are only performed on the Amazon datasets. It is not sufficient to prove the effectiveness of the proposed algorithms.

QUESTIONS:
see the attachments.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces TIGER (Transformer Index for Generative Recommenders), a novel generative retrieval model for sequential recommender systems. This innovative approach involves generating Semantic IDs—sequences of semantic tokens derived from an item's text features. A hierarchical quantizer (RQ-VAE) is employed on content embeddings to formulate these tokens, thereby allowing the model to predict the Semantic IDs for generative retrieval. The authors claim that TIGER outperforms current state-of-the-art models, while offering a new solution for cold-start recommendations and increasing item diversity by adjusting the temperature parameters. Furthermore, it showcases impressive scalability and the capacity to generalize to novel and unseen items.\n\nSTRENGTHS:\n1. This paper is well-written and technically solidity, effectively combining novel concepts such as Differentiable Search Index (DSI) using Transformer memory and a hierarchical quantizer, RQ-VAE, for generative retrieval in sequential recommendation task.\n\n2. The idea driving the work is both logical and compelling, providing a fresh perspective on generative retrieval tasks in the context of recommender systems.\n\n3. Empirical results show that TIGER not only outperforms the SOTA baselines on the SR task but also exhibits an ability to generalize to unseen items and enhance item recommendation diversity.\n\nWEAKNESSES:\n1. The paper could benefit from more detailed information about the text encoder's input, given that quantization is a crucial part of Semantic ID generation. It's unclear whether all components depicted in Figure 2(a) are utilized. Additionally, handling missing information/data in datasets, such as the Amazon Beauty dataset, needs addressing (E.g., in the Amazon Beauty dataset, approximately 50% of the brand information is missing.). The authors should explain how they preprocessed the data. \n\n2. The paper's treatment of related work and baseline comparisons raises some concerns. Specifically, the performance results of the P5 model appear notably lower than what's presented in its original paper. The authors have explained the data reprocessing using a random item ID, but a comparison using the more updated benchmark in [2] - which also uses a random item ID - could offer a more balanced analysis. It would be insightful to understand the reasons behind the observed performance degradation of the P5 model in authors’ setting.\n\n3. The proposed solution to collision problems, appending an extra token at the end of the ordered semantic codes, raises questions about scalability. For instance, if ten items share the Semantic ID (12,24,52), would the model need to add two extra tokens and maintain a new lookup table?\n\n4. The experimental design for cold-start recommendations in section 4.3 could be revisited. The removal of 5% of test items from the training data split might change the original training sequences, potentially leading to shorter sequences and artificially enhancing recall results in the cold-start retrieval setting.\n\n5. A comparison with additional ID encoding methods, similar to those presented in [2], could strengthen the argument for RQ-VAE's superiority and effectiveness.\n\n6. While the authors state that inference cost is not their main contribution, a rough comparison of TIGER's inference cost to LSH and NN-based methods would help to understand this method's trade-offs.\n\n\nReferences.\n[1] OpenP5: Benchmarking Foundation Models for Recommendation(https://arxiv.org/pdf/2306.11134.pdf)\n[2] How to Index Item IDs for Recommendation Foundation Models. (https://arxiv.org/abs/2305.06569)\n\nQUESTIONS:\nPlease refer to the weaknesses section.\n\nLIMITATIONS:\nPlease refer to the weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}