Towards Scalable Semantic Representation for Recommendation

## Abstract

Abstract With recent advances in large language models (LLMs), there has been emerging numbers of research in developing Semantic IDs based on LLMs to enhance the performance of recommendation systems.
However, the dimension of these embeddings needs to match that of the ID embedding in recommendation, which is usually much smaller than the original length.
Such dimension compression results in inevitable losses in discriminability and dimension robustness of the LLM embeddings, which motivates us to scale up the semantic representation.
In this paper, we propose Mixture-of-Codes, which first constructs multiple independent codebooks for LLM representation in the indexing stage , and then utilizes the Semantic Representation along with a fusion module for the downstream recommendation stage .
Extensive analysis and experiments demonstrate that our method achieves superior discriminability and dimension robustness scalability, leading to the best scale-up performance in recommendations.

## 1 Introduction

Recently, the emergence of large language models (LLM) (Dubey et al., ; Achiam et al., ) sheds light in improving the recommendation systems via the semantic knowledge from LLM (Hou et al., ; Bao et al., ).
An intuitive practice is to simply project the LLM embeddings to low-dimension embeddings via only MLPs into the recommendation systems for feature interactions. However, such application
is ineffective, largely due to the massive semantic gap between the embedding spaces of LLM and recommendation systems (Lin et al., ; Pan et al., ).

Several works (Rajput et al., ; Singh et al., ) have proposed to derive Semantic IDs (*i.e.*, codes) based on clustering methods such as VQ-VAE (Van Den Oord et al., ) or RQ-VAE (Lee et al., ) to capture information from the LLM embedding. In particular, they first train an auto-encoder with discrete codes and then apply these codes to downstream tasks such as retrieval or ranking. Such methods aim to transfer knowledge from the LLM embedding space to recommendation systems, utilizing the codes to capture the local structure of the original space. Besides, embedding these codes in the downstream stage facilitates the effective training in an end-to-end manner.

Notably, the LLM embeddings usually have very large dimensions, ranging from 4,096 to 16,384 (Dubey et al., ).
When generating the embeddings for these codes, their dimension needs to match that of the recommendation IDs.
However, the dimension in recommendation is usually small due to the *Interaction Collapse Theory* (Guo et al., ).
Therefore, the code embeddings are also only able to span a low-dimension space.
With one single semantic embedding as the semantic representation, it may fail to capture the complex, high-dimensional structure of the original LLM embeddings, *lead to inevitable information loss and performance deterioration* during the knowledge transfer.
This motivates us to study how to *scale up the semantic representation effectively*.

We delve into two approaches based on existing works, including Multi-Embedding (Guo et al., ) and RQ-VAE (Lee et al., ), to scale up the semantic representation by either using one codebook with multiple embeddings, or multiple hierarchical codebooks and embeddings.
Nevertheless, the analysis and empirical results demonstrate that the representations of these two methods are not scalable in terms of discriminability and dimension robustness.

In this paper, we propose Mixture-of-Codes (MoC), a novel two-stage approach to effectively scale up semantic representations for recommendation.
First, we propose a Multi-Codebooks VQ-VAE method that learns multiple independent discrete codebooks in the indexing stage.
Once we have these codes for all items, we adopt a Mixture-of-Codes module to fuse the learnable embeddings of multiple codes in the downstream recommendation stage.
We use the name MoC to refer both the second stage itself, as well as the whole two-stage approach.
Comprehensive analysis shows that our method successfully achieves scalability regarding discriminability, dimension robustness, and performance.
Our contributions can be summarized as follows:

- •
We pioneer a study on the scalability of semantic representation on transferring knowledge from LLM to recommendation systems, and reveal that several baseline approaches fail to scale up effectively.
- •
We propose a novel two-stage Mixture-of-Codes approach, which learns multiple codebooks in the indexing stage based on LLM embeddings and then employs a Mixture-of-Codes module to fuse the embeddings of multiple codes in the downstream recommendation stage.
- •
Comprehensive experiments on three public datasets show that our method successfully achieves scalability regarding both discriminability, dimension robustness, and performance.

## 2 Preliminaries

VQ-VAE.
VQ-VAE first encodes the input $x$ with an encoder $\mathcal{E}$ and then train a codebook to transform embedding into discrete tokens.
Formally, a codebook $\mathcal{Z}=\{z_{k}\}_{k=1}^{K}$ is define as a finite set with prototype vectors $z_{k}\in\mathbb{R}^{n_{z}}$, where $K$ is the codebook size and $n_{z}$ is the dimensionality of code embeddings.
Given the encoder output $\mathbf{z}:=\mathcal{E}(x)\in\mathbb{R}^{n_{z}}$, VQ-VAE quantities the embedding with the code whose embedding is nearest to $\mathbf{z}$, that is,

$$ $\mathbf{z}^{\mathbf{q}}=\operatorname*{\arg\min}_{z_{k}\in Z}\|\mathbf{z}-z_{k }\|_{2}^{2}.$ (1) $$

Then the reconstruction is derived based on the quantized output $\mathbf{z}_{\mathbf{q}}$ and a decoder $\mathcal{D}$: $\hat{x}=\mathcal{D}(\mathbf{z}_{\mathbf{q}})$.
The model and codebook can be trained end-to-end via the loss function

$$ $\mathcal{L}_{\mathrm{VQ}}(\mathcal{E},\mathcal{D},\mathcal{Z})=\|x-\hat{x}\|^{ 2}+\left\|\mathrm{sg}[\mathbf{z}^{\mathbf{q}}]-\mathbf{z}\right\|_{2}^{2}+ \left\|\mathrm{sg}[\mathbf{z}]-\mathbf{z}^{\mathbf{q}}\right\|_{2}^{2},$ (2) $$

where sg[·] denotes the stop-gradient operation, the first term $\mathcal{L}_{\mathrm{rec}}=\|x-\hat{x}\|^{2}$ is a reconstruction loss, the second term is the commitment
loss that is used to force the encoder output $\mathbf{z}_{\mathbf{q}}$ commits to the codewords and the bottleneck codewords are optimized by the third term.
In practice, we perform moving averages update (Van Den Oord et al., ) instead of adding auxiliary losses for stable training of the codebook.
Then the selected index can be used as Semantic IDs for clustering in the context of the semantic codebook, therefore capturing local structure of the original embedding.

Semantic IDs for Feature Interaction.
In recommendation system, feature interaction models how different attributes from users and items influence each other to affect recommendation outcomes. When incorporating the Semantic ID from quantization, the models treat the Semantic ID
$x_{sid}$
as a new feature field. The features are fed with other $N$ features into the embedding layer $E_{i}$
for each field and subsequently into the feature interaction modules for prediction.

$$ $\displaystyle\bm{e}_{i}$ $\displaystyle=\bm{E}_{i}^{\top}\bm{1}_{x_{i}},\mathrm{~{}}\forall i\in\{1,2,.. .,N\},$ (3) $\displaystyle\bm{e}_{sid}$ $\displaystyle=\bm{E}_{sid}^{\top}\bm{1}_{x_{sid}},$ $\displaystyle h$ $\displaystyle=I(\bm{e}_{1},\bm{e}_{2},...,\bm{e}_{n},\bm{e}_{sid}),$ $\displaystyle\hat{y}$ $\displaystyle=F(h),$ $$

## 3 On the Scaling of Semantic Representation

In this section, we first revisit the design of one single codebook for recommendation and discover the information loss due to dimension compression.
Based on this observation, we are motivated to design scalable semantic representations and propose two quantitative metrics to measure the information and dimension gain from scaling.
Next, we conduct a detailed analysis of existing scaling methods based on these metrics.

### 3.1 Limination of a Single Code

The original LLM embeddings span a high-dimensional space, *e.g.*, ranging from 1,024 to 416,384 dimensions. (Dubey et al., ).
When we build a semantic representation based on the LLM embeddings to transfer their knowledge to recommendation, the dimension of these representation needs to match that of the recommendation ID embeddings.
However, the dimension of recommendations is usually no more than 256 due to the Interaction Collapse Theory (Guo et al., ), and hence is much smaller than that of the LLM embeddings.
Such dramatic dimension compression may result in huge information loss.

To illustrate this empirically, we conduct a toy study based on the reconstruction error of the LLM embeddings in a VAE-based generative model.
As shown in Fig 2, we utilize a two-layer MLP with 512 hidden units to reconstruct the original 4096-dimensional LLM embedding.
When using only single set of Semantic ID as input, the reconstruction error is extremely high, indicating the significant information loss.
However, the error drops significantly when we scale up the dimensions via using multiple (*i.e.*, 2x and 3x) independent codebooks.
This demonstrates the original single code embedding only preserves limited information of the LLM embeddings.
Therefore, we aim to scale up the semantic representation appropriately to preserve the rich information from the LLM, thereby improving the performance of downstream recommendation tasks.

Figure: Figure 1: Reconstruction Error with different sets of Semantic IDs.
Refer to caption: x1.png

### 3.2 Baseline approaches to Scalable Semantic Representation

Below we present two baseline approaches to scale up semantic representation based on Multi-Embedding (Guo et al., ) and RQ-VAE (Lee et al., ).

Single Codebook with Multi-Embeddings.
Inspired by the recent Multi-Embedding (Guo et al., ) approach to scale up embeddings in recommendation systems, our first choice is to assign multiple embeddings for each semantic ID.
Specifically, we still learn only one single codebook during the indexing stage, while we build $M$ independent embeddings $\{\bm{e}_{sid}^{1},\dots,\bm{e}_{sid}^{M}\}$ in the downstream stage.
Formally, we have

$$ $\displaystyle\bm{e}_{sid}^{i}$ $\displaystyle=(\bm{E}_{sid}^{i})^{\top}\bm{1}_{x_{sid}},\mathrm{~{}}\forall i \in\{1,2,...,M\},$ (4) $\displaystyle h$ $\displaystyle=I(\bm{e}_{1},\bm{e}_{2},...,\bm{e}_{n},\bm{e}_{sid}^{1},...,\bm{ e}_{sid}^{M}).$ $$

RQ-VAE (*i.e.*, Multiple Hierarchical Codebooks and Multi-Embeddings).
RQ-VAE is a common practice for deriving Semantic IDs in recommendation systems (Rajput et al., ; Jin et al., ; Zheng et al., ). It applies quantization on residuals at multiple levels with different codebooks. The reconstructed target in the next level is the residual representation in the current level:

$$ $\displaystyle\mathbf{z_{i}}^{\mathbf{q}}$ $\displaystyle=\operatorname*{\arg\min}_{z_{k}\in Z_{i}}\|\mathbf{z_{i}}-z_{k} \|_{2}^{2},$ (5) $\displaystyle\mathbf{z_{i+1}}$ $\displaystyle=\mathbf{z_{i}}-\mathbf{z_{i}}^{\mathbf{q}}.$ $$

Due to the hierarchical design of RQ-VAE, the Semantic IDs obtained from the codebooks are highly dependent and entangled.
With $M$ levels of hierarchical Semantic IDs $\{x_{sid_{i}}\}_{i=1}^{M}$, it is practical to utilize to scale the Semantic Representation as follows.

$$ $\displaystyle\bm{e}_{sid_{i}}$ $\displaystyle=(\bm{E}_{sid_{i}})^{\top}\bm{1}_{x_{sid_{i}}},\mathrm{~{}} \forall i\in\{1,2,...,M\}$ (6) $\displaystyle h$ $\displaystyle=I(\bm{e}_{1},\bm{e}_{2},...,\bm{e}_{n},\bm{e}_{sid_{1}},...,\bm{ e}_{sid_{M}}).$ $$

Figure: (a) NMI w.r.t. Semantic ID.
Refer to caption: x5.png

Figure: (a) 3x scaling factor.
Refer to caption: x7.png

### 3.3 Measurement on Scalability of Semantic Representation

Below we present two ways to quantify the scalability of semantic representations, one from a discriminability perspective and another from a dimension robustness perspective.

Figure: Figure 5: Comparison among Multi-Embedding VQ, RQ-VAE and Mixture-of-Codes. The codebooks with deeper color contain more information relevant to the input data. (a) Multi-Embedding VQ builds independent embeddings for a single set of semantic IDs and is equivalent to perform index copying for downstream models. (b) RQ-VAE utilizes hierarchical codebooks and high-level semantic IDs are less informative. (c) Our Mixture-of-Codes uses parallel codebooks to capture important semantics in the original LLM space and employs a fusion network for better generalization in downstream tasks.
Refer to caption: x10.png

###### Definition 3.1 (Discriminability Scalability of Semantic Representation) .

With each scaling factors from $1$x to $M$x,
the discriminability of a Semantic Representation in the continuous space is defined as the mutual information between its quantized representation $Q(\bm{r})$ and the supervised label in the downstream tasks $Y$, *i.e.*, $\text{MI}(Q(\bm{r}),Y)$.

Following previous approach (Jawahar et al., ), we apply K-means as the discrete method for normalized mutual information (NMI) calculation and analyze the flatten embedding of the concatenated Semantic Representations among different methods in Fig. 2. We present results with respect to different scaling factors with 100 clusters and provide flattened embedding NMI under varying cluster numbers in Fig. 2(b).
We observe that the discriminability of the ME does not increase with the scaling factor and may even decrease slightly. This is because all these extra embeddings still correspond to the same Semantic IDs from a single codebook, thus containing minimal additional information and redundancy.

Regarding RQ-VAE, its discriminability also does not consistently increase with the scaling factor due to the fact that the additional fine-grained Semantic IDs introduced at higher level contain diminishing information.
We illustrate this by comparing NMI across different Semantic IDs and their representations in downstream tasks in Fig. 3. Another surprising observation from the results in Fig. 3 is that the lowest level Semantic ID, *i.e.*, SID 1, and its representation contain more information than the flattened embedding of all the Semantic IDs.
We provide the NMI results across various cluster numbers in Fig. 2(c) and observe that the NMI of SID 1 is consistently larger than that of the flattened embedding when the cluster number exceeds 50, demonstrating that higher-level Semantic IDs may hinder the generalization of lower-level Semantic IDs.

When scaling up the Semantic Representation, the interaction between low-frequency information and high frequency becomes important as the dimension increases. Therefore, we propose a new metric to measure the dimension robustness of the scaling Semantic Representation.

###### Definition 3.2 (Dimension Robustness Scalability of Semantic Representation) .

The dimension robustness scalability of Semantic Representation can be measured by the singular spectrum of the Semantic Representation under different scaling factors. A robust Semantic Representation should have higher top singular values without suffering from dimension collapse.

We plot the singular spectrum of ME and RQ-VAE in different scaling factors in Fig. 4 to compare the dimension robustness and its scalability between models. And we have the following observations.

Observation 1. RQ-VAE doesn’t suffer from dimensional collapse since that its long-tail singular values don’t diminish. However, its top singular values are not large enough
compared with ME.

Observation 2. ME has the largest top singular values. However, it suffers from dimensional collapse since its long-tail singular values diminishes suddenly after index 250 in 5x and 275 in 7x setting.

We conclude with the following finding:

###### Definition 3.1 (Discriminability Scalability of Semantic Representation) .

###### Definition 3.2 (Dimension Robustness Scalability of Semantic Representation) .

## 4 Method

In this section, we propose Mixture-of-Codes (MoC) as a novel two-stage method to scale up semantic representation.
We first introduce multiple codebooks in the indexing stage to generate multiple sets of Semantic IDs, and then present Mixture-of-Codes in the downstream recommendation modeling stage for better knowledge transfer.

### 4.1 Multi Codebooks for vector quantization

Motivated by the observation that multi-embedding does not provide new information due to the same Semantic ID while RQ-VAE focuses on hierarchical indices that are less informative for downstream tasks, we aim to uncover more information from the LLM embedding at a more fundamental level. Specifically, instead of the hierarchical design of RQ-VAE, we utilize multiple parallel codebooks to capture complementary information of the LLM embedding.
Different codebooks project the hidden embedding into various spaces and collaborate to extract information from the LLM embedding.
Formally, given encoder $\mathcal{E}$, decoder $\mathcal{D}$ and $N$ Codebooks $\{\mathcal{Z}_{i}\}_{i=1}^{N}$, we perform an average over their quantized embedding and the training loss is

$$ $\displaystyle\mathcal{L}_{\mathrm{MoC}}(\mathcal{E},\mathcal{D},\{\mathcal{Z}_ {i}\}_{i=1}^{N})$ $\displaystyle=\|x-\hat{x}\|^{2}+\left\|\mathrm{sg}[\mathbf{z}^{\mathbf{q}}]- \mathbf{z}\right\|_{2}^{2}+\left\|\mathrm{sg}[\mathbf{z}]-\mathbf{z}^{\mathbf{ q}}\right\|_{2}^{2},$ (7) $\displaystyle\mathbf{z^{q}}$ $\displaystyle=\text{AVG}(\{\mathbf{z_{i}^{q}}\}_{i=1}^{N}),$ $$

where $\mathbf{z}^{q}$ is the average quantization results and we select the corresponding indices as Semantic IDs.

Figure: Figure 6: The overall architecture of MoC Fusion. A bottleneck network is adopted for feature fusion in the downstream stage.
Refer to caption: x11.png

### 4.2 Mixture-of-Codes for Implicit Fusion

For traditional mixtures of experts, a gating router is used to select some of the experts and perform mixing based on the weights generated by the router with the help of the task-specific loss. However, this approach is impractical in MoC since we do not train the codebooks in an end-to-end style, and the embeddings are initialized and tuned in the downstream stage.

Therefore, we propose a fusion network in the downstream stage for implicit fusion of the codebooks. Specifically, we employ a bottleneck network following the embedding layer to ensure information flow across different features before the feature interaction modules, as shown in Figure 6. This implicit fusion design is trained using task-specific loss and mixes the embeddings for better
performance, serving a role similar to the gating network.
Formally, given $N$ original attributes and $M$ Semantic IDs, we have

$$ $\displaystyle e_{\text{concat}}$ $\displaystyle=\text{CONCAT}(\bm{e}_{1},...,\bm{e}_{n},\bm{e}_{\text{sid}_{1}}, ...,\bm{e}_{\text{sid}_{M}}),$ (8) $\displaystyle e_{\text{concat}}^{\prime}$ $\displaystyle=e_{\text{concat}}+e_{\text{concat}}\cdot\mathbf{W}_{\text{down}} \cdot\mathbf{W}_{\text{up}},$ $\displaystyle\bm{e}_{1},...,\bm{e}_{n},\bm{e}_{\text{sid}_{1}},...,\bm{e}_{ \text{sid}_{M}}$ $\displaystyle=\text{SPLIT}(e_{\text{concat}}^{\prime}),$ $$

where $\mathbf{W}_{\text{down}}$ and $\mathbf{W}_{\text{up}}$ denotes the down and up projection layer, respectively.

### 4.3 In-depth Scalability Analysis of MoC

Scalability of Discriminability
We study the discriminability of MoC by the mutual information between quantized representation and the label at various scaling factors in Fig. 3(b).
It can be observed that the discriminability of MoC at 1x is comparable with that of RQ-VAE and much higher than ME.
Furthermore, the discriminability of MoC gets higher with larger scaling factors from 1x to 7x, indicating that it has better scalability regarding discriminability.

Scalability of Dimension Robustness
We plot the singular spectrum of MoC on various scaling factors in Fig. 4., and find that it gets higher values on the low-index singular, compared to the RQ-VAE, even though not as high as ME.
Besides, its singular values on high-indices are also robust, not diminishing as ME in 5x and 7x factors.
In conclusion, the dimension of MoC are more robust than ME and RQ-VAE when we scale up its representation.

Based on the observations above, we conclude with the following finding:

## 5 Experiments

**Table 1: Test AUC of different methods over various models. We report the test AUC results with 1x, 2x, 3x and 7x scaling factors.**
| Model | Toys | Beauty | Sports |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1x | 2x | 3x | 7x | 1x | 2x | 3x | 7x | 1x | 2x | 3x | 7x |  |  |
| DeepFM | ME | 0.7406 | 0.7403 | 0.7397 | 0.7390 | 0.6651 | 0.6651 | 0.6649 | 0.6638 | 0.6931 | 0.6942 | 0.6928 | 0.6917 |
| RQ-VAE | 0.7409 | 0.7405 | 0.7398 | 0.6676 | 0.6670 | 0.6687 | 0.6945 | 0.6932 | 0.6937 |  |  |  |  |
| MoC | 0.7408 | 0.7415 | 0.7418 | 0.6656 | 0.6674 | 0.6681 | 0.6931 | 0.6936 | 0.6953 |  |  |  |  |
| DeepIM | ME | 0.7404 | 0.7396 | 0.7404 | 0.7395 | 0.6648 | 0.6620 | 0.6635 | 0.6637 | 0.6931 | 0.6907 | 0.6910 | 0.6925 |
| RQ-VAE | 0.7401 | 0.7403 | 0.7404 | 0.6651 | 0.6660 | 0.6678 | 0.6918 | 0.6925 | 0.6938 |  |  |  |  |
| MoC | 0.7401 | 0.7417 | 0.7422 | 0.6641 | 0.6668 | 0.6691 | 0.6927 | 0.6935 | 0.6942 |  |  |  |  |
| AutoInt+ | ME | 0.7415 | 0.7430 | 0.7419 | 0.7414 | 0.6630 | 0.6648 | 0.6630 | 0.6641 | 0.6911 | 0.6935 | 0.6930 | 0.6929 |
| RQ-VAE | 0.7430 | 0.7419 | 0.7418 | 0.6672 | 0.6642 | 0.6677 | 0.6934 | 0.6933 | 0.6915 |  |  |  |  |
| MoC | 0.7414 | 0.7420 | 0.7447 | 0.6661 | 0.6651 | 0.6689 | 0.6939 | 0.6926 | 0.6927 |  |  |  |  |
| DCNv2 | ME | 0.7445 | 0.7445 | 0.7449 | 0.7459 | 0.6701 | 0.6717 | 0.6716 | 0.6722 | 0.6962 | 0.6955 | 0.6963 | 0.6976 |
| RQ-VAE | 0.7457 | 0.7457 | 0.7469 | 0.6719 | 0.6720 | 0.6726 | 0.6965 | 0.6966 | 0.6979 |  |  |  |  |
| MoC | 0.7462 | 0.7458 | 0.7474 | 0.6714 | 0.6730 | 0.6729 | 0.6970 | 0.6972 | 0.6989 |  |  |  |  |

### 5.1 Setup

Datasets.
We conduct experiments on three domains from Amazon review benchmark (He & McAuley, ): Amazon-Beauty, Amazon-Sports, and Amazon-Toys.
We follow LMINDEXER (Jin et al., ) and keep the users and items that have
at least 5 interactions to filter out unpopular interacting behavior.
Given textual description of items comprising of title, brand and categories, we utilize LLM2Vec (BehnamGhader et al., ) with LLama3 (Dubey et al., ) as backbone to obtain their LLM embeddings.
Early stop strategy are adopted over 8/1/1 training/validation/test splits of all the three datasets.

Implementation Details.
We follow TIGER (Rajput et al., ) to set 256 as the codebook size and 32 as the latent representation. The encoder in the indexing stage has three hidden layers of size 512, 256 and 128 with ReLU activation.
We evaluate the performance of DeepFM (Guo et al., ), DeepIM (Yu et al., ), AutoInt+ (Song et al., ) and DCNv2 (Wang et al., ) in the downstream tasks.
We adpot the Adam optimizer with batch size 8012 and learning rate 0.001.
All the experiments are run across three trials with different seeds and the averaged results are reported.

### 5.2 Overall Performance

We compare the three semantic scaling methods upon four representative CTR models, *e.g.*, DeepFM, DeepIM, AutoInt+ and DCN V2 on three public datasets.
With the same number of IDs, our MoC outperforms the baselines by a large margin, especially in scenarios with a larger number of IDs, *e.g.*, 7x.
Specifically, with 7x scaling factor, all the four models benefits from MoC on Toys datasets and surpass RQ-VAE by 0.20%, 0.18%, 0.29% and 0.05%, respectively.

More interestingly, in many scenarios, *our MoC succeed in achieving scaling law regarding the scaling factors*, *i.e.*, the performance increases when we include more Semantic Representations.
In contrast, ME suffers from performance degradation due to the redundant semantic information, while RQ-VAE gain slight performance gain because of the less informative Semantic IDs at the high level.

Figure: (a) MoC correlation.
Refer to caption: x12.png

### 5.3 Correlation between Multiple Representation.

The Semantic Representation from different IDs have a deep influence on each other in downstream tasks, and hence we analyze the correlation between different representations to measure the extent of their similarity.
Here we calculate Person correlation coefficient (Cohen et al., ) between Semantic Representation $e_{sid_{i}}$ and $e_{sid_{j}}$ over $n$ samples, and adopt dot product for multiplication of embedding vectors:

$$ $\displaystyle r_{ij}=\frac{\sum_{k=1}^{n}(e^{k}_{sid_{i}}-\bar{e}_{sid_{i}})(e ^{k}_{sid_{j}}-\bar{e}_{sid_{j}})}{\sqrt{\sum_{k=1}^{n}(e^{k}_{sid_{i}}-\bar{e }_{sid_{i}})^{2}}\sqrt{\sum_{k=1}^{n}(e^{k}_{sid_{j}}-\bar{e}_{sid_{j}})^{2}}}.$ $$

As shown in Fig. 7, the Semantic Representation in ME are highly correlated with each other, *i.e.*, many different Semantic Representation have strong correlation, *i.e.*, representation 1 and 3, 4 and 6 are strongly correlated with each other.
Such strong correlation makes the representation easily influenced by each other, leading to unstable optimization and ineffective scalability.
Regarding MoC and RQ-VAE, the correlation between different Semantic Representation are low, as evidenced by the low correlation score in the off-diagonal cells in Fig. 7.

### 5.4 More comparison Results with RQ-VAE

To further verify the information contained in the semantic IDs at each level, we provide a more detailed comparison with RQ-VAE in terms of adding a single ID at each level in Fig. 8(a) and adding multiple IDs starting from the lowest level in Fig. 8(b).
As the results in Fig. 8(a) indicate, adding a single semantic ID at the low level of RQ-VAE provides a larger performance gain than at the high level, proving that semantic IDs at the high level hold little information.
In contrast, MoC performs uniformly across various Semantic IDs and consistently show better performance than RQ-VAE.
When equipping multi Semantic IDs starting from the lowest level, MoC gains significant improvements than RQ-VAE under different scaling factors, showcasing better generalization.

### 5.5 Ablation on MoC Fusion

We conduct an ablation study on MoC Fusion to verify the importance of mixing in the downstream stages. We equip both RQ-VAE and MoC with the fusion module and surprisingly find that both methods benefit from it when scaling up. We also examine the discriminability and dimension robustness scalability of MoC Fusion.
In Fig. 9, it can be observed that the fusion module enhances the overall discriminability scalability of MoC by mixing the features. In Fig. 10, we truncate the singular spectrum and surprisingly find that the fusion module amplifies the principal components of the Semantic Representation, resulting in significantly higher top singular values. Additionally, the long-tail part is close to that of RQ-VAE and does not suffer from dimension collapse.

Figure: (a) AUC w.r.t. Single ID.
Refer to caption: x15.png

**Table 2: Ablation on MoC Fusion. The experiments are conducted over Toys dataset with DeepFM as the backbone.**
| Method | 2x | 3x | 7x |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| w/o | w/ | w/o | w/ | w/o | w/ |  |
| RQ-VAE | 0.7409 | 0.7414 | 0.7405 | 0.7407 | 0.7398 | 0.7413 |
| MoC | 0.7409 | 0.7408 | 0.7404 | 0.7415 | 0.7416 | 0.7418 |

## 6 Related Work

Discrete representation learning.
Discrete representation is first introduced by (Van Den Oord et al., ) and shows feasibility and great success in the field of generate models (Esser et al., ; Lee et al., ; Rombach et al., ; Peebles & Xie, ; Mentzer et al., ). It utilize vector quantization (VQ) to model distributions over discrete variables with a codebook and define a simple uniform prior instead of Gaussian prior in VAE (Kingma, ) to avoid posterior collapse. RQ-VAE (Lee et al., ) further introduces residual quantization for better minimization of reconstruction error. FSQ (Mentzer et al., ) remove auxiliary losses and replace the vector quantizer in VQ-VAE with a simple scalar quantization.

Semantic IDs.
The discrete representation obtained through VQ-VAE can be employed as a semantic clustering IDs, thus capturing the local structure of the LLM embedding to a certain extent.
In the context of the recommendation system, TIGER (Rajput et al., ) takes advantage of a hierarchical quantizer (Lee et al., ) to convert items into tokens for generative recommendation and retrieval.
LC-Rec (Zheng et al., ) improves TIGER by incorporating knowledge from LLMs like LLama (Touvron et al., ) and introducing instruction tuning tasks for effective adaptation to recommender systems.
LMINDEXER (Jin et al., ) learns the Semantic IDs in a self-supervised styles to obtain the document’s semantic representations and their hierarchical structures.

## 7 Conclusion

In this paper, we investigate the scalability of semantic representation based on LLM for recommendation.
We unveil that simple methods, such as using a single codebook with multiple embeddings and scaling with hierarchical codebooks in RQ-VAE, do not scale effectively in Semantic Representation.
We propose a novel multiple codebooks method which learn multiple independent Semantic IDs in the VQ-VAE based on LLM embeddings, and then employ these codebooks with a fusion module in the downstream recommendation models.
Comprehensive experiments show that the proposed method successfully achieves scalability regarding discriminability, dimension robustness, and performance.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: 1.	To my best knowledge, this paper discretizes item text representations as additional discrete features to assist the CTR model learning, rather than directly transforming and incorporating the raw features extracted from LLM (e.g., through adapters) into the model. If I have understood this correctly, why do the authors emphasize various issues in the third paragraph of the introduction regarding the transition from high-dimensional LLM features to low-dimensional recommendation features? Since the core idea is "feature engineering”, even if the dimensionality of the features from LLM and the recommendation model is the same, these issues should still exist.

2.	(Followed by Question 2 and Weakness 1) The author's approach of using quantized semantic codebooks as additional features is innovative; it differs significantly from most works that use discrete codewords as item identifiers in generative recommendations. Therefore, I believe that the author should emphasize the advantages of this approach compared to not adding these features. For example, I think it is necessary to include the comparison of the proposed method with the original method that does not utilize discrete representations (but it does not provide results for this basic baseline). Additionally, the benefits of adding these features should be further analyzed.
soundness: 3
strengths: 1.	This paper discusses an interesting problem: how to scale up semantic representations on transferring knowledge from LLM to recsys, and provides a detailed analysis of the shortcomings of some traditional approaches.
2.	The authors propose an effective algorithm (Mixture-of-Codes) to improve the scalability of discrete semantic codebooks, and the experimental results validate the effectiveness of the proposed method.
summary: This paper investigates the scalability of semantic representation based on LLM for recommendation. The authors introduce an innovative approach termed Mixture-of-Codes, which involves the creation of multiple independent codebooks during the indexing phase. Subsequently, it employs semantic representations in conjunction with a fusion module during the recommendation phase. Comprehensive experiments demonstrate that this proposed method effectively achieves scalability in terms of discriminability, dimensional robustness, and overall performance.
weaknesses: 1.	The writing of this paper needs to be improved. For example, Sections 4.1 and 4.2 use the same symbol N to represent two different variables, which seems inconsistent. Besides, there are many experiments and analyses through the paper, and the authors should pay more attention to the relationships between them.
2.	The use of discrete semantic codebooks in this paper is innovative, but before the authors further discuss how to use them better, they should first elaborate on the advantages and benefits of the basic method in detail (please refer to question 2).
3.	This paper aims to scale up semantic representation, but the authors do not explain why scaling up is necessary and what’s the significance. I agree that “scaling up” could be impactful, but the authors should provide a detailed discussion.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: See Weaknesses.

If the authors address these concerns, I can consider a higher rating.
soundness: 2
strengths: 1. The paper is well-written and easy to understand.
2. The paper pioneers as study on the scalability of semantic representation on transferring knowledge from LLM to recommendation systems.
summary: The authors find when applying LLM embeddings to recommendation systems: the high dimensionality of LLM embeddings must be reduced to fit the lower-dimensional space used in recommendation models, leading to information loss. To solve this problem, authors propose a two-stage method called "Mixture-of-Codes" (MoC), which uses multiple indepentent codebooks for vector quantization of the LLM embeddings, then utilizes the Semantic representation for downstream task. This approach aims to improve both the discriminability and dimension robustness of semantic representations, ensuring that more detailed semantic information is preserved.
weaknesses: 1.	The current experimental design does not sufficiently validate the authors' claims

   a)	The improvements are not significant. In table1, compared to the non-scaling version, the improvement is often only around 0.1%.

   b)	The necessary baseline comparison using IDs to scale embeddings without using codebooks is missing. If there is no significant improvement compared to this baseline, then there is no need to explore the use of semantic IDs.

   c)	Contradic to this paper's claim: MoC will help for scaling up, using the MoC method leads to a decrease in AUC metrics on some datasets. (i.e. in Table 1, model DCNv2, Toys dataset, and model AutoInt+, Beauty dataset). Moreover, MoC does not consistently outperform the baseline, and in some cases, the RQ-VAE method achieves the best metrics. (i.e. in Table 1, model DeepFM, Beauty dataset, 7x).

   d)	The components in MoC do not always contribute positively. removing the MoC fusion module actually led to metric improvements in certain cases. (Table2, MoC method, 2x)

   e)	Lack of ablation of main component of MoC. There lack a fair comparison with ME [1] with fusion.

2.	The mathematical definitions in the article could be further improved.

   a)	There is a lack of definition for 1_{x_1} indicates the one-hot encoding of x_i in X_i in lines 096 and so does line 150

   b)	In lines 321, The authors have omitted the mathematical modeling of the activation layer.


[1] Guo, Xingzhuo, et al. "On the Embedding Collapse when Scaling up Recommendation Models." Forty-first International Conference on Machine Learning.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 1
questions: Please refer to "Weaknesses" for details.
soundness: 1
strengths: 1. Timely study on semantic IDs, semantic representations, and scaling law in recommender systems.
2. Experiments are conducted on three public datasets.
3. The idea of calculating mutual information as an indicator of discriminability is interesting.
summary: The paper first conducts experiments to demonstrate that ME (multi-embeddings) and RQ-VAE are not scalable approaches for semantic representation. The author(s) then introduce mixture-of-codes, which learns multiple parallel codebooks as semantic representations and employs a fusion module to capture feature intersections. Experiments are conducted on three public datasets to demonstrate the effectiveness of the proposed approach.
weaknesses: 1. Taking RQ-VAE-based semantic IDs for semantic representations is impractical.
    1. RQ-VAE-based semantic IDs are structured to maintain sequential dependence across different levels, making them commonly used in generative retrieval or generative recommendation models. In these cases, an autoregressive model is trained to capture these semantic IDs in a sequence-to-sequence fashion. From my understanding, RQ-VAE is not well-suited for direct application as features or semantic representations for tasks like CTR. Additionally, I have not found any papers that describe the use of RQ-VAE-based semantic IDs as features (representation learning) for ranking, either in the references of this paper or in my own literature review. The author(s) cite several papers to suggest that RQ-VAE is widely used for deriving semantic IDs in recommender systems; however, the cited methods apply these semantic IDs in a fundamentally different way than in this paper (specifically, as features for ranking). Therefore, RQ-VAE-based semantic IDs appear to be unreliable as one of the two most important base methods for both analytical and performance experiments.
    2. When discussing RQ-VAE-based semantic IDs, I’d like to suggest two alternative approaches for learning semantic representations.
        1. Product/Vector Quantization [1]: This method can also encode input vectors into multiple semantic IDs but without sequential dependencies. It’s been used in existing literature as a semantic representation approach. Vector quantization is much simpler than the RQ-VAE-based method and is worth exploring further.
        2. Multiple Adaptors on LLM Embeddings [2, 3]: Instead of using semantic IDs, this approach directly maps input LLM embeddings to the recommendation space through adaptors (e.g., MLP). The scalability of this method can be explored by increasing the number of adaptors.
    3. In Section 3.1, the author(s) discuss the limitation of a single code. However, few methods actually use a single code. For example, TIGER uses 4 codes per item, and VQ-Rec [1] uses 32 codes per item. It would be better if more practical settings were adopted, like 4 -> 32 instead of 1 -> 3 semantic IDs, when drawing the observations in Figure 1.
2. Experiments are not solid enough.
    1. In Table 1, only methods using additional semantic representations (e.g., ME, RQ-VAE semantic IDs, MoC) are compared. Including a baseline method that relies solely on the original features, without any added semantic representations, would provide a clearer sense of the value these scalable semantic representations offer.
    2. Does "1x, 2x, 3x, 7x" in Table 1 mean "1, 2, 3, 7" levels of semantic IDs (or embeddings of ME)?
    3. It appears that there is no indication of repeated experiments or statistical significance testing, especially since the AUC metrics do not show large differences across all methods.
    4. Ablation study (Table 2).
        1. The effectiveness of applying the proposed fusion method on MoC seems not significant. On settings "2x" and "7x", the fusion module doesn't bring gains for MoC but downgrades the performance.
        2. Only one dataset is used for ablation study.
3. Presentation issues.
    1. In line 124, "416,384" -> "16,384"?
    2. In Eqn. (3) and (4), $h$ and $n$ are not defined. What are the connections between $n$ and $N$?
    3. In lines 201 - 202, what does "quantized representation $Q(r)$" actually mean? Does it mean the clustering centroids of RQ?
    4. What are the references of "a robust semantic representation should have higher top singular values" in Definition 3.2?
    5. Eqn. (8) doesn't match Figure 6. There is no activation function between $W_{down}$ and $W_{up}$ in Eqn. (8).
    6. Line 244, "low-frequency information" and "high-frequency" are not defined in the paper.
4. Code is not available during the reviewing phase.

[1] Learning Vector-Quantized Item Representation for Transferable Sequential Recommenders. WWW 2023.

[2] Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisited. SIGIR 2024.

[3] Towards Universal Sequence Representation Learning for Recommender Systems. KDD 2022.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. what are x-axis and y-axis in Figure 10?
2. In section 4.1, average over quantized embedding is introduced into loss function for a shared VAE decoder. What about train different VAE decoder for each codebook?
soundness: 3
strengths: 1. Compared with the hierarchical design of RQ-VAE, MoC develops a parallel framework with multiple codebooks. 
2. The study conducts intensive visualizations and experiments on three public datasets (Amazon Beauty, Sports, and Toys) and compares MoC with existing approaches.
summary: It is still a challenge to incorporate the great power of LLMs into enhancing recommender systems. Simply compressing the the high-dimensional nature of LLM embeddings to fit the smaller ID embeddings of recommendation will lead to information loss.
Following the multi-embedding setting and VQ-VAE, this paper proposes a two-stage Mixture-of-Codes generating semantic IDs by
multiple codebooks in a VQ-VAE framework, and then fuse them using a fusion module in the downstream recommendation stage. The performance table and visualization further verify its effectiveness and scalability.
weaknesses: 1. The 3rd paragraph of Introduction section seems like the paraphrase of the 1st paragraph of section 3.1. 

$\textbf{Introduction: }$"Notably, the LLM embeddings usually have very large dimensions, ranging from 4,096 to
16,384 (Dubey et al., 2024). When generating the embeddings for these codes, their dimension needs
to match that of the recommendation IDs. However, the dimension in recommendation is usually
small due to the Interaction Collapse Theory (Guo et al., 2023)." 

$\textbf{Section 3.1: }$"The original LLM embeddings span a high-dimensional space, e.g., ranging from 1,024 to 416,384
dimensions. (Dubey et al., 2024). When we build a semantic representation based on the LLM embeddings to transfer their knowledge to recommendation, the dimension of these representation needs
to match that of the recommendation ID embeddings. However, the dimension of recommendations
is usually no more than 256 due to the Interaction Collapse Theory (Guo et al., 2023), and hence is
much smaller than that of the LLM embeddings. "

I think the illustration of section 3.1 will lead to redundancy and therefore not needed.

2. The illustration on fusion module is not clear. It is just a self-gating network. Did authors compare with other fusion experiments (e.g., concatenation, single MLP, or cross-attention)?