From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models
Authors: Mingjia Yin
Junwei Pan
Hao Wang
Ximei Wang
Shangyu Zhang
Jie Jiang
Defu Lian
Enhong Chen
Venue: ICML 2025 poster (2025)
Keywords: Generative model
Recommender system
Feature interaction
Primary area: applications

Abstract: Click-Through Rate (CTR) prediction, a core task in recommendation systems, aims to estimate the probability of users clicking on items. 
Existing models predominantly follow a discriminative paradigm, which relies heavily on explicit interactions between raw ID embeddings. 
However, this paradigm inherently renders them susceptible to two critical issues: embedding dimensional collapse and information redundancy, stemming from the over-reliance on feature interactions *over raw ID embeddings*. 
To address these limitations, we propose a novel *Supervised Feature Generation (SFG)* framework, *shifting the paradigm from discriminative "feature interaction" to generative "feature generation"*. 
Specifically, SFG comprises two key components: an *Encoder* that constructs hidden embeddings for each feature, and a *Decoder* tasked with regenerating the feature embeddings of all features from these hidden representations. 
Unlike existing generative approaches that adopt self-supervised losses, we introduce a supervised loss to utilize the supervised signal, \ie, click or not, in the CTR prediction task.
This framework exhibits strong generalizability: it can be seamlessly integrated with most existing CTR models, reformulating them under the generative paradigm. 
Extensive experiments demonstrate that SFG consistently mitigates embedding collapse and reduces information redundancy, while yielding substantial performance gains across various datasets and base models. 
The code is available at https://github.com/USTC-StarTeam/GE4Rec.

From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models
Mingjia Yin
Junwei Pan? Hao Wang's Jie Jiang?
Ximei Wang² Enhong Chen
Shangyu Zhang? 2
Defu Lian'
Abstract
1. Introduction
Click-Through Rate (CTR) prediction, a core task in recommendation systems, aims to estimate the probability of users clicking items. Existing models predominantly follow a discriminative paradigm, which relies heavily on explicit inter- actions between raw II) embeddings. However, this paradigm inherently renders them suscepti- ble to two critical issues: embedding dimensional collapse and information redundancy, stemming from the over-reliance on feature interactions over raw ID embeddings. To address these limitations, we propose a novel Supervised Feature Gener- ation (SFG) framework, shifting the paradigm from discriminative "feature interaction" to gen- erative "feature generation". ". Specifically, SFG comprises two key components: an Encoder that constructs hidden embeddings for each feature, and Decoder tasked with regenerating the fea- ture embeddings of all features from these hid- den representations. Unlike existing generative approaches that adopt self superviee losses, we introduce a supervised loss to utilize the super- vised signal, i.e., click or not, in the CTR predic- tion task. This framework exhibits strong gener- alizability: it can be seamlessly integrated with most existing CTR models, reformulating them under the generative paradigm. Extensive exper- iments demonstrate that SFG consistently miti- gates embedding collapse and reduces informa- tion redundancy, while yielding substantial perfor- mance gains across various datasets and base mod- els. The code is available at https: ptttt:::/iiiii //gttuub. com/USTC- SwEssC--taaree SeeCC--ttareaaeeeeeeeea TeeeTe//////...... GE4Rec.
Click- Thrkuhhhrough Rate (CTR) prediction models estimate the probability of users clicking items based on feature in- teractions. While conventional wisdom holds that CTR models inherently follow a discriminative paradigm, they are prone to embedding dimensional collapse (Guo et al., 2024) and information redundancy (Zbontar et al., 2021) issues, primarily due to the interactions between raw ID embeddings (Guo et al., 2024).
Different from sequential recommendation models (Kang & McAuley, 2018; Zhai et al., 2024; Yin et al., 2024), very limited research has focused on formulating CTR models under generative paradigm. This is possibly due to the fact that there are no explicit partial orders among the inputs of CTR models, making it difficult directly fit them into the popular next-token prediction framework (Vaswani et al., 2017; Kang & McAuley, 2018).
Next-token prediction is an autoregressive generative paradigm (Fig. (a)) that treats the sequence up to position N as the source I source and the token at position N + 1 as the target Zurget, using the former to predict the latter. Recently, computer-vision researchers have reconsidered the ordering of image data to better align this modality with the autore- gressive framework. For instance, MAR (Li et al., 2024b) (Fig. 1(b)) employs unmasked image tokens as Tsource to generate masked tokens as Furget, whereas VAR (Tian et al., 2024) (Fig. I(c)) introduces a next-scale prediction frame- work that establishes a "coarse-to-fine" ordering. These works design generative frameworks that align with inher- ent data characteristics, rather than rigidly imposing the next-token prediction paradigm.
This inspired us to reconsider the "order" or "inherent struc- ture" of CTR prediction data. Specifically, CTR prediction handles the multi-field categorical data (Zhang et al., 2016; Pan et al., 2018), where there are usually no explicit partial orders between input features. However, there are com- plex co-occurrence relationships among those features (Ren- dle, 2010), which motivates us to treat one side of the co- occurrence features as Tourcce and the other side as Fuaget Such a method can be regarded as a "feature generation"
'State Key Laboratory of Cognitive Intelligence, Uni- versity of Science and Technology of China, Hefei, China TTencent Inc, China. Correspondence to: Hao Wang <wang- hao3 @ds....du.cn>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models
(Eneoder &)
penerate
(Encoder &.) Decoder
geterale
Decoder
Previous item
loss (seli)
Unneskal
loss (self)
Figure 1. Different generative paradigms. Specifically, this framework employs an (optional) encoder to process source data of particular form and generate an output embedding. Subsequently, a decoder utilizes this embedding to generate the larget, rep- resenting data of different form. Finally, a loss function will he used to evaluate the generation quality. Lacking an explicit data structure, our feature generation paradigm adopts an "all predict all" paradigm to predict each feature with all features. Notably, we employ supervised generative loss function, optimizing the cross-entropy loss regarding the sample-wise label Yup- rather than the self-supervised loss.
Next Aon
122
Rew
Raw
Split by tomponil onder (a) Sequential Generation
Using randomly input
(b) Masked Generation
(Encoder& &) Decoder
getierate
Decoder
(self)
loss (super)
Coarse scale!
Fine scale
Encoder
cls
9
Multi-ccll quantization
y
Raw daza From counse to fine scale gramulrity
data esplicc onder, using ingui
(c) Coarse-to-fine Generation
(d) Feature Generation
paradigm, which models the data distribution P(X) on the unordered multi-field categorical data.
performance, achieving an average of 0.272% AUC lift and 0.435% Logloss reduction, while incurring only marginal increase in computational overhead an average increase of 3.14% in computation time and 1.45% in GPU memory con- sumption. can produce feature embeddings with reduced collapse and redundancy compared raw ID embeddings. Additionally, conduct extensive ablation studies to vali- date the framework design. We successfully deployed it to Tencent's 's advertising platforms for click prediction, with a 2.68% GMV lift on primary scenario, leading to one of the largest revenue lift in 2024.
Specifically, the -geuurrignnn paradigm employs an Encoder to transform Vsource into a latent space, yielding new representation for each feature. Decoder then projects these latent representations back to the original space to generate all original features Zaagat- Since the Encoder takes all original features input and simultane- ously generates all original features, this framework adheres to an Predicccit-- framework. Such a design avoids di- rect interaction (product) between vanilla ID embeddings as done in traditional CTR models (Rendle, 2010; Guo al., 2017; Sun et al., 2021; Wang et al., 2021), and hence pre- vents the embeddings from dimensional collapse due to Interaction-Collapee Theory (Guo et al., 2024). Moreover, the Encoder produces sample-specific representations that are decorrelated from the original embeddings, reducing the redundancy of feature embeddings.
2. Preliminaries
In this section, present the traditional CTR prediction problem in the discriminative paradigm and its challenges.
2.1. Problem definition
Conventionally, generative paradigms are accompanied by self-supervised loss. In the next-prediction paradigm in sequential modeling, the label of the next item is usually self- self-supervised signal, that is, whether the next item is the ground-truth one or just random one. However, in CTR prediction, it is unnecessary since natural supervised signals, i.e., click or not in CTR prediction, already exist. Therefore, we adopt a supervised loss with the proposed fea- ture generation paradigm. The feature generation, together with the supervised loss, leads to novel Supervised Feature Generation framework for CTR prediction.
The CTR prediction task predicts the probability of users clicking items based on multiple features. It can be formally defined using features X' {0, 13 and a label set y {0, 1}. indicating whether users click the candidate item. Typically, X' consists of hundreds of features from user, item, the context sides, each belonging to a feature field.
2.2. CTR prediction in a discriminative paradigm
Formulation. Existing CTR models are usually formu- lated under discriminative form, by first conducting ex- plicit and/or implicit feature interactions between all fea- tures through (mme(-)). then employing classifier fas. e.g., MLPs, upon its output, to get the final prediction score, and lastly optimizing binary coss-entroppy loss regarding the
This framework can reformulate nearly every existing fea- ture interaction model, ranging from FM to DeepFM, XDeepFM, and DCN V2. Comprehensive experiments demonstrate that this new framework significantly improves
2

---

From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models
supervised label Vsup'
These limitations call for rethinking existing feature interac- tion models in CTR prediction and motivate resort to generative approaches.
(Bcre (9mter ((u;)))
where v, Rd is the embedding for feature i,d denotes the embedding dimension. Taking the classic DCN V2 (Wang al., 2021) model as example, it can be formalized as:
3. Method
3.1. Revisit Generative Paradigm for Recommendation
(Vsup; DNN ΣΣΣ N
By O
1 M u
F(0-
(2)
Next-item prediction is one of the most popular genera- tive paradigms, which predicts the next item in a sequence based on preceding inputs, and is widely adopted in re- cent generative recommendation models (Zhai et al., 2024; Rajput et al., 2023). Recently, VAR (Tian et al., 2024) re- considered the "order" of images and proposed a next-scale prediction paradigm. It designs generative frameworks that align with inherent data characteristics, i.e., the "multi-scale, coarse-to-fine nature" of images. As long as the data or- der Zsource Ftarget is defined, generative model can be applied to generate Furrget upon Tsource.
where L denotes the number of cross layers; I denotes the layer index; N denotes the total number of features; i and j denote the feature indices; v denotes the embedding of the j-th feature in the (-th layer; M P()) denotes the projection matrix between the F(i) and F()) field pair in the (-th layer; and F(i) and F(j) denote the field of the feature i and j. respectively.
(1)
Limitations of the Discriminative Paradigm. Despite the great success of many existing discriminative models for CTR prediction (Rendle, 2010; Juan et al., 2016; Pan et al., 2018; He& & Chua, 2017: Guo et al., 2017: Lian et al., 2018; Sun et al., 2021; Wang et al., 2021), they still suffer from the following issues:
This inspired us to reconsider the "order" or the "inherent data structure" in CTR prediction. The data used in CTR prediction is the multi-field categorical data (Zhang et al., 2016; Pan et al., 2018), with categorical features from the user, item, and context sides. Even though there are hierar- chies between some features, for example, the item ID belongs f fine Eine-grained category ID, and the latter belongs to a coarse-gra ned category ID, there is no global partial order between all features.
1. Dimensional Collapse due to raw ID embedding Inter- action. Existing CTR models usually capture the correla- tion between features via the feature interaction function. The feature embeddings in these models tend to span a low-dimensional space due to the Interaction Collaase-- Theory (Guo et al., 2024): feature fields with low in- formation abundance, i.e., with collapsed dimensions, constrain the information abundance of other fields.
However, the most essential characteristic of multi-field categorical data is the co-occurrence of features (Rendle, 2010; Koren et al., 2009). Denote the active features of the s-th sample (i.e., an explicit click an implicit non-click) as (() {z.). the joint occurrence of each feature pair, such as (I;, Ti e X²() denotes the collaborative signal regarding users' explicit or implicit feedbacks. Such co-occurrence of features is the inherent "order", or more precisely, the inherent "structure" of data in CTR prediction.
2. Limitation to learn data distribution. Discriminative paradigms learn the distribution P X') while ignor- ing P()). focusing solely on establishing a decision boundary for classification (Harshvardhan et al., 2020; Oussidi & Elhassouny, 2018). This makes these models unable capture the rich co-occurence correlation be- tween the input (features), and hence limits the quality of the learned representations.
3.2. The Feature-Generation Paradigm
To this end, we propose treating one side of the co- occurrence features as the Source and the other side as the Target. Specifically, given all feature pairs {(x,, (v) and their corresponding embedding ((v,, (i,), we define Twauree and narget as:
3. Information redundancy. Redundaany-rrduuction reduccioo prin- ciple (Barlow et al., 1961) has been fruitful in different application domains (Barlow, 2001; Grill et al., 2020; Zbontar et al., 2021). This principle necessitates mini- mizing information redundancy between the two views, that is, their mutual correlation. We have empirically verified this principle in Sec. 4.3.2, and find that models with aadundancy-redueee interacted embeddings achieve better recommendation performance (Fig, 4). Existing models mainly use the same raw ID embeddings in the in- teraction function, exhibiting a strong tendency towards learning redundant representations.
Esource = (v; i
(4)
Fuarget {u, j j (1,.....,,
(5)
where N is the number of features, and vs and by is the embedding vector of feature and j.
Given Isource and Tragett we employ an Encoder
I Sencuder: 2 to transform Foource into hidden space
3

---

From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models
The feature generation framework builds en- coder based on all features as the dsource. generates an output embedding. and utilizes it to predict all features simultane- ously as the Tmaget For multi-layer generation, generated representations in each layer will serve the Zyounee and Funget in the next layer gemeration. Specifically, the encoder is implemented as field-wise single-layer non-linear MLP, while the decoder is implemented with feature interaction functions in previous CTR models.
Figure 2.
Non- tnnel
Pascoderr
Inneraction Functions
Escodar
(10 (%,
Z to build new representations for each feature. We then employ Decoder f : 2 - V to map the hidden space Z back to the original input space V/ to generate all features. Taking feature i as an example, the encoder constructs new representation 2; for i, and the decoder further maps 2; to generated representation ( ((ccoder(()))) or Vz-j for abbreviation. Notably, each encoded feature representation generates all features, i.e., { simultancously, which fol- lows an All Predict All framework. Implementations of the Encoder and Decoder will be detailed in Section 3.3.
3.3. Architecture Design
Encoder
The encoder transforms the input into a hidden space, generating new representations for each feature. We employ simple field-wise single-layer non-linear MLP, which takes all feature representations as input, then applies field-wise projection matrix Wr and a ReLU activation function. Specifically, the encoder for feature is:
feacodas, ([v]) or ([(=))=_]]wrg
(6)
Finally, a classifier is employed to calculate the relevance (e.g.. dot or Hadmard product, or a simple concatenation) between the output of the decoder, i.e., Vı--j. and the target representation ty. pooling these results across all feature pairs {(i,)) J) and then get the final prediction score. The choice of classifier largely depended the architecture of the original CTR model, and corresponds to the Layer Pooling and Layer Aggregator in (Kang et al., 2025). For example, in the generative FM, we just need to sum up the inner products between Uzzaj and By across all feature pairs, i.e., i = o(E; (1; 02)), where a is the sig- moid function. We ignore the linear and bias terms here for simplicity.
where a is the ReLU activation function, [[ RNd denotes the concatenation of all feature embeddings! F(i) denotes the field of feature i, and Wro) E Ndxd denotes the projection matrix for the field of feature The Eq. 6 fol- lows the principle of minimal sufficient complexity: ablation of any constituent element results in significant performance deterioration, whereas augmentations with more sophisti- cated components fail to demonstrate performance gains. Refer to Sec. 4.4 for details.
The decoder aims to transform the encoder out- put in hidden space 2 back to the original space V. It may consist of multiple stacked layers, with each layer conduct- ing the space mapping through a projection matrix W:
Decoder
The entire generative paradigm is formalized in Eq. 3 and depicted in Fig. 2. In Eq. 3. we first employ fencoder, to trans- form sourres (the set of all feature embeddings (1t ) to obtain hidden representation of feature i. faecoder,_.. further maps the hidden representation back to the original input space. Then, a classifier fas will be used to calculate the final prediction score. Finally, the entire model will be optimized with supervised label Vsup
fdecoder, (z:) z, W.
Notably, W with different properties correspond to vari- ous feature interaction functions( (Wang et al., 2021; Kang et al., 2025). For example, a field-pair wise scale iden- tity matrix W
For single-value fields, we can simply concatenate the em- beddings of the active features from them; while for multi-value fields, need to first aggregate the multiple values for each field through sum or mean pooling.
E.g., fencoder, ([e)) o[[fvx (ll Supervised label
T Tsource
Starget
C( Ysup fers faccodor...,
fencoder,
(1 N 1), Vj
(3)
E.g., fdecuder,.., WF(O)-F()) KKKK
Classifier
4

---

From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models
FF)) correspond to the fielh-weighted interac- tion function used in FWFM (Pan et al., 2018) and xDeepFM (Lian et al., 2018), while a field-pair wise full matrix W := W F(j) correspond to the one used in FmFM (Sun et al., 2021) and DCN V2 (Wang et al., 2021), ) Using FmFM as an example, it can be reformulated into our feature generation paradigm as:
2019; Mao al., 2023; Chang et al., 2023), which treat it as gating mechanism to generate attentive weights. For exam- ple, Fibinet (Huang et al., 2019) introduced a SENET mecha- nism, which also resembles our encoder, to "pay more atten- tion to the feature importance". Inspired by LHUC (Swieto- janski et al., 2016), PEPNet (Chang et al., 2023) proposed a Gate Neural Unit to "personalize network parameters", which takes the domain-side features as inputs, conducts nonlinear activation, and then interacts with the results with the embeddings of the backbone models. Mao et al. (2023) employed context-aware feature aging layer for feature selection. Our work offers an alternative representation learning interpretation of these methods.
C
V,
(olluu))_lWWoooWooooon Oo, 1
More formal generative reformulations of popular CTR models can be found in Appendix A.
4. Experiments
In this section, aim to address these research questions:
3.4. Loss
Existing generative approaches for recommendation usually employ self-supervised losses (Kang & McAuley, 2018: Zhai et al., 2024). For example, next-item prediction treats the next item as an unsupervised label and adopts a cross- entropy loss to evaluate the generation quality. However, in our feature generation paradigm, we need to be careful when we try to employ such a self-supervised loss, For example, if we follow the next-item prediction, and use a similar "another feature prediction" loss, that is, utilize the encoder to predict whether the another feature is true or not, then it would lead to label leakage since the true "another feature" " is already in the input. Instead, employing supervised loss in paradigm would force the encoder to learn the collaborative information about the input regarding the supervised label Usup-
To what extent can the paradigm shift improve existing discriminative feature interaction models?
RQ1:
Can the generative paradigm mitigate the inher- ent drawbacks of raw ID embeddings in discriminative paradigms, specifically in terms of embedding dimen- sional collapse and information redundancy reduction?
RQ2:
RQ3: Is the current paradigm design optimal for feature generation? ? What will happen if we use different X Foouree Encoder, or Fuaret?
4.1. Setup
Datasets & Evaluation protocols. In this work, we have conducted experiments based on two widely adopted large- scale datasets, namely Criteo (cri, 2014) and Avazu (ava, 2014). Dataset statistics are summarized in Appendix B. As for evaluation, we evaluate the recommendation perfor- mance with AUC and Logloss.
3.5. Discussion
Representation Learning Our core contribution is to em- play an Encoder network to build new representation for each feature, thereby overcoming the limitation of original ID embeddings (Guo et al., 2024). Many existing works also pay attention to constructing new representations. Guo et al. (2024) proposed build several independent embeddings (o(m) (m)) for each feature i by constructing multiple indepen- dent embedding tables, and such multi- cablddinmbedingg paradigm can be further attributed back to FFM (Juan et al., 2016). Rajput et al. (2023) proposed first map the LLM em- bedding of each feature into hidden space through an RQ- VAE, and then quantize the hidden representation to discrete Semantic IDs. We study the quality of these newly built em- beddings by the dimensional collapse analysis (Sec. 4. and redundancy-reducttoo analysis (Sec. 4.3.2).
Baselines. To verify versatility of our method, we in- tegrate it with various representative models, including explicit feature interaction models FM (Rendle, 2010),
FmFM (Sun et al., 2021). CrossNetv2 (Wang et al., 2021). and DNN-based models DeepFM (Guo IPNN (Qu et al., 2016), xDeepFM (Lian et al., 2018), DCN V2 (Wang et al., 2021). All experiments are based on pop- ular library FuxiCTR (Zhu et al., 2020; 2022). More details can be found in Appendix B.2. Besides, the computational complexity are provided in Sec. B.4.
al., 2017).
4.2. Recommendation performance comparison between discriminative and generative paradigms (RQ1)
Relationship with Gating Mechanism Our encoder can be seen as a generalization to existing works (Huang et al.,
Offline results. We apply the feature generation frame- work with various recommendation models, with results
5

---

From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models
Table Recommendation performance of models with the DiSeriminative (DIS) and GENrative (GEN) paradigm. We conduct two-tailed T-test to calculate the statistical significance, with results presented in form of (heannvariance). Bolded values refer to the best performance, and means the corresponding p-values are less than 0.05,
Critco
Avazu
Model
AUC1
Logloss
AUC1
Logloss
DIS
0.80236(9c-05) 0.44889(7c-05)
0.78877(1e-04)
0.37529(4c-05)
FM
GEN
0.81108(1e-04)*
0.44077(1e-04)
0.79260(1c-04)+
0.3727966c-05)
Explicit
DIS
0.80552(3e-04)
0.44626(3e-04)
0.78990(2e-04)
0.37519(7e-04)
FmFM
GEN
0.80992(7e-04)** 0.44258(8e-04)*
0.79266(9e-05)*
0.37287(2e-04))
DIS
0.81312(1c-04)
0,43918(2c-04)
0.79106(1c-04)
0.37319(2c-04)
CrossNet V2
GEN
0.81540(4e-05)* 0.43661(5e-05)*
0.79301(2e-04)#
0.3720065c-05)
DIS
0.81380(8e-05)
0.43804(6e-055
0.79285(1e-04)
0.37224(1e-04)
DeepFM
GEN
0.8139666e-05)* 0,43788(5e-05)*
0.79333(7c-05)
0.37181(1c-04)
DNN-basel
DIS
0.81365(1e-04)
0.43819(1e-04)
0.79222(1e-04)
0.37246(5e-05)
xDeepFM
GEN
0.81421(7e-05)* 0,43775(9e-05)
0.79429(1e-04)*
0.37123(7e-05)
DIS
0.81341(5c-05)
0,43850(2c-05)
0.79348(3c-04)
0.37159(1c-04)
IPNN
GEN
0.81415(8e-05)* 0.43776(1e-04)*
0.79451(86-05)9
0.37105(1e-04)
DIS
0.81387(6c-05)
0.43826(4c-05)
0.79282(2c-44)
0.37222(1c-4))
DCN V2
GEN
0.81472(6e-05)*
0.43713(5e-05)
0.79342(5e-05)
0.37180(5e-05))
presented in Tab.4. Overall, the proposed method ex- hibits promising effectiveness and achieves consistent per- formance lift across different models, achieving an average of 0.272% AUC lift and 0.435% Logloss reduction. Usually, a 0.1% AUC (gAUC) lift regarded as a huge improvement recommendation systems (Zhu al., 2022).
coefficient of variation for each paradigm, and Fig, 6 shows that the results of the generative paradigm are markedly smaller, corroborating its ability to reduce the gap between different models. These findings further underscore the importance of the paradigm shift.
In summary, our framework consistently improves the per- formance of various existing feature interaction models der the original discriminative paradigm and markedly nar- rows inter- anter-architectural performance gaps.
Specifically, the generative paradigm explicit feature in- teraction models can bring an average of 0.428% AUC lift and 0.689% Logloss reduction. Notably, DCN V2 incorpo- rates DNN based on CrossNet, enhancing its modeling capability. The discriminative DCN V2 surpasses the dis- criminative CrossNet by 0.157% lift AUC and 0.235% in Logloss reduction. Surprisingly, when CrossNet refor- mulated within generative paradigm, it can outper- form the discriminative DCNv2 by 0016 lift in AUC and 0.089% reduction in Logloss, verifying the promising poten- tial of generative paradigms. For DNN-based models, the improvement is less pronounced. Nevertheless, even with complex DNN-based models, the paradigm shift still brings significant enhancements, achieving an average improve- ment of 0 16% in AUC and 081% in Logloss reduction.
Online A/B Testing. We deployed the proposed genera- tive paradigm one the world's 's largest advertising plat- forms. The production model employs Heterogeneous Ex- perts with Multi-Embedding architecture (Guo et al., 2024; Su et al., 2024; Pan et al., 2024). We switch the IPNN expert in the production model into a generative paradigm, which models the interactions between more than five hundred user-, ad-, and context-side features. During the one-week 20% A/B testing, demonstrated promising results, achiev- ing 2.68% GMV lift and 2.46% CTR lift on several vital scenarios, including Moments pCTR, Content and Platform PCTR, and DSP PCTR. These improvements were statisti- cally significant according to t-tests. The proposed feature generation framework has been successfully deployed as the production model in the above- mentioned scenarios, leading to a revenue lift by hundreds of millions of dollars per year.
Moreover, our generative paradigm can narrow the per- formance gap caused by different model architectures. For example, the strongest discriminative model (DCNv2) surpasses the weakest (FM) by 1.151% AUC on Criteo, whereas the gap between their generative counterparts shrinks to 0.364%. In detail, Appendix B.3 compares the
6

---

From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models
Liger hwwebbo 2ag.arrr..rrg. 1004
Upgart m Sor Stg.
-5
10
-10
-10
-10
Avazu
Avazu DIS
Arzzu D16
Avazu DIS
AvacU-GIN
15
Arszu- GEN
Crite-ODi5
Ligarg
Criteo- DA5
Criee-D15
OriteoGENN
Drite--GEN
Crtec-EEN
100
260
150
400
1ão
300
450
150
203
350
460
100
200
100
460
Singuler Value Ranc indax
Singuar Va up Rank Index
Singular Va us Ranc Indux
Singular Value Rank iddx
(a) FM
(b) DeepPM
(c) CrossNet
(d) DCN V2
Figure 3. Normalized singular value spectrum of embeddings used to interact with raw embeddings. It the concatenation of raw ID embeddings for the discriminative paradigm, while the embedding immediately constructed by the encader for the generative paradigm.
4.3. How does the generative paradigm work? (RQ2)
1 x 10 However, they drop dramatically to 1 10015 index 280, a reduction of 1010 times. This indicates an ex- treme imbalance among dimensions, i.e. only minority of dimensions dominate the embedding space. After index 280, the singular values remain around 1 x 10-15, essentially zero These singular values account for approximately 30% of the total singular values, implying that 30% of the dimen- sions in the embedding carry no meaningful information, which is clearly unfcasible.
4.3.1. GENERATIVE PARADIGM MITIGATES EMBEDDING DIMENSIONAL COLLAPSE
Dimensional collapse evaluation protocols. Dimen- sional collapse means that the embeddings only span low-dimensional subspace of the available representation space (Jing et al., 2021; Guo et al., 2024). With a slight abuse of notation, we will detail how to measure the di- mensional collapse issue by singular value decomposition. Specifically, we evaluate the dimensional collapse issue at the sample level. We begin by obtaining the sample em- bedding matrix Z E B using the validation dataset, where B denotes the batch size and d the dimension size (Notably, this batch-wise setting will greatly enhance the analysis efficiency, and we have verified the robustness of this setting in Appendix B.5). The covariance matrix is then derived as C' + == B
These phenomena are significantly mitigated in the gener- ative one. With the exception of FM, the singular value spectra of the other methods do not exhibit the abrupt de- cay mentioned earlier. Instead, they decline at a relatively slower rate, indicating a more balanced embedding space. Even for simple models like FM, our generative paradigm can increase the number of meaningful dimensions by 25%. We attribute this improvement to the integration of all fea- ture fields when constructing embeddings using the feature generation framework. We conclude the following result:
(z; - z)², with values S =i=1 Zi- Subsequently, we determine the singular diag(a*) of C' via singular value decomposition (SVD) and normalize them by the maximum singular value: S' diag ) Finally, we present these normalized singular values in descending order, as shown in Fig. 3.
Result 2. The generative paradigm substantially miti- gates the issue of embedding dimensional collapse.
4.3.2. REDUNDANCY REDUCTION VIA GENERATIVE FEATURE LEARNING
Evaluated embeddings. We focus on the direct impact of the feature generation framework on the embedding space. Specifically, we analyze the embedding used interact with raw ID embeddings. In the discriminative paradigm, it is the concatenation of raw ID embeddings, formally defined as [(>x)] }R In the generative paradigm, it is the concate- nation of feature embedding immediately constructed by the Encoder, i.e., [=1 We study this embedding to investigate the direct influence of the generative paradigm.
Information redundancy evaluation protocols. Accord- ing to the information redundancy reduction principle (Bar- low et al.. 1961; Zbontar et al., 2021), the two interacted embeddings are expected to exhibit low correlation. To quantify this, we employ the Pearson Correlation Coeffi- cient between each dimension of the two interacted embed- dings, defined as px.y Cov(X,Y) For the discriminative paradigm, X and Y are the two interacted embeddings, while X and Y are the transformed dssurce and Turget embed- dings for the generative paradigm. sx and sy denote their respective standard deviations.
sxsv
Generative paradigm mitigates dimensional collapse. For brevity, we illustrate the singular value spectrum of the embedding space for four representative models in Fig. 3. Visualization of all models can be found in Appendix C.I. In each sub-figure, the spectrum exhibits a rapid decay. Tak- ing Fig. 3d as an example, the singular values of DCN V2 on Criteo remain high up to index 250, with values around
Negative connection between redundancy metric and recommendation performance. We have visualized the correlation matrix of FM, DeepFM, DCN V2 in Fig. 4. More visualizations are provided in Appendix C.3. In Fig. 4a,
7

---

From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models
Conelation behaeenn
Cormlattn belween
Fearson Cormittion battomen Tash
Peareet Carelatics bebueen Baid
1.09
3.00
L0
19
1676
5.6.
4n
6.50
029
059
130
140
130
130
(a) FM (DIS)
(b) DeepFM (DIS)
DON V2 (DIS)
(d) DCN V2 (GEN)
Figure+ 4. Pearson correlation matrix between two interacted embeddings. (a)-(b)- means more complex models, which also exhibits a trend redundancy reduction. This reveals the importance of information redundancy reduction when designing CTR models. In (d), we can find the generative DCN V2 almost produces a zero correlation matrix, perfectly aligning with the redundancy reduction principle.
we have derived two major observations: (1) Intra-field correlation. It forms some obvious diagonal blocks, while each block corresponds to feature field. This means the information within a feature field is highly correlated, i.e., redundant information. (2) Inter-field correlation. The index 32 - 160 forms a big diagonal block, which is exactly the correlation between field with index 3 10. This means these feature fields also highly correlated, violating the redundancy reduction principle. These observations may explain the inferior performance of FM.
Ablation on the Tworre design. As stated in Discussion of discriminative paradigms, one of their major limita- tions is the direct interactions between raw ID embeddings, especially those of low-cardinality fields. To tackle this issue, we propose to utilize all field embeddings as Tsource to build new representations for all fields. For comparison, we will investigate the following configurations to reveal the significance of our design: (a) using only each field's embedding as Isouce for all fields; (b) using all field embed- dings as Tsurre for 10 fields with the highest cardinality; and (c) using all field embeddings as 2source for 10 fields with the lowest cardinality. Results are presented in Fig. 5a.
Then we analyze by comparing different models. DeepFM builds a parallel DNN upon FM, which greatly decreases inter-field correlation and increases recommendation per- formance. DCN V2 further incorporates a more advanced explicit feature interaction module based on DeepFM. In Fig. 4c, the diagonal blocks representing intra-field correla- tion are almost reduced, which explains the recommendation performance lift. All these results reveal negative connec- tion between the redundancy metric and recommendation performance, which can guide model designing.
We can observe that using all field embeddings as sosuree outperforms other settings, revealing the necessity of con- structing embeddings with all features. Additionally, the results of only constructing www-cadinality fields with all features are significantly better than the high-cardinalin counterpart, which corroborates our previous assertion that low-cardinality fields suffer more from severe information insufficiency. We conclude the following result:
Result 4. Treating all feature fields Isowce is effective for feature generation. In particular, low- cow-cardinality field embeddings suffer from more severe issues than high-cardinality field ones, underscoring the impor- tance of constructing new embeddings for these fields.
Generative paradigm reduces information redundancy. Despite the transformations applied to raw ID embeddings in DCN V2, we still observe correlations in Fig. 4c. In con- trast, the correlation matrix is nearly a zero matrix in Fig. 4d, indicating that the two vectors are highly de-correlated and thus adhere to the redundancy reduction principle. This demonstrates our framework's 's ability to reduce information redundancy effectively. We conclude the following result:
Ablation on the Encoder design. The adopted Encoder is field-wise one-layer non-linead MLP. To further inves- tigate its properties, we first construct the following model variants: (b.1) using a field-shared MLP, (b.2) removing non-linear activations, and (b.3) stacking one more layer. In Fig. 5b, simplifying the Encoder with either (b. 1) or (b.2) leads to significant performance degradation. The former underscores the importance of constructing distinct embed- dings for different fields, which aligns with our intuition. The latter highlights the necessity of modeling non-linear relationships among features, which also contributes sig- nificantly to alleviating the dimensional collapse issue, as
framework Result 3. The feature framework produces embeddings highly de-correlated with raw II) embed- dings, adhering to the redundancy reduction principle.
4.4. Ablation on the feature generation framework design (RQ3)
For simplicity, all ablation studies are based on DCN V2.
8

---

From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models
for on petraiyyes
9-4 stusing Seet
relimia madki proii SaY Rebic al buck
0.7985
0.7935
0.7935
3 0.7930
0.7910
0.7911
0.7925
0.7925
0.7925
0,7920
0.7520
0.7923
OUT oPent modd varants
Offerent model vananta
Eiffarant mode vananns
(a) Ablation on source
(b) Ablation on encoder
(c) Ablation on largef
Figure 5. Ablation study on the feature generation framework design using DCN V2 on Avazu.
verified in Appendix C.4. On the other hand, increasing the complexity of Encoder with (b.3) even greatly degrades recommendation performance, AUC decreases from 0.7935 to 0.7929, which may be caused by over-fitting
outperforms all others, demonstrating its superiority.
5. Related Works
Feature-interactiinn based recommender systems. De- signing improved feature interaction models has consistently represented a significant area of research within the field of recommender systems (Zhang et al., 2019; Cheng & Xue, 2021). A key focus in the advancement of modern recom- mendation systems the development of more sophisticated feature interaction modules, including first-order (Richard- son et al., 2007), second-order (Rendle, 2010; Pan et al., 2018; Sun et al., 2021), and high-order interactions (Lian et al., 2018; Wang et al., 2021; Li et al., 2024a). With the rise of deep learning, Deep Neural Networks (DNNs) with non-linear activation functions have been integrated into recommendation systems to capture implicit high-order fea- ture interactions (Cheng et al., 2016: Guo et al., 2017; He Chua, 2017; Lian et al., 2018; Wang al., 2021). In addition to incorporating non-linearity in DNNs, several studies have explored the introduction of non- non-linearity in embeddings through gating mechanisms, such as FiBiNET (Huang et al., 2019), FinalMLP (Mao et al., 2023), and PEPNet (Chang et al., 2023). Orthogonal to these works, we propose a novel Supervised Feature Generation framework for CTR models, shifting from a discriminative "feature interaction" paradigm to generative "feature generation" paradigm.
Next, we investigate whether other generative models are feasible: (b.4) self-attention networks. In Fig. 5b, we ob- serve that both (b.4) outperform the original discriminative paradigm but underperform the generative paradigm with the MLP-based encoder. This confirms the effectiveness of the generative paradigm and further demonstrates that encoder is simple yet effective design for constructing meaningful embeddings. We conclude the following result:
Result 5.
The field-wise non-linear one-layer MLP is a simple yet effective Encoder. Common modifications, including simplification or increased complexity. lead to inferior recommendation performance.
Ablation on the Ttarget design. In our paradigm, we gener- ate all features simultancously. We compare different imple- mentations of uurget: (c.1) ppredict-randowd meedomm-eeeeted.. gen- erating only randomly selected feature fields; (c.2) "masked feature modeling": randomly masking some fields in Zsource with learnable mask vector and predicting them as Ptarget> akin to masked image modeling (He et al., 2022); (c.3) "warld-aware masked feature modeling": similar to (c.2) but using field-specific mask vectors; (c.4) "hard masked feature modeling": similar to (c.2) but with zero vectors as mask. Formal definitions are detailed in Appendix B.6.
6. Conclusion
The results are depicted in Fig. 5c. In the figure, all paradigms outperform the discriminative approach except (c.2), which we attribute to the superior feature distribution modeling ability of generative paradigms. For (c.2), the semantic gap between different feature fields renders using single mask vector for all fields inherently impractical. Therefore, adopting (c.3) with a field-aware mask signifi- cantly improves performance. Counterintuitively, zero vector outperforms learnable mask vectors. We hy- pothesize this discrepancy stems from differences between unsupervised and supervised generative paradigms. Using a learnalle mask vector with supervised signals may impede feature distribution learning. Our "Predict All" paradigm
In conclusion, this work introduced a novel Supervised Feature Generation framework that shifts CTR modeling from discrininative feature interaction to generative feature generation. The framework's 's versatility through reformulating various existing feature interaction models into generative ones, ranging from explicit inter- action models to complex DNN-based models. It could produce feature embeddings with reduced collapse and re- dundancy compared to raw ID embeddings.
demonstrated
fixed
9

---

From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models
Acknowledgements
Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 1-, 2020, virtual, 2020.
This work was supported by the National Natural Science Foundation of China (62441239, 62472394, U23A20319, 62441227, 62202443) as well the Anhui Province Science and Technology Innovation Project (202423k09020011),
Guo, H., Tang, R., Ye, Y., Li, Z., and He, X. Deepfim: machine based neural network for ctr pre- diction. arXiv preprint arXiv: 700. 2017.
Guo, X., Pan, J., Wang, X., Chen, B., Jiang. J., and Long,
Impact Statement
M. On the embedding collapse when scaling up recom- mendation models. ICML, 2024.
This paper presents work whose goal is to advance the field of Machine Learning. The research outcomes may have various societal implications, such as optimizing online ad- vertising strategies or enhancing e-commerce functionalities, none of which we feel must be specifically highlighted here.
Harshvardhan, G., Gourisaria, M. K., Pandey, M., and
Rautaray, S. S. A comprehensive survey and analysis of generative models in machine learning. Computer Science Review, 38:1 08285, 2020.
References
He, K., Chen, X., Xie, S., Li, Y.. Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Avazu dataset.
bttps://ww.kagglL. com/
data, 2014.
He, X. and Chua, T.-S. Neural factorization machines for sparse predictive analytics. In Proceedings of the 40th International ACM SIGIR conference research and development in information retrieval (SIGIR), pp. 355- 364, 2017.
Criteo dataset. https://www.kagghtt com/ c/
criteo-dispe sipoodddsslya----e-dda
ad- cha ch- data, 2014.
Barlow, H. Redundancy reductionrevisited. Network: com-
putation in neural systems, 12(3):241, 2001.
Barlow, H. B. et al. Possible principles underlying the trans- formation of sensory messages. Sensory communication, 1(01):217-233, 1961.
Huang, T., Zhang, Z., and Zhang, J. Fibinet: combining feature importance and bilinear feature interaction for click-through rate prediction. In Proceedings of the 13th ACM conference recommender systems (Recsys), pp. 199,
Chang, J., Zhang, C., Hui, Y., Leng, D., Niu, Y., Song, Y. and Gai, K. Pepnet: Parameter and embedding per- sonalized network for infusing with personalized prior information. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1795-3804, 2023.
Jing, L., Vincent, P., LeCun, Y., and Tian, Y. Understand- ing dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.09344. 2021.
Juan, Y., Zhuang, Y., Chin, W.-S., and Lin, C.-J. Field-aware factorization machines for ctr prediction. In Proceedings of the 10th ACM conference recommender systems (RecSys), pp. 43-50, 2016.
Cheng, H.-T., Koc, L.,, Harmsen, J., Shaked, T., Chandra, T., Aradhye, H., Anderson, G., Corrado, G., Chai, W., Ispir, M., et al. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems, pp. 7-10, 2016.
Kang. W.-C and McAuley, J. Self-attentive sequential recommendation. In 2018 IEEE international conference on data mining (ICDM), pp. 197-206. IEEE, 2018.
Cheng, Y. and Xue, Y. Looking at etr prediction again: Is attention all you need? In Proceedings of the 44th international ACM SIGIR conference on research and development in information Retrieval, pp. 1279-1287, 2021.
Kang, Y., Pan, J., Jin, J., Huang, S., Gao, X., and Xiao, L. Towards unifying feature interaction models for click- through rate prediction. ECML-PKDD, 2025.
Grill, J., Strub, F., Altché, F., Tallec, C., Richemond, H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z., Azar, M. G., Piot, B., Kavukcuoglu, K., Munos, R., and Valko, M. Bootstrap your own latent A new approach to self- supervised learning. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual
Koren, Y.. Bell, R., and Volinsky, C. Matrix factorization techniques for recommender systems. Computer, 42(8): 30-37, 2009.
Li, H., Zhang, Y., Zhang, Y., Li, H., and Sang, L. Denv3: Towards next generation deep cross network for ctr pre- diction. arXiv preprint 4xXiv:2407.13349 2024a.
10

---

From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models
Li, T., Tian, Y., Li, H., Deng, M., and He, K. Autoregres- sive image generation without vector quantization. arXiv preprint arXiv:2406.11833 2024b.
Richardson, M., Dominowska, E., and Ragno, R. Predicting clicks: estimating the click-through rate for new ads. In Proceedings of the 16th international conference world wide web (WWW), pp. 521-530. ACM, 2007.
Li, Z., Cui, Z., Wu, S., Zhang, X., and Wang. L. Fi-gnn: Modeling feature interactions via graph neural networks for ctr prediction. In Proceedings of the 28th ACM in- temaationa conference on information and knowledge management, pp. 539-548, 2019.
Su, L., Pan, J., Wang, X., Xiao, X., Quan, S., Chen, X., and Jiang, J. Stem: Unleashing the power of embeddings for multi-task recommendation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 9002-9010) 2002-9010,40 2024.
Lian, J., Zhou, X., Zhang. F., Chen, Z., Xie, X., and Sun, G. xdeepfm: Combining explicit and implicit feature interactions for recommender systems. In Proceedings of the 24th ACM SIGKDD international conference knowledge discovery and data mining (SIGKDD), pp- 7754-17655 2018.
Sun, H., Yu, G., Zhang, P., Zhang, B., Wang, X., and Wang, D. Graph based long-term and short-term interest model for click-through rate prediction. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp.
2022.
Mao, K., Zhu, J., Su, L., Cai, G., Li, Y., and Dong, Z. Finalmlp: An enhanced two-stream mlp model for ctr prediction. arXiv preprint vrXiv:2344.00900 2023.
Sun, Y., Pan, J., Zhang, A., and Flores, A. Fm2: Field- matrixed factorization machines for recommender sys- tems. In Proceedings of the Web Conference (WWW), pp. 2021.
Oussidi, A. and Elhassouny, A. Deep generative mod- els: Survey. In 2018 International conference on in- telligent systems and computer vision (ISCV), pp. 1-8. IEEE, 2018.
Swietojanski, P., Li, J., and Renals, S. Learning hidden unit contributions for unsupervised acoustic model adap- tation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(8):1450-1463, 2016.
Pan, J., Xu, J., Ruiz, A. L.,, Zhao, W., Pan, S., Sun, Y., and Lu, Q. Fiegh-weighted factorization machines for click- through rate prediction in display advertising. In Proceed- ings of the 2018 world wide web conference (WWW), pp. 1349-1357, 2018.
Tian, K., Jiang, Y., Yuan, Z., Peng. B., and Wang, L.. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint urXiv:2404.02900 2024.
Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Pan, J., Xue, W., Wang, X., Yu, H., Liu, X., Quan, S., Qiu, X., Liu, D., Xiao, L., and Jiang, J. Ads recommendation in collapsed and entangled world. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5566-5577, 2024.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems (NeurIPS), 30, 2017.
Qu, Y., Cai, H., Ren, K., Zhang, W., Yu, Y., Wen, Y., and Wang, J. Product-based based neural networks for user response prediction. In 2016 IEEE 16th international conference on data mining (ICDM), pp. 1 1149-1154. IEEE, 2016.
Wang, F. Gu, H., Li, D., Lu, T., Zhang, P., and Gu, N. Mcrf: Enhancing ctr prediction models via multi-channel feature refinement framework. International Confer- ence on Database Systems Advanced Applications, pp. 359-374. Springer, 2022a.
Rajput, S., Mehta, N., Singh, A., Keshavan, R. H., Vu, T., Heldt, L., Hong, L., Tay, Y., Tran, V. Q., Samost, J., Kula, M., Chi, E. H., and Sathiamoorthy, M. Recommender systems with generative retrieval. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans,
Wang, F,, Wang, Y., Li, D., Gu, H.,, Lu, T., Zhang, P., and Gu, N. Enhancing ctr prediction with context-aware feature representation learning. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 343-352, 2022b.
LA, USA, December 10 16, 2023, 2023.
Wang. F., Gu, H., Li, Lu, T., Zhang, P., Shang, L., and Gu, N. A comprehensive summarization and evaluation of feature refinement modules for ctr prediction. arXiv preprint arXiv:231 1.04625, 2023.
Rendle, S. Factorization machines. In 2010 IEEE interna- tional conference on data mining (ICDM), pp. 995-1000, 2010.

---

From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models
Wang, R., Shivanna, R., Cheng, D., Jain, S., Lin, D., Hong. L.,, and Chi, E. Den-v2: Improved deep & cross network and practical lessons for web-scale learning to rank sys- tems. In Proceedings of the web conference (WWW), pp. 1785-1797, 2021.
Wang, Y., Qin, Y., Sun, F., Zhang, B., Hou, X., Hu, K., Cheng, J., Lei, J., and Zhang, M. Disenctr: Dynamic graph-based disentangled representation for tlick-through rate prediction. In Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval, pp. 2314-2318, 2022c
Yin, M., Wang, H., Guo, W., Liu, Y., Zhang, S., Zhao, S., Lian, D., and Chen, E. Dataset regeneration for sequen- tial recommendation.
Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. pp. 3954 -965, 2024.
Zbontar, J., Jing, L., Misra, I., LeCun, Y., and Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. Mcila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learn- ing, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 12310-12320. PMLR, 2021.
Zhai, J., Liao, I.., Liu, X., Wang, Y., Li, R., Cao, X., Gao, L.., Gong, Z., Gu, F., He, J., Lu, Y., and Shi, Y. Actions speak sequential trans- ducers for generative recommendations. In Forty-first International Conference Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
louder than words: Trillion-paramee
Zhang, S., Yao, L., Sun, A., and Tay, Y. Deep learning based recommender system: A survey and new perspectives. ACM computing surveys (CSUR), 52(1):1-38, 2019.
Zhang, W., Du, T., and Wang, J. Deep learning over multi- field categorical data: A case study on user response prediction. In Advances in information retrieval: 38th European conference on IR research (ECIR), pp. 45-57, 2016.
Zhu, J., Liu, J., Yang, S., Zhang, Q., and He, X. Fuxictr: An open benchmark for chick-through rate prediction. arXiv preprint arXiv:2009.0579, 2020.
Zhu, J., Dai, Q., Su, L.,, Ma, R., Liu, J., Cai, G., Xiao, X.. and Zhang, R. Bars: Towards open benchmarking for recommender systems. In Proceedings of the 45th International ACM SIGIR conference on research and development in information retrieval (SIGIR), pp. 2912- 2923, 2022.
12

---

From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models
A. Formal reformulation of existing feature interaction models
Tab. 2, we provide the formal definition of how to reformulate existing discriminative models into generative paradigms. Notably, we only present the "feature interaction" or "feature generation" " part in each paradigm for simplicity.
Table 2. Feature interaction models in discriminative & generative paradigm
Model
Discriminative
Generative
- 2j-r+1
M>
ey
Σ 2j-t=1
Wro)) @
FmFM
① le MP()-PO))
>x 2j-111
Wro)
@ [u MP()-Foi
CrossNet V2
Ei= Σmy=1
(o) Ma P()
E -
<[v]''
W7 (
C (o) MM -P)))
DecpHM
D DNN((E]
Σx- 2j-i+1
N
o[[v] W FG)
O o DNN((u])
xDeepEM
Ei= Eij=1 Conv'(
O of)
DNN((e])
Ee -mj= Conv'l [[[v|' W7
о of) DNN((e])
IPNN
DNN([[O].E. Σ
DNN E. E ش [ W Wr() mm)
DCN
E
EN SLEN
of
O (v) M (F))F() DNN((U])
E ش a[(o| w (o) Mo)) DNN((U))
B. Detailed experimental configuration
B.1. Dataset statistics
We adopt the Criteo x1 and Avazu x4 datasets provided by FuxiCTR (Zhu et al., 2020; 2022), whose statistics are summarized Tab. 3.
B.2. Implementation details of baseline methods
We first introduce common settings for all models: (1) For Criteo dataset, the embedding size is set to 10, batch size is set to 4,096, and learning rate is set to le-3. (2) For Avazu dataset, the embedding size is set to 16, batch size is set 10,000, and learning rate is set 1e-3. All experiments will be early stopped when results validation dataset decrease for consecutive two training epochs.
Then we list the detailed setting of different baseline models. Notably, we do not tune these taree--aas when fitting these models into the proposed generative paradigm:
FM: embedding regularization coefficient set to 5.0e-06 for Criteo and 1.0c-06 for Avazu.
FmFM: parameter regularization coefficients are set to 1.0e-06 for the both datasets; we adopt matrixed field embedding transform type (Sun et al., 2021) for both datasets.
CrossNet V2: embedding regularization coefficient is set to 1. 1.0e-05 and 0 for Criteo and Avazu, respectively; number of cross layers is set to 3, 5 for Criteo and Avazu.
DeepFM: embedding regularization coefficient is set to 1. 1.0e-05 and 0 for Criteo and Avazu, respectively: a parallel DNN with size [400, 400, 400) and [2000, 2000, 2000, 2000] are used for Criteo and Avazu, respectively.
Table 3. The number of user-item interactions of the adopted two datascts.
Train
Valid
Test
Criteo
33M
8M
4M
Avazu
32M
4M
4M
13

---

From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models
MeepPMM embedding regularization coefficient is 1.0e-05 and 0 for Criteo and Avazu, respectively; CIN hidden units set to [16, 16] and [276] for Criteo and Avazu, respectively; DNN size is set to [400, 400, 400] and [500, 500, 500] for Criteo and Avazu, respectively.
IPNN: embedding regularization coefficient is set 1.0e-05 and 1.0e-09 for Criteo and Avazu, respectively; DNN size is set [400, 400, 400] and [1000, 1000, 1000] for Criteo and Avazu, respectively.
DCN V2: based on the setting of CrossNet V2, a parallel DNN with size [500, 500, 500] and [2000, 2000, 2000, 2000] are used for Criteo and Avazu, respectively.
All experiments can fit into a GPU with 14GB memories.
B.3. Variation coefficients comparison between discriminative and generative paradigms
In Fig. 6, we have depicted the variation coefficients of the discriminative (DIS) and generative (GEN) paradigm concerning different datasets and recommendation performance metrics.
Criteo AUC
Criteo 1.03%
Logloss
Avazu AUC
Avazu
Logloss
Coefficient of Variation (%)
0.6
0.59%
1.0
0.22%
0.4
0.40%
0.20
0.8
0.3
0.4
0.15
0.6
0.25%
0.50%
0.10
0.09%
0.2
0.19%
0.4
0.2
0.2
0.05
0.0
0.0
0.00
0.0
DIS
GEN
DIS
GEN
GEN
DIS
Model
Model
Model
Model
Figure 6. Variation coefficients comparison between discriminative and generative paradigms.
B.4. Computational complexity analysis
Assuming the original model in the discriminative paradigm has complexity (A), the primary computational overhead when transitioning to generative paradigm arises from the encoder. The encoder is implemented as field-wise non-linear MI., formally defined as:
Pencoder ([v)) = [[u)
where [v] e NK denotes the concatenation of all feature embeddings, and WF()) E RNNK represents a field-wise weight matrix. Consequently, the total encoder complexity becomes dBL where the batch size, L denotes the number of encoder layers, N the number of feature fields, and d the embedding dimension. This computational complexity aligns with mainstream discriminative feature interaction models (e.g., DCN V2), indicating comparable efficiency, Furthermore. demonstrated in the source ablation study (Sec. 4.4), the complexity be reduced to O(BLN d?) by using all field embeddings source only for fields with the lowest cardinality, achieving this optimization with moderate performance trade-offs.
Result 7. The extra computational burden introduced by reformulating existing discriminative feature interaction paradigms to the generative feature generation paradigm is marginal.
B.5. Robustness analysis of the batch-wise setting in Sec. 4.3.1.
In Sec. 1. we have conducted embedding analyses in dimensional collapse with a batch-wise setting, which greatly accelerates the analysis process compared with that based on the full validation dataset. But this batch-wise setting may introduce randomness to the analysis results, so we further provided the analysis of different seeds in Fig. 7. the figure,
14

---

From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models
Table Computational complexity when reformulating. a discriminative feature interaction model into generative feature generation The proposed generative paradigm achieves significant recommendation performance improvements. as detailed in Section 4.2, while incurring only marginal increase in computational overhead averaging 3.14% more computation and 1.45% additional GPU
memory consumption.
Criteo
Avazu
Model
Speed (time/epoch)
GPU memory (MB)
Speed (time/epoch)
GPU memory (MB)
DIS
8m40s
1600
3m08s
2554
FM
GEN
8m58s
1605
3ml3s
2626
Explicit
DIS
9m38s
4846
3m48s
10148
FmFM
GEN
9m45s
4892
3m57s
10180
DIS
5m01s
1050
Im43s
2100
CrossNetv2
GEN
5m09s
1096
Im57s
2190
DIS
8m23s
2090
4m36s
3622
DeepFM
GEN
8m33s
2122
4m44s
3676
DN DNN- based
DNN-basce
DIS
8ml5s
1906
2m43s
3268
DDeepFM
GEN
8m24s
1908
2m45s
3322
DIS
6m37s
1544
3m06s
2592
IPNN
GEN
6m41s
1558
3m14s
2614
DIS
5m31s
1238
5m44s
2982
DCNv2
GEN
5m58s
1282
6m01s
3070
the trend of embedding spectra is consistent across all seeds, demonstrating the robustness of batch-wise analysis setting. Specifically, on both Avazu and Criteo, the spectrum curves of discriminative paradigms exhibit abrupt singular decay from x 10 to 1 x 10-15 15 reduction of t This indicates a severe dimensional collapse issue. But in our generative paradigm, the abrupt singular value decay has been greatly alleviated. This verifies that the generative paradigm substantially mitigates the embedding dimensional collapse issue, forming a more balanced embedding space.
B.6. Formal definition of different target design
We provide detailed formal definition of the different targel designs mentioned in Sec. 4.4.
(c.1) "Prcdictranddom-sceetee""" which generates only randomly selected feature fields:
Σ
Σ
a[vo]
WF() O
(10)
where F Frandom of fields randomly sampled from all fields.
(c.2) "Masked feature modeling": randomly masking some fields in source with learnable mask vector and predicting them as targer, akin to masked image modeling (He et al., 2022):
Σ
Σ
( not masked WF()) j.mnak
(1))
JE(F unmaek F Fmack)
where F Funmask U Fmask F, Ujmmos = mask if in Fmask else Uz.-
(c.3) "Field-aware masked feature modeling": similar (c.2) but using field-specific mask vectors:
Σ
Σ
( not masked W WF(3) O U5.mask
(12)
je(F Fun(Funmn Fmark)
where Funmask U Fmask
F, Uj.mask mask, if Fmask else Vz-
15

---

From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models
Logarthen of Nomaltzd Sinqular Value
Logarbre: of Normalized Singular Singuaar Va
0-
-5
-5
-5
-10
-10
Avazu-DIS
Avazu-DI5
Avazu-DIS
-15
Criteo- DIS
Avazu-GEN
-15-
Avazu-GEN
Criteo DIS
Lopantin
-15
Avazu-GEN
Criteo D.S
Criteo-GEN
-20
Criteo-GEN
Criteo-GEN
20
100
200
300
400
100
200
300
400
Singular Rank
Singular Value Rank Index
Singular Value Rank Index
Logarbim hommatzed Siquuir volur
Logaritim Normltzed Singuler Valse
0
5
-5
-10
-10
10-
AvaZu-DI5
Avazu-DIS
-15
Avazu GEN
-15
Avazu GEN
-15
AvazU-GEN
Critco-DIS
Criteo-DIS
Logerthmm
Criteo D
Criteo-GEN
Criteo-GEN
Criteo-GEN
100
200
300
400
100
200
300
400
100
200
300
400
Singular Value Index
Singular Value Rank Index
Singular Value
Figure Normalized embedding spectrum visualization batch-wise setting different seeds. We can observe that trend embedding spectra is consistent across all seeds, which demonstrates the robustness of our batch-wise analysis setting.
(c.4) "Hard masked feature modeling": similar (c.2) but with vectors mask:
= Σ Σ
o not masked W WF()) ④ Ujjmask
(13)
ja([ ja(Funmase UI mankk)
where Funmask U Fmask F, Uj.mask jf in Faskk else Vz.
C. Supplemental results
C.1. Normalized singular value spectrum visualization of all models
The normalized singular value spectrum of all models are illustrated in Fig. 8. Similar to results concluded in Sec. 4.3.1, the feature generation framework substantially mitigates the embedding dimensional collapse issue, forming a more balanced and meaningful embedding space.
C.2. Dimensional collapse analysis of embedding lookup tables
In Sec. 1,3.1, we focus on analyzing the spectrum of embeddings used to interact with the original embeddings, since we are mainly motivated address the dimensional collapse issue of these embeddings. On the other hand, we can also follow Guo al. (2024) visualize the spectrum of embedding lookup tables, i.e.. D. defined in Sec. 2.2, where denotes one of the feature field, D; is the field's cardinality, and K is the embedding dimension size of the embedding table. The results have been depicted in Fig. 9. In the figure, the spectrum of high-cardinality embedding lookup tables in the generative paradigm is higher than the discriminative one. This indicates the embedding space will be less dominated by some specific dimensions, which will greatly enhance the robustness of these embeddings. However, for those low-cardinality embeddings, the improvement remains limited. This is fundamentally because these field embeddings are inherently constrained by nature. For instance, the number meaningful singular values matrix sized x cannot exceed four.
C.3. Pearson correlation matrix of all models
Similar to Sec. 4.3.2, we provide Pearson correlation matrix of all models on the Avazu dataset in Fig. 13. The conclusion remains the same as in Sec. 4.3.2: (1)There is a strong connection between redundancy reduction metric and recommendation performance: The most simple model FM yields the most matrix with intra-field and fier-field correlations, while the
16

---

From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models
20
-10
-10
110
Avozu B15
Auzzu DIS
Avazu DIS
A/33U DIS
Aeazu-GEN
AuLAu-GE GEN
Catieo-DiS
Oriteo-DIS DIS
Cries-Di5
OfteesDiS Citee-GEN
CrttnoGEN
20
Dri
Crieo-GEN
150
200
160
400
150
203
350
405
100
250
100
400
1000
250
100
400
Singuar Va Rank Insex
Singuar Value Rank indox
Sigglar Value Pank index
S ar Value Ra Indix
(a) FM
(b) FmFM
(c) CrossNet
(d) DeepFM
Lggerte Ohg
-15
-15
-15
-20
150
200
300
400
100
200
300
400
100
2:00
300
450
(c) XDeePFM
(f) IPNN
(g) DCNv2
Figure 8. Normalized singular value spectrum of embeddings used to interact with raw ID embeddings. It is the concatenation of raw ID embeddings for the discriminative paradigm, while the embedding immediately constructed by the encoder for the generative paradigm.
correlation matrix of other models are reduced some extent, depending on whether DNN (DeepFM, IPNN) or more advanced interaction modules (CrossNet V2, xDeepFM, DCN V2). (2)The feature generation framework produces embeddings highly de-correlated with raw IDD embeddings: We observe that the correlation matrices of all models become a nearly zero matrix within the generative paradigm.
C.4. Comparison of different non-linear activation functions
We employ field-wise non-lime con-linear single-layer MLP encoder, with the non-linear activation function being one of its most critical components. A natural question arises regarding the role of the non-linear activation function and the criteria for selecting an appropriate one. We have empirically assessed the effects of various activation functions on the encoder, with the findings illustrated in Fig. 10. As depicted in Fig. 10a, the absence of a non-linear activation function in the encoder results in a notable decline in performance, underscoring the importance of incorporating non- non-lincarity within the encoder. Conversely, all non- non-linear activation functions enhance recommendation performance relative to the discriminative paradigm, with the rank of recommendation performance being Sigmoid < Tanh ReLU SiLU. Furthermore, we present the normalized singular value spectrum of embeddings in Fig. 10b. Initially, the spectrum of the linear activation is highly collapsed, potentially accounting for its inferior recommendation performance. Subsequently, it is observable that the spectra of all non- non-linear activation functions exhibit greater smoothness than that of the discriminative one. This suggests that non-linear activation functions play pivotal role in alleviating the embedding dimensional collapse issue. Additionally, the spectrum adheres to the rank Sigmoid < Tanh < ReLU < SiLU, mirroring the ranking of recommendation performance. This observation further implies a strong correlation between the mitigation of embedding dimensional collapse and the enhancement of recommendation performance.
Result 8. The non-linear activation function is important component of the field-wise MLP encoder, crucial for embedding dimensional collapse mitigation. Besides, many non-lineat activation functions, including Sigmoid, ReLU, Tanh, and SiLU, can get consistent performance lift while mitigating the dimensional collapse.
C.5. Comparison with feature refinement and graph-based models
Some other methods also target enhancing the embeddings of CTR models with field graphs (Sun et al., 2022; Li et al., 2019; Wang et al., 2022c) or feature enhancement modules (Wang et al., 2023; 2022b;a). Our paradigm differs from these works in the sense that we aim to tackle the dimensional collapse issue due to the direct interaction of ID embeddings. We have empirically compared our paradigm with several representative feature refinement models, with results depicted in Tab. 5. We observed that some models outperform the discriminative DCNv2 models, but still underperform our generative
17

---

From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models
0
0
0
DI5 GEN
DIS GEN
DIS GEN
DIS GEN
-5
-5-
-10
10
-10
-10
-5
-15
-15
-20-
-20
-20
-20
15
10
10
(a) Field 24 (D, 4)
(b) Field 19 (D, 6)
(c) Field 14 (D: 6)
(d) Field 13 (D; 7)
0-
0-
0-
0:
Dis
DIS GEN
DI5 GEN
DI5 GEN
GEN
-2-
-1-
-1
-4
-2
-4
-2-
-6-
-3
-3
5
15
5
10
15
10
15
10
15
(c) Field 7 (Di = 6, 545)
(f) Field 12 (D; = 7, 259)
(g) Field 10 (D; = 820, 509) (h) Field 11 (D; = 2, 903, 322)
Figure 9. Normalized singular value spectrum of embeddings lookup tables R R.. where i denotes one of the feature field, D2 the field's cardinality, and K is the emhedding dimension size of the embedding table.
paradigm. Besides, we also studied the singular spectrum in Fig. 11, and we find that the feature enhancement methods can mitigate the dimensional collapse on the tail singular values compared to the vanilla discriminative DCN V2. However, our generative paradigm leads to more robust values across all dimensions.
Table 5. Comparison with other methods that also target enhancing embeddings of CTR models. We have compared with one classic field-graph method Fi-GNN (Li et al., 2019), and two feature enhancement methods GFRL (Wang al., 2022a) and FRNet (Wang al., 2022b). Notably, we also visualize the normalized spectrum of these methods in Fig.
Criteo
Avazu
Model
AUC1
Logloss
AUC1
Logloss
FiGNN
0.81352
0.43845
0.79156
0.37343
DIS
0.81387
0.43826
0.79282
0.37222
GFRL 0.81427
0.043773
0.79296
0.37194
DCNv2
FRNet
0.81431
0.43789
0.79313
0.37191
GEN
0.81472
0.43713
0.79342
0.37180
C.6. T-SNE visualization comparison
In Fig. 12, we have visualized discriminative and generative embeddings with different cardinalities with T-SNE (Van der Maaten & Hinton, 2008). Fig. 12d and Fig. 12h depict embeddings of the highest cardinality field in the dataset, where we observe that the generative embeddings retain the separability as the discriminative paradigm. However, the improvement brought by the generative paradigm is substantial for embeddings of fields with less cardinality, In Fig. 12a, Fig. 12b, and Fig. 12c, the embeddings coalesce in the latent space, even for the field with the second-highest cardinality (Fig. 12c and Fig. 12g). After the generative reformulation, all three embeddings can form a more uniform distribution in the latent space, as illustrated respectively in Fig. 12e, Fig. 12f, and Fig. 12g. These results demonstrate that our generative paradigm can greatly improve the separability of embeddings, especially for embeddings with fewer cardinalities. This also supplements the aforementioned dimensional collapse phenomena analysis from field-wist wise perspective.
18

---

From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models
Logarithm of Normalized Singular Value
DIS
0.7940
Linear Sigmoid
Tanh
-5
0.7935
ReLiJ
DIS
AUC
-10
Linear
0.7930
Sigmoid
0.7925
-15
SiLU
ReLU
0.7920
-20
100
200
300
400
Different model variants
Singular Value Rank Index
(a) Recommendation performance
(b) Singular value spectrum
Figure 10. We have implemented the encoder with different non-ling non-linear activation functions, including ReLU, Sigmoid, Tanh, and SiLU, and providing the corresponding results based on DCN V2: (a) The recommendation performance with different non- non-linear activation functions. (b) The normalized singular value spectrum of the embedding space with different non-linear activation functions.
Logorith Normalized lizedddd Siggular Value
0
-5
-10-
DIS
GEN
-15
FRNet
GFRL
-20
0
100
200
300
400
Singular Value Rank Index
Figure Normalized embedding spectrum feature enhancement methods. can find that these feature euhancement methods can mitigate the dimensional collapse on the tail singular values compared to the vanilla discriminative DCN V2. However, our generative model leads to more robust values on all dimensions.
19

---

From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models
(a) Feature (Discriminative)
(b) Feature (Discriminative)
(c) Feature (Discriminative)
(d) Feature (Discriminative)
(e) Feature (Generative)
(1) Feature (Generative)
(g) Feature (Generative)
(h) Feature 4 (Generative)
Figure 12. T-SNE visualisation of discriminative and generative embeddings of four features, numbered from I to 4. The cardinality of these features 4,051, 820,509, and 2,903,322, respectively. (a-d) illustrate embeddings the four features within discriminative paradigm; (e-h) illustrate embeddings of the four features within the generative paradigm.
20

---

From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models
Candiation Bohwoon
Combeata
Condosice
163
0.5
015
035
033
5.50
933
083
0335
031
035
-100
(a) FM (DIS)
(h) FM (GEN)
(c) FmFM (DIS)
(d) FmFM (GEN)
Fearsor Conndation bathemen Falts
Peurmon Comelaiis between to
Fearson Corriation Detzaenn Eaids
Peursst Carnlatice beboeen Baid
LG
003
-1.00
-059
-330
-336
-1.
1.00
-10
1200
(e) CrossNet V2 (DIS)
(f) CrossNet V2 (GEN)
(g) DeepFM (DIS)
(h) DeepFM (GEN)
Comelaia between
Corriction
Carreaaice
031
031
075
003
130
903
003
-125
-035
-22
-1.0
"5830 0100
"2E
Bide
(i) xDeepFM (DIS)
(i) xDeepFM (GEN)
(k) IPNN (DIS)
(I) IPNN (GEN)
Correiatin beteecn
Poasson Corroictio oetwoon
C75
035
032
025
100
1:0
(m) DCN V2 (DIS)
(n) DCN V2 (GEN)
Figure 13. Pearson correlation matrix between two interacted embeddings. For all discriminative feature interaction models, the correlation matrix becomes nearly zero matrix after reformulating them into a generative paradigm, which perfectly aligns with the redundancy reduction principle.
21


Reviews:
[{"review_id": "mET3KXajBA", "reviewer": "Reviewer_Rab2", "summary": "This paper proposes a Supervised Feature Generation (SFG) framework that reformulates the conventional discriminative CTR prediction paradigm into a generative paradigm. Rather than modeling direct interactions among raw ID embeddings, the proposed method generates each feature embedding based on the concatenation of all other feature embeddings. The goal is to address two issues in CTR modeling: embedding dimensional collapse and information redundancy. The framework is designed to be model-agnostic and can be applied to many standard CTR models such as FM, DeepFM, CrossNet, and DCN V2. Experiments on public benchmarks and an online A/B test on a large-scale advertising platform show small improvements in AUC and Logloss, as well as practical gains in industrial deployment scenarios.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We have addressed the comments in the rebuttal below.\n\n## Clarification on the Motivation. \n\n> Response to Point 1 of \"Claims And Evidence\" & \"Methods And Evaluation Criteria\".\n\nWe'll elaborate more on our claim. Our work is mainly inspired by the Interaction-Collapse Theory, that is, **direct interactions between ID embeddings in existing CTR models with cross networks lead to dimensional collapse**. We resolve this challenge by shifting from the discriminative paradigm that involves direct interactions between raw ID embeddings to a generative paradigm **that constructs new embeddings by a decoder network and interacts the constructed embeddings with the raw ID embeddings**.\n\nWe don't aim to claim  the entire feature interaction paradigm as flawed, and **we apologize of we made such unintentional and misleading claim**. As researchers in this area, we appreciate the promising progresses in the last decade. In particular, we'd like to recognize the **positive impact of cross network and DNN** as follows, and will add them in the revised version.\n\nThe cross network, especially CrossNet in DCN V2, has been proved effective in CTR prediction. A recent work [1] validates that its cross function mitigates dimensional collapse to some extent via field-pair-wise transformation matrix. However, we find that our generative variants can further improve the embeddings' dimensional robustness (Fig 3.c).\n\nRegarding deep networks, **DNNs can also mitigate dimensional collapse compared to direct cross networks**. In fact, **we have a submited paper studying DNNs in feature interaction models from this perspective**, showing that **non-linear activations greatly improve dimensional robustness**. We will add this discussion to the revised version.\n\n[1] Towards Unifying Feature Interaction Models for Click-Through Rate Prediction. 2024.\n\n## Concerns about the Cost-effectiveness.\n\n> Response to Point 2 of \"Claims And Evidence\".\n\nWe thank the reviewer for pointing out the cost-effectiveness trade-off. In industry CTR prediction, even a 0.1% AUC lift is considered significant [1]. Our analysis shows **the ROI (GMV lift/computation cost) for several scenarios far exceeds our release threshold** (typically in the dozens; we omit the exact value since it's a commercial secret). Since February, **three generative models have passed performance/cost reviews and been fully deployed**.\n\n[1] FuxiCTR: An Open Benchmark for Click-Through Rate Prediction. 2020.\n\n## Discussion on Essential References.\n\n> Response to \"Relation To Broader Scientific Literature\" & \"Essential References Not Discussed\".\n\nThanks for your valuable suggestion. We will specifically discuss these methods in the Related Works section of the final paper. We list the main differences as follows due to space limits:\n\nOur paradigm differs from these works in the sense that **we aim to tackle the dimensional collapse issue due to the direct interaction of ID embeddings**. We argue that the above-mentioned related works can't achieve this by refinement, denoising or adopting a GNN architecture. \n\nWe empirically compared our paradigm with several representative feature refinement models, with results as follows. We observed that some models outperform the discriminative DCN V2 models, but still underperform our generative model. \n\n| Model |  | Criteo | Avazu |\n|:---:|:---:|:---:|:---:|\n| FiGNN[3] | - | 0.81352 | 0.79156 |\n| DCNv2 | DIS | 0.81387 | 0.79282 |\n| GFRL[1] | - | 0.81427 | 0.79296 |\n| FRNet[2] | - | 0.81431 | 0.79313 |\n| DCNv2 | GEN | 0.81472 | 0.79342 |\n\nWe also studied the singular spectrum and found that **they can mitigate the dimensional collapse on the tail singular values**compared to the vanilla discriminative DCN V2. However, **our generative model leads to more robust values on all dimensions**. Refer to the spectrum analysis in https://anonymous.4open.science/r/ICML2025-1748/supp/refinement.png.\n\n[1] MCRF: Enhancing CTR Prediction Models via Multi-channel Feature Refinement Framework.\n\n[2] Enhancing CTR prediction with context-aware feature representation learning.\n\n[3] Fi-gnn: Modeling feature interactions via graph neural networks for ctr prediction.\n\n## Clarification of the Discussion Parapragh. \n\n> Response to \"Other Comments Or Suggestions:\"\n\nWe will elaborate on this paragraph and revise the manuscript accordingly with the following:\n\n**Dimensional Collapse of Raw ID Embedding Interaction**. The embeddings of some fields may only span a low-dimensional space due to various reasons, such as the low cardinality of this field. For example, the embeddings of the gender field with values of Male, Female, and Unknown can only span a 3-dimensional space. According to the Interaction-Collapse-Theory, **the interactions with these low-dimensional field embeddings may lead to the dimensional collapse of the embeddings of the other fields**.", "claims_and_evidence": "The paper makes several claims, including:\n\n- That the proposed generative paradigm leads to more semantically meaningful feature embeddings.\n\nWhile the framework is novel in formulation, the motivation behind the paradigm shift is weakly supported. The core claim that “we shift from raw ID embedding interactions to semantically meaningful feature generation” is misleading. In practice, most modern CTR models do not rely solely on raw ID embedding interactions; they use cross networks or deep modules to overcome known limitations. Therefore, the paper constructs a false dichotomy by framing the entire feature interaction paradigm as flawed, when in fact the issue lies with simplistic interaction mechanisms.\n\n- That it provides a general-purpose enhancement applicable to a wide range of CTR models.\n\nFurthermore, although performance improvements are reported (e.g., ~0.272% AUC lift), they are relatively modest considering the additional model complexity. The method increases computation time by 3.14% and GPU memory by 1.45%, raising questions about the cost-effectiveness of the generative design, especially for production systems where latency and efficiency are critical.", "methods_and_evaluation_criteria": "The paper does not make it entirely clear why the embedding reconstruction process should yield better representations in all settings. The notion of “generating semantically meaningful features” remains vague, and the benefit of reconstruction versus learned interaction is not theoretically or empirically justified beyond intuitive arguments.", "theoretical_claims": "There is no formal theoretical contribution.", "experimental_designs_or_analyses": "The experiments are fine.", "relation_to_broader_scientific_literature": "The paper relates to feature embedding learning, representation redundancy, and autoencoding principles. However, it does not clearly position itself against existing embedding refinement, denoising, or feature interaction methods based on field graphs (there are many), which may share similar goals with more principled frameworks.", "essential_references_not_discussed": "Please check recent embedding refinement, denoising, or feature interaction methods based on field graphs (there are many) for recommendation systems.", "other_strengths_and_weaknesses": "Pros:\n\n- Conceptually novel formulation with a modular implementation.\n\n- Broad compatibility with many CTR models.\n\n- Real-world deployment.\n\nCons:\n\n- Why generative paradigm is needed is still unclear. Overstated framing of the paradigm shift—feature interaction is mischaracterized.\n\n- Modest offline improvements raise concerns about cost-effectiveness. The computation cost is less mentioned.\n\n- Vague definition of “semantic generation”; unclear benefit of reconstructing embeddings over learning them directly.", "other_comments_or_suggestions": "\"Inherent drawbacks of raw ID embeddings. Embed-\ndings of low-cardinality feature fields only span a low-\ndimensional embedding space, intrinsically leading to a\nbottleneck for representing abundant information. More-\nover, according to the interaction collapse theory (Guo\net al., 2024), direct interactions between raw ID embed-\ndings can lead to severe dimensional collapse issue (Jing\net al., 2021). Consequently, even embeddings of high-\ncardinality feature fields will be constrained to a low-\ndimensional subspace of the available embedding space,\nthereby limiting their information abundance.\" This paragraph needs to be explained or corrected.", "questions_for_authors": "Please check my comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "0txnCAi7i6", "reviewer": "Reviewer_SGPG", "summary": "This paper introduces a feature generation framework that reformulates conventional CTR models through a generative paradigm, effectively addressing dimensional collapse and information redundancy issues in feature embeddings. The claims are substantiated by rigorous empirical evidence spanning widely adopted benchmark datasets. The proposed methodology is thoroughly evaluated under well-designed experimental settings, demonstrating consistent superiority over baseline approaches. The core implementation is publicly accessible in the supplementary materials to ensure reproducibility. While a related work exploring embedding collapse phenomena is cited, its methodological distinctions from the current approach warrant deeper analysis.\n\nOverall, this paper studies a foundational problem in CTR prediction by innovatively bridging generative paradigms with feature interaction models, with well-designed experiments and insightful conclusions to support the proposed method. I am inclined to recommend acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your valuable comments. We hope the following responses can address your concerns.\n\n## Concerns about Batch-wise Analysis\n\n> Response to Weaknesses of \"Experimental Designs Or Analyses\" & Q1 of \"Questions For Authors\".\n\nThank you for your suggestion. The suggested analysis of the entire validation dataset is time-consuming, so we have adopted this batch-wise setting. To ensure the experiment's robustness, we have **repeated the analysis experiments with 6 different random seeds** when sampling the batches, with results in https://anonymous.4open.science/r/ICML2025-1748/supp/seed.jpg. **The trend of embedding spectra is consistent in all batches**: On both Avazu and Criteo, the spectrum curve of discriminative paradigms exhibit an abrupt singular decay from ~$1\\times 10^{-5}$ to ~$1\\times 10^{-15}$, a reduction of $10^{10}$ times. This indicates a severe dimensional collapse issue. But in our generative paradigm, the abrupt singular value decay has been greatly alleviated. This verifies that the generative paradigm substantially mitigates the embedding dimensional collapse issue, forming a more balanced embedding space.\n\n## Relationship to Multi-Embedding\n\n> Response to \"Essential References Not Discussed\" & Q2 of \"Questions For Authors\".\n\nThanks for your question. **The proposed paradigm is orthogonal to the multi-embedding method, and these two methods can be seamlessly combined.** We have conducted experiments based on one of the most representative models DCNv2 on the Avazu dataset, with results presented as follows:\n\n- Recommendation performance (AUC) comparison.\n\n  | Model\\Embedding size | 16          | 16 $\\times$ 2 | 16 $\\times$ 4 | 16 $\\times$ 8 | 16 $\\times$ 10 |\n  |----------------------|-------------|---------------|---------------|---------------|----------------|\n  | DCNv2 - DIS          | 0.79282     | 0.79402       | 0.79434       | 0.79539       | 0.79577        |\n  | DCNv2 - GEN          | **0.79342** | **0.79469**   | **0.79534**   | **0.79599**   | **0.79617**    |\n  \n  We can observe that, the generative DCNv2 with multi-embedding outperforms discriminative DCNv2 with multi-embedding.\n\n- Embedding spectrum analysis. We have illustrated the spectrum of DCNv2 (DIS), DCNv2 (GEN), and DCNv2(GEN + MultiEmbedding) in https://anonymous.4open.science/r/ICML2025-1748/supp/collapse.png. The results show that:\n    \n  - **Embedding collapse of DCNv2 (DIS)**: The spectrum curve of DCNv2 (DIS) exhibits a dramatic decay after the singular value index 250, which indicates a collapsed embedding space.\n  - **Embedding robustness of DCNv2 (GEN)**: Different from DCNv2 (DIS), the spectrum curve of DCNv2 (GEN) does not exhibit the abrupt decay. Instead, they decline slowly, indicating a more balanced embedding space.\n  - **Multi-embedding alleviates collapse**: DCNv2(GEN + MultiEmbedding) can lead to a slightly slower decline rate than DCNv2 (GEN), which indicates a more robust embedding space.\n\nWe compute the Information Abundance[1] (IA) values of these 3 variants respectively: 9.0082, 13.5031, 13.6999. A higher IA value indicates a less-collapsed embedding space. The IA results are consistent with the embedding spectrum figure. **These spectra and IA results both demonstrate that our method and multi-embedding can be effectively combined to achieve both better performance and less collapse.**\n\n[1] Guo, Xingzhuo, et al. On the Embedding Collapse when Scaling up Recommendation Models. ICML, 2024.\n\n\n## Typos\n\n> Response to \"Other Comments Or Suggestions\"\n\nThank you for your corrections and we will modify the corresponding symbols accordingly. In addition, we will further review the entire manuscript to revise any potentially unclear sections and enhance the overall clarity of our work.", "claims_and_evidence": "This paper claims to reformulate existing feature-interaction models into a novel feature generation paradigm. This claim is substantiated through comprehensive comparative experiments and ablation studies.\n\nIt claims to mitigate the inherent drawbacks of conventional ID embeddings in traditional feature interaction models, i.e., dimensional collapse and information redundancy. This claim is validated via two inspiring and sound experiments.", "methods_and_evaluation_criteria": "The proposed method establishes a feature generation paradigm designed to address feature embedding challenges. The framework's effectiveness is validated through empirical evaluation, employing well-established baseline models and standardized datasets consistent with common protocols.", "theoretical_claims": "This paper does not make theoretical claims.", "experimental_designs_or_analyses": "I have verified the experimental validity, including the main comparison, embedding analysis, and the ablation studies.\n\nStrengths:\n\n- The proposed paradigm delivers substantial performance improvements across diverse existing CTR models, with successful deployment in production-scale advertising systems.\n- The analysis experiments are well-designed, validating that the proposed method can address the claimed dimensional collapse and information redundancy issues. The correlation shift trend from weak to strong models is insightful.\n- The authors conduct a systematic investigation of paradigm components through well-structured ablation studies.\n\nWeaknesses:\n\n- The analysis experiments employ a batch-wise processing. I'm concerned the results may be inconsistent on the full validation dataset.", "relation_to_broader_scientific_literature": "The dimensional collapse issue in feature embeddings has been investigated in recent literature[1], where this challenge was effectively addressed via a multi-embedding method.\n\n[1] On the Embedding Collapse when Scaling up Recommendation Models. International Conference on Machine Learning_. PMLR, 2024.", "essential_references_not_discussed": "Although mentioned, the multi-embedding[1] method is not fully discussed in the paper, which is also specially designed for embedding dimensional collapse mitigation.\n\n[1] On the Embedding Collapse when Scaling up Recommendation Models. International Conference on Machine Learning. PMLR, 2024.", "other_strengths_and_weaknesses": "Overall, this is an interesting paper that studies a fundamental problem in a real-world application problem. PLS see the \"Experimental Designs Or Analyses\" section.", "other_comments_or_suggestions": "Typos\n\t- \"hadamard product\" should be \"Hadamard product\"\n\t- Line 113, \"has focus\" -> \"has focused\"\n\t- An extra annotated \"(1)\" in Line 269", "questions_for_authors": "Will the analysis results be inconsistent when adopted on the entire validation dataset?\nWhat is the relationship between this work and the multi-embedding[1] method in embedding collapse mitigation?\n\n[1] On the Embedding Collapse when Scaling up Recommendation Models. International Conference on Machine Learning_. PMLR, 2024.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UDJ69xwfd1", "reviewer": "Reviewer_z1Cm", "summary": "The paper “From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models” proposes a novel Supervised Feature Generation framework for Click-Through Rate (CTR) prediction models. The main algorithmic idea is to shift from the discriminative “feature interaction” paradigm to a generative “feature generation” paradigm. Instead of relying on raw ID embedding interactions, the framework predicts each feature embedding based on the concatenation of all feature embeddings.\n\nThe main findings indicate that the existing discriminative paradigm has limitations such as embedding dimensional collapse and information redundancy. The proposed generative paradigm mitigates these issues. Experimental results show that the framework can reformulate nearly every existing CTR model and brings significant performance improvements. Across different models, it achieves an average of 0.272% AUC lift and 0.435% Logloss reduction. It also reduces the embedding dimensional collapse and information redundancy, and has been successfully deployed in a large - scale advertising platform, leading to a 2.68% GMV lift in a primary scenario.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly thankful for your review efforts. We apologize for the missing information on datasets and notations, and would clarify them as follows.\n\n## Dataset Details\n\n> Response to \"Methods And Evaluation Criteria\", \"Other Strengths And Weaknesses\", and Q1 of \"Questions For Authors\"\n\nThank you for your question. The numbers in the table are the number of samples, i.e., user-item interactions. \n\nAs for the user or item scales, the user and item features are not annotated in the original dataset. But we can make the following guess:\n- [Criteo](https://github.com/reczoo/Datasets/tree/main/Criteo/Criteo_x1) has 13 numerical feature fields and 26 categorical feature fields. All 26 categorical features have been anonymized, but we can guess based on this assumption: user or item ID features are usually with the highest cardinalities. Feature fields with the top-5 cardinalities are (C3: 413,424), (C12: 409,749), (C21: 397,981) (C16: 365,811), (C4: 248,543), where the first denotes feature name and the second denotes the feature cardinality. These features have a relatively high probability of being user or item ID features. **Therefore, the scale of users and items could be on the order of ~100K**.\n\n- [Avazu](https://github.com/reczoo/Datasets/tree/main/Avazu/Avazu_x4) has 24 categorical features after preprocessing, part of which have been anonymized. Feature fields with the top-5 cardinalities are (device_ip: 2,903,322), (device_id: 820,509), (device_model: 7,259), (app_id: 6,545) (site_domain: 5,461). We infer that 'device' features represent users' devices, which are highly correlated with the number of users. **Therefore, the scale of users could be on the order of ~1M**. The remaining named features are not likely to be item features, so the item feature may be one of the anonymized features: (C14: 2,556), (C17: 434), (C20: 173). **Therefore, the scale of items could be on the order of ~1K**.\n\nBesides, we provide more statistics about our industrial dataset: **Our industrial model is trained on billions of samples daily, with hundreds of millions of unique users and around 1 million items**.\n\n## Notations in Eq. 2\n\n> Response to \"Other Strengths And Weaknesses\", and Q2 of \"Questions For Authors\"\n\nThank you for your question, and we apologize for missing explanations of Eq. 2. We clarify Eq. 2 as follows:\n\n- Eq. 2 is the formulation of DCNv2, which is a high-order feature interaction model. $L$ denotes the number of cross layers, $l$ denotes the layer index, $N$ denotes the total number of features, $i$ and $j$ denote the indices of features, $\\mathbf{v}_i^{(0)} $ denotes the embedding of feature $i$ in the embedding layer, $\\mathbf{v}_j^{(l)}$ denotes the embedding of the $j$-th term in the $l$-th layer.\n- $M_{F(i) \\to F(j)}^{(l)}$ denotes the projection matrix between the $F(i)$ and $F(j)$ field pair in the $l$-th layer.\n- $F(i)$ and $F(j)$ denotes the field of feature $i$ and $j$, respectively.\nIn addition, we will further review the entire manuscript to revise any potentially unclear sections and enhance its overall clarity.", "claims_and_evidence": "The claims are, in general, backed by various experiments, including the claims on embedding dimension collapse and redundancy reduction.", "methods_and_evaluation_criteria": "The evaluation makes sense in general, where the mainstream recommendations algorithms are compared as baselines. Some details of the datasets are missing, and I hope the authors can add them.", "theoretical_claims": "The paper does not contain theoretical claims.", "experimental_designs_or_analyses": "The experiment design is sound, and I think the experiments back the main proposed claims well.", "relation_to_broader_scientific_literature": "The paper is broadly related to the generative recommendation, which is the application of generative models for recommendation systems.", "essential_references_not_discussed": "The paper does not have miseed reference as far as I know.", "other_strengths_and_weaknesses": "Advantage:\n- The paper introduces an important research topic, especially for industry recommendations. The paper clearly states the disadvantage of discriminative models. \n-  Experiments back up the claims that generative recommendations have an advantage over discriminative models. Moreover, the paper gives A/B test results. Though some details are missing (probably because of privacy or the double-blind policy), it makes the method much more convincing.\n\n\nDisadvantage:\n- The paper does not fully give every detail of methods and datasets, like the meaning of some notations in equation 2 and the dataset details in appendix B.1, including the dataset introduction, the number of users, items,etc. (If I missed the notations, please remind me). Particularly, without the explanation of important notations, It may get harder to understand the equation.", "other_comments_or_suggestions": "This paper could be valuable for the industry recommendations. I think that many engineers and researchers in the industry may also be exploring this direction, and the results of thispaper  could be quite inspiring for industry recommendations.", "questions_for_authors": "- In Table 3 in the appendix, what is the meaning of the numbers in the dataset? Does it mean number of user-item interactions? how is the scale of users or items?\n- In equation 2, what is the meaning of $l$ and $L$? What is meaning of $F(i),  F(j)$？", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pA8TqvuQiG", "reviewer": "Reviewer_7DJA", "summary": "This paper highlights issues in discriminative CTR-based recommendation models, such as information redundancy and information collapse. To address these challenges, it proposes a feature generation framework that reformulates CTR prediction as a generative problem using a customized decoder network. The decoder network predicts all feature embeddings based on the input.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your kind remarks. We hope the following responses can address your remaining concerns.\n\n## Concerns about Theoretical Analysis\n\n> Response to \"Theoretical Claims\", and Point 3 of \"Other Strengths And Weaknesses\"\n\nFollowing DirectCLR[1], we have explored theoretical justification for our method's ability to mitigate collapse.\n\nUnder gradient flow analysis (gradient descent with infinitesimal learning rate), the embedding update process follows a differential equation $\\dot{v}_1 = - \\frac{\\nabla L}{\\nabla v_1}$, where $v_1$ is a feature embedding and $L$ the loss. We aim to solve this equation and prove it will be rank deficient during training. **However, even for an FM with two features, the solution involves a complex Lambert W function[2]. We are still working on analyzing its rank properties.**\n\n[1] Understanding Dimensional Collapse in Contrastive Self-supervised Learning. ICLR 2022\n\n[2] On the Lambert W function. Advances in Computational mathematics, 1996.\n\n## Questions about the Hyperparameter Setting\n\n> Response to \"Experimental Designs Or Analyses\"\n\nOur decoder is a one-layer MLP with fixed input and output dimensions. **The only tunable hyper-parameter is the nonlinear activations.** We evaluated different non-linear activations and found they are crucial for collapse mitigation (see Fig. 7 in Appendix D). Based on our experiments, we recommend ReLU or SiLU for the decoder.\n\n## Discussion of Generative Sequential Recommendation with Semantic IDs\n\n> Response to \"Essential References Not Discussed\", and Q3 of \"Questions For Authors\".\n\nThank you for your question. The suggested empirical comparison with them is difficult if not imposible since **Tiger is mainly designed for the sequential recommendaiton scenario while we focus on the feature interaction scenario**. \n\n## Concerns about the Ablation Study\n\n> Response to Point 4 of \"Other Strengths And Weaknesses\", and Q1 of \"Questions For Authors\".\n\nThank you for your insightful question. Regarding the overfitting issue, we acknowledge that our initial observation was speculative. To further investigate, we conducted a detailed spectral analysis comparing 1-layer and 2-layer MLPs, as shown in https://anonymous.4open.science/r/ICML2025-1748/supp/2layer.png.\n\nThe figure shows that both 1- or 2-layer MLPs effectively alleviate the singular value decay (dimensional collapse) observed in discriminative paradigms. **However, singular values of the 2-layer MLP decline at a higher rate than the 1-layer MLP, suggesting that the extra layer leads to a more imbalanced embedding space.** This aligns with the recommendation performance.\n\n## Theoretical or Intuitive Proof for Result 5\n\n> Response to Q2 of \"Questions For Authors\".\n\nThanks for the question. We only have empirical evidence to justify result 5. We'll revise Result 5 as follows: \n\n\"Result5. The field-wise non-linear one-layer MLP is a simple yet effective decoder. **Common** modifications, such as simplifying the model with field-shared MLPs or removing non-linearities, or increasing complexity through stacking MLP layers or self-attention, lead to inferior recommendation performance.\"\n\nWe apologize for our imprecise statement and thank you for pointing this out, which helps us make the statement more rigorous.\n\n## Question on Paradigm Shift for Sequential Recommendation\n\n> Response to Q4 of \"Questions For Authors\". \n\n**Many existing sequential recommendals already employ a generative paradigm under the next-item prediction framework**. Specially, pre-ranking models such as SASRec follows a self-supervised next-item generative paradigm, while ranking models such as DIN, DIEN follow a supervised next-item generative paradigm. The overhead of generative sequential models such as SASRec and DIN is low for short sequence, leading to a widely deployment of them in industrial systems. The computation cost becomes a challenge for long sequences, but there are many works (such as SIM, TWIN) to employ a two-stage Search & Modeling approach to resolve it.", "claims_and_evidence": "In Table 1, the authors demonstrate the consistent improvement of using the generative paradigm over the discriminative one, highlighting an increase in AUC and a reduction in log loss.", "methods_and_evaluation_criteria": "The authors use two common evaluation metrics—Area Under the Curve (AUC) and log loss error reduction—to assess the model's quality. They also discuss the trade-off between quality and performance, noting the increase in computation time and memory usage.", "theoretical_claims": "Given that the primary objective of this work is to reduce information redundancy and dimensional collapse, the theoretical justification is not adequately provided. Although the authors claim that both issues are mitigated empirically in lines 198-201, it remains unclear why passing raw ID embeddings through an MLP would effectively reduce information redundancy from a theoretical standpoint.", "experimental_designs_or_analyses": "In Table 1, can the authors shed some light on the hyperparameter setting for the decoder and MLP?", "relation_to_broader_scientific_literature": "Although the authors focus on a specific problem within the recommendation models domain, the approach proposed in the paper can be applied to other domains to reduce information redundancy and dimensional collapse.", "essential_references_not_discussed": "The key contribution is a paradigm shift from discriminative to generative models aimed at reducing information redundancy and mitigating dimensionality reduction. One important reference could be the inherently generative recommendation models with semantic IDs (Rajput et al. (2023)), comparing the results with the aforementioned references.", "other_strengths_and_weaknesses": "1. The paper is written with a clear flow.\n2. The problem that the paper targets builds a bridge between the well-established discriminative recommendation models and the state-of-the-art generative models.\n3. There is a lack of sufficient theoretical proof.\n4. Some results seem slightly counterintuitive. Please refer to Q1.", "other_comments_or_suggestions": "no comments", "questions_for_authors": "Q1. Can the authors elaborate on this observation? \"On the other hand, increasing the complexity of the decoder with (b.3) significantly degrades recommendation performance, as the AUC decreases from 0.793512 to 0.792931. This may be caused by overfitting\". Intuitively, a more complex decoder could capture more intricate relationships. What do the authors mean by overfitting?  \n\nQ2. Regarding result 5, do the authors have any theoretical or intuitive proof, aside from empirical evidence, to justify it?  \n\nQ3. Compared to inherently generative models in terms of AUC, log loss reduction is not included. If the generative recommendation model is indeed of higher quality, then this should be mentioned.  \n\nQ4. Can this approach lead to a paradigm shift for specific sequential recommendation models like Taobao Alibaba? Does the performance overhead, in this case, justify deploying this approach?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Most of the reviewers have favorable opinions; the first reviewer's negative comment can be fixed before the camera-ready deadline.
Decision: Accept (poster)