Towards Hybrid-grained Feature Interaction Selection for Deep Sparse Network
Authors: Fuyuan Lyu
Xing Tang
Dugang Liu
Chen Ma
Weihong Luo
Liang Chen
xiuqiang He
Xue Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: Feature Interaction Search
Deep Sparse Network
TL;DR: We propose a hybrid-grained feature (both value and field) interaction selection method for deep sparse network.

Abstract: Deep sparse networks are widely investigated as a neural network architecture for prediction tasks with high-dimensional sparse features, with which feature interaction selection is a critical component. While previous methods primarily focus on how to search feature interaction in a coarse-grained space, less attention has been given to a finer granularity. In this work, we introduce a hybrid-grained feature interaction selection approach that targets both feature field and feature value for deep sparse networks. To explore such expansive space, we propose a decomposed space which is calculated on the fly. We then develop a selection algorithm called OptFeature, which efficiently selects the feature interaction from both the feature field and the feature value simultaneously. Results from experiments on three large real-world benchmark datasets demonstrate that OptFeature performs well in terms of accuracy and efficiency. Additional studies support the feasibility of our method. All source code are publicly available\footnote{https://anonymous.4open.science/r/OptFeature-Anonymous}.

Towards Hybrid-grained Feature Interaction Selection for Deep Sparse Network
Fuyuan Lyu', Xing Tang Dugang Liu', Chen Ma²; Weihong Luo?, Liang Chen², Xiuqiang He², Xue Liu' McGill University, 9FT,, Tencent, *City University of Hong Kong, 4 GGanngoog Laboratory of Artificial Intelligence and Digital Economy (SZ.
corresponding author
f fuyuan. 1ywdmail. ca, xing.tang@h tang@hotmail.com com,
dugang wwwggmaill..cccmommmmmmm....meem com. chenma@cityu edu. hk, {lobbyluo, leocchen, x Stenceetttttt..eeee com,
xue www.lluuus.... mue.li@cs.mmggll.ccm mogill..c..
Abstract
Deep sparse networks are widely investigated neural network architecture for prediction tasks with high-dimensional sparse features, with which feature interac- tion selection is critical component. While previous methods primarily focus how to search feature interaction in moorse-graine space, less attention has been given finer granularity. In this work, we introduce hybrid-grainee grained feature interaction selection approach that targets both feature field and feature value for deep sparse networks. To explore such expansive space, propose decomposed space which is calculated on the fly. We then develop a selection algorithm called OptFeature, which efficiently selects the feature interaction from both the feature field and the feature value simultancously. Results from experiments on three large real-world benchmark datasets demonstrate that OptFeature performs well in terms of accuracy and efficiency. Additional studies support the feasibility of our method. All source code are publicly available
Introduction
Deep Sparse Networks (DSNs) are commonly utilized neural network architectures for prediction tasks, designed to handle sparse and high-dimensional categorical features as inputs. These networks find widespread application in real-world scenarios such as advertisement recommendation, fraud detection, and more. For instance, in the context of advertisement recommendation, the input often comprises high-dimensional features like user and City, which significantly contribute to the final prediction.
As depicted Figure [a). the general DSN architecture consists of three components. Firstly, the embedding layer transforms different feature values into dense embeddings. Following this, feature interaction layers create feature interactions 4 based on the embeddings of raw features, as illustrated in Figure b)). Finally, the predictor makes the final prediction based on the features and their interactions. A core challenge in making accurate DSNs predictions is effectively capturing suitable feature interactions among input features 8
16 15].
Various methods have been proposed to address the issue of modelling feature interactions. Wide&Deep first models human-crafted feature interactions using linear regression [22]. To eliminate the need for human expertise, DeepFM [ models all second-order feature interactions by utilizing
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

factorization machine [21] and adopts multi-layer perceptron (MLP) as predictor. Further advancements have replaced the factorization machine with different operations, such as product operations [8], cross network [24 25 or an MI.P component 19] However, these methods have inherent drawbacks as they directly model all possible feature interactions. This inevitably introduces noise into the models, increasing their complexity and potentially degrading performance.
Neural architecture search(NAS) [12] has been introduced as a powerful approach for feature in- teraction selection in DSNs for both efficiency and effectiveness I 7 13]. AutoFIS I| first
propose to select feature interactions by adding an attention gate to each possible one. PROFIT 7 suggests progressively selecting from a distilled search space with fewer parameters. Alongside feature interaction selection, some works like AutoFeature [|] and Optinter [15] aim to search for the interaction operation. However, all these works focus on evaluating the whole feature field as a whole instead of individual feature value. To be more precise, the entire user id or City feature field would be retained or discarded as a unit instead of considering individual feature values like User x479bs or New Orleans. Considering only feature fields inevitably be grase-grained, which may overlook informative values uninformative fields, and vice versa.
In this work, we propose extending the se- lection granularity of feature interactions from the field to the value level. The ex- tension of granularity would significantly increase the complexity of the entire selec- tion space, leading to an increase in explo- ration time and memory usage. We manage this challenge by decomposing the selec- tion space using lensor factorization and calculating the corresponding parameters on the fly. To further improve the selection efficiency, we introduce hybrid-grained feature interaction selection space, which explicitly considers the relation between field-level and velue- To perform feature interaction selection, develop a pparsificationn-asee selection algorithm named OptFeature(short for Optimizing Feature Interaction Selection), which effi- ciently selects the feature interaction con- currently from both feature fields and fea- ture values. We conduct experiments over three large-scale real-world benchmarks and compare aceuracy and efficiency with state-of-the-ar models. Empirical results demonstrate the superiority of our method on both dimen- sions. Additional studies support the feasibility of our method.
(o) DGN framework
[b) Feature Interaction Layer
Prediction
Feature inrracionn
cor
2
Feature Layor
714
1-t Embodddnn Layor
Interaction Selection
DBN Irpul
Mat
thvid
of Field
Flait
Villuee of Fiiit
and relationshpp between feature field feature value
Figure 1: Illustration of deep sparse network and feature interaction layer
2 Related Work
2.1 Deep Sparse Networks (DSNs)
It is commonly believed that feature interaction is the critical challenge towards accurate prediction in deep sparse networks [15]. Various models have been proposed to address the feature interaction layer. DecpFM [8] utilizes factorization machine [21] to model all second-order feature interactions, followed by a multi-layer perceptron as the predictor. More sophisticated feature interaction layers have been proposed to replace the factorization machine, such as the inner product operation in IPNN [8] the cross-network in DCN 24 25 or the MLP component in PIN 19].
With the advancement of neural architecture search B1 14, [2] and continuous sparsification [.] various methods have been proposed to select the informative feature interactions [ 7 and reduce computational costs. AutoFis [1] employs a sparse optimizer to select suitable feature interactions the field level. PROFIT 7 formulates field-level feature interaction selection within a distilled
search space. It employs progressive search for efficient exploration. AutolAS (26 takes one step

---

further to integrate the feature interaction selection as part of the search space and jointly conduct the search with other components like MLP architecture or embedding dimension. GAIN [3, on the other hand, focuses on the DCN 24 like architectures and conducts the feature interaction selection jointly with the model training- However,
previous works conduct feature interaction selection on the field level. Our work builds on the existing approaches for modelling feature interactions in DSNs. We extend the selection granularity to the value level and propose a Hybrid-grained selection approach.
Furthermore, there also exists some works such as Optinter [5] take different angle and search for suitable operations (such as inner product, outer product or element-wii sum) model each feature interaction properly. These works are perpendicular to our study.
AutoFeature 9 and NAS-CTR [29]
2.2 Feature Interaction Selection in DSNs
To conduct feature interaction selection DSNs, previous works M from neural architecture search [1 and continuous sparsification 7 these two techniques in the following. Previous works focusing on feature selection from the discussion.
7 26 13 borrow the ideas
We briefly introduce
are excluded
2.2.1 Neural Architecture Search
Neural architecture search (NAS) automatically identifies optimal architectures for specific tasks and datasets, eliminating the need for expert design [3I
20 This approach has led substantial
advancements in various domains, including vision 31 14, natural language processing [23] and sparse representation learning 7 There are two key aspects to neural architecture search: search space and search algorithm. The search space comprises all potential architectures and is usually desk-dependent. An effective search space should neither too complex, leading to high search costs, too shallow, ensuring the inclusion of suitable architectures. The search algorithm explores the search space to find appropriate architectures. Search algorithms can generally be categorized
into controller-based [31], evaluation-base (20), and -basdd [14 12 classes. Our work distinguishes itself from existing research by addressing the challenge of feature interaction selection for deep sparse networks, where the primary obstacles involve high search costs and expansive search spaces.
2.2.2 Continuous Sparsification
Continuous sparsification focuses on reducing a continuous value to a sparse form and is closely related to method [ It is often used to prune less informative components parameters from neural networks, which is intrinsically equivalent to solving an Lo normalization problem. This technique has been leveraged in various aspects of deep sparse networks (DSNs), notably in the embedding layer [17] and the feature interaction layer [T]. Our work employs continuous sparsification discretize the selection of value-level feature interactions during the search process. By doing so, we can effectively manage the complexity of the search space and maintain high level of efficiency in the search algorithm.
Methods
This section will first formulate the general DSNs in Section 3.1
Then will elaborate the feature
interaction selection problem from field and value perspectives in Section 3.22 We further concrete the proposed OptFeature method, which conducts hybrid-grained feature interaction selection in Section
3.1 DSNs Formulation
Feature Field and Value The input of DSNs comprises multiple feature fields, each containing set of feature values. Hence, two forms of expression exist for each data sample, deriving from feature field and feature value perspectives. More precisely, a data sample x consists of feature fields can be written as:
3

---

field perspective: value perspective:
x 21 Ze,,..., 2nd: [kk 3kg: -.. I.k.]-
The notation z, pz k E [1, m.] K; E N4 represents the set of feature values within the i-th field. Here m, denotes the number of values in each field, and m = m refers to that across all fields. An illustration figure is shown in Figure [[c). We highlight this formulation as it is one of the core differences between our method, which focuses on both field fiel and value perspectives, and previous works 7I which focuses only the field perspective. This allows us to capture more granular and informative interactions, potentially leading to better model performance.
Deep Sparse Networks As mentioned in Section | the DSN commonly consists of three major components: the embedding layer, the feature interaction layer and the predictor. The embedding layer usually employs an embedding table to convert z; from high high-dimensional, sparse vectors into low-dimensional, dense embeddings. This can formulated as:
C; E 2, E Fkg
where E Rmxd is the embedding table, is the pre-defined embedding size. These embeddings are further stacked together as the embedding vector e [c1, C2 Cy,l which also serves as the input for feature interaction layer.
The feature interaction layer further performs interaction operations over its input. The I-th order feature interaction can be generally represented as:
v' V'))
(3)
where V(-), the f-th order interaction operation, < n1, can vary from a multiple layer perceptron [19] to cross layer [24]. In this paper, we select the inner product, which is recognized be the most commonly used operation for feature interaction [8 7 as our interaction operation. For t-th order feature interactions. cardinality |v'| C. Note that the selection of f is usually task-dependent. For instance, in the flick-throuse rate prediction,
2 is considered sufficient [IS]
In the prediction layer, both the embedding vector e and feature interaction vectors vf are aggregated together to make the final prediction éb(e, {v' 2 S n)), where (. represents the prediction function parameterized on f. The mainstream prediction function is MLP [18] 19]. Finally, the loss
function is calculated over each data sample and forms the training object:
L(Dura W)
Σ
(V..0)
Σ
E(y, d0 E (2 (4)
Dual
(x.
(x.
where W {E, 0) denotes the model parameters, D refers to the training set.
3.2 Feature Interaction Selection
Feature interaction selection is erucial in DSNs 7 In general DSNs, all feature interactions, whether informative or not, are included as inputs for the final predictor. This indiscriminate inclusion of all interactions inevitably introduces noise into the model, which can hinder its performance [T Selecting informative feature interactions is called feature interaction selection. Proper feature interaction selection can significantly improve the accuracy and efficiency of DSNs by reducing the amount of noise and focusing learning the most informative interactions [T].
For a 1-th order feature interaction v = (V (....,a.....))) ..), S i S Mi,, we aim to determine whether to keep each element or not. We take = 2 as example to illustrate the problem. The selection can be formulated as learning an gate weight (1),))) E (0, 1} corresponding to each (() All (()).2)) compose the second-order feature interaction selection tensor A?, which is formulated as:
I
(112))
a(13))
(1,n)
T
a
(1.2)
a(233)
a (2,m)
a T
(1.3)
(2,3)
(3.a)
(5)
T (.a)
a T
(2.n)
(33m)
1

---

Here the tensor is symmetric due to the feature interaction being invariant to perturbation and the diagonal is composed of identical tensors. According to different perspectives in Equation 5 there are two forms for Equation 1 (i) when ((i,,a)) is a scalar, A 2 is a 2- matrix which
indicates the field level interaction selection A} E {0,1)'* (ii) when ((i.t))) E {0, ) Xmm1 is tensor, A is a 2-D matrix which indicates the value level interaction selection A E {0, 13m². Therefore, feature interaction selection tensor A' of different order compose the selection tensor set A (A < 1 S n). which is the learning goal of the feuture interuction selection problem.
Field-grained In previous methods I the feature interaction selection is usually conducted at the field level. This can be formulated as learning the field-grained selection tensor set A {A} A E (0, 13m" 2 s / S nj. where A refers to the field-grained feature interaction selection tensor at i-th order. Finding a suitable A is an NP-hard problem [7]. Some works have been devoted to investigating the search strategy. AutoFis [ intuitively assigns a continuous tensor Cj2 E R²' and adopts a sparse optimizer for continuous sparsification so that Cý2 will approximate C eventually, PROFIT 77. on the other hand, uses symmetric CP decomposition [10) to approximate A result.
This can be formulated as A ~ E 3r o ...... 3", where denotes broadcast time operation,
times
(3 E 1 refers to the decomposed vector and R < n is pre-defined positive integer.
Value-grained To extend the feature interaction selection from field-grained to value-grained, focus on the value- salueegraiied selection tensor set A = { A E [0.1] 2 n). Although this formulation is analogous to the field-level one, it is substantially more complex given m > This expansion in complexity poses challenge, referred to the size explosion problem, as the computational cost and memory usage for the value-level selection increases dramatically. So it is hard to utilize previous methods [T] 7 directly into solving the value- value-grained selection problem. We will discuss how we tackle this size explosion problem in Section 3.3.1 in detail.
Hybrid-grained Apart from learning the field and value- galleegrained selection tensor Ay and A.. the hybrid-grainen selection also needs to learn a proper hybrid function, we formulate the hybrid function as a binary selection equation,
A= ( 1 A' A'= o'A; (1-
a')
(6)
where tensor a' E {0, 11=² denoting the hybrid choice. As the quality of the field and value-grained selection tensors will heavily influence the hybrid tensor a', joint optimization of both selection tensors A & A, and hybrid tensor a is highly desired. We will discuss how we tackle this joins training problem in Section 3.3.2 in detail.
Previous field- -rdiine and value- grained selection can be viewed as special case for Equation 6 By setting the hybrid tensor a in Equation ōlas an all-one tensor 1, the original hybrid-graine selection problem is reduced to field-graing feld-grained one. Similarly, the value-grained selection problem can also be obtained from Equation 6 by assigning the hybrid tensor a an all-zero tensor O.
3.3 OptFeature
In this section, we formalize our proposed method OptFeature aiming to tackle the hybrid-grained feature interaction selection problem depicted in Section (.2 Such a problem contains two critical issues:
Size explosion problem: As the number of feature values m is significantly larger than the number of feature fields n, denoted as m >> n, the value- -aaline selection tensor A, increase dramatically and is hard to be stored or explored. To provide concrete example regarding the size explosion, we take the commonly used benchmark Crited as an example. This dataset contains 39 feature fields and 6.8 x 106 feature values. Even if we only consider t = 2 for the second-order feature interactions, the corresponding selection tensor increase from || | = 392 1521 to || =
http:://wwwkaagee..o///ctteedddo challe
5

---

6.8 x 1062
4.6 10Ls, making it impossible to be stored or explored using vanilla NAS 12
approaches
20
continuous sparsification methods [ 6).
Joint training problem: As the quality of the field and value-grained selection tensors A & A will heavily influence the hybrid tensor a, jointly optimization of both selection tensors and hybrid tensor is required. How to efficiently optimize so many large binary tensors remains unsolved.
To address the two critical issues mentioned in Section 3.2 and above, propose method OptFeature with 1) the selection tensor decomposition to address the size explosion issue and 2) sparsification-basee selection algorithm for the joint training problem.
3.3.1 Selection Tensor Decomposition
To efficiently explore a large space Ag. we first decompose the feature interaction selection tensor. Without the loss of generality, only focus the second order, i.e. = value-grained feature interaction selection tensor A More details the general cases 2 2 are listed the Appendix We omit the superscript for simplicity and use A, instead of A in later sections. Given that the selection tensor semi coni-positive and symmetric, we have the Takagi factorization 2 as:
A u En x U²
Here Σ, is a d' x d' diagonal tensor, U Rmxd and d' < m. To further reduce the consumption of memory and reduce factorization error, introduce the deep multi-mode tensor factorization that replaces the U as an output of one neural network, denoted as:
U Sa, (E.);
(8)
where So. : Rmmxd - Rmxxd & d' a neural network with parameter O,, and E,, E Rmxd is additional embedding table for generating feature interaction selection tensor. Figure shows detailed illustration our decomposition.
Notably, during the training and inference stage, do not need to calculate the entire metric A all in once. We only need to calculate a batch of data in each trial. The element of architecture metric A, [ki, ky] can be calculated given the following equation:
A Ak ky] fe. (E, E. x f0, (E,l)y,)))
(9)
The original value- gralied selection tensor A, consists of O(m" elements. By introducing the Takagi factoriza- tion 2 the number of trainable elements is reduced to
mid
Ommd") ). We further reduce that number to O(d(m + d' d')) through the multi-mode tensor factorization [5]. Hence, can train the neural network over the value-level selec- tion on modern hardware.
A
U
uf"
d'm
3.3.2 Sparsification-based Selection Algorithm
Figure 2: Illustration the selection ten- sor decomposition
After decomposing the selection tensor, still need to jointly conduct feature interaction selection and train the model parameters for efficient search. To help convert the continuous feature selection tensor into an accurate binary selection, we adopt the traighhtthrooug estimator(STEE) function [I] for continuous sparsification. The STE can be formulated as customized function S(.), with its forward pass as a unit step function
0,
I
S(x)
=1.
[1.
r x> 0
(10)
and backward pass S(x) = 1, meaning that it will directly pass the gradient backward. Therefore, we can mimic a discrete feature interaction selection while providing valid gradient information for the value-level selection parameters E, & 6., making the whole process trainable. Hence, Equation is re-written as:
(E, A Aklks, ky) =S(fo. (E, [k.,)) E. x fo (E, (.lk))))))
(11)

---

Similar to Equation| 11 we can obtain the field-level selection matrix A, using the following equation:
A = Sig, (i x E x Jē, (i)(;:)))
(12)
where Ey E Raxd the field-level embedding table for feature interaction selection and ga, Raxd + Ruxd is another neural network parameterized by Oy-
After obtaining the value-level selection matrix A, and field-level selection matrix A f. we need to merge them and obtain the hybrid selection result A. Inspired by DARTS [2] and success in previous works [5] [3], we relax the hybrid tensor a into a continuous tensor ag E IR",, which can be trained jointly with other parameters via gradient descent. To ensure convergence, apply the sigmoid function (-) over de- Hence, during training time, Equation 6 can be re-written as:
A A= A + (1 (1-o((a)) A
(13)
Hence, the final search objective can be formulated as:
W = min C(Dua (W", WJ), s.t. W = min L(Daal[W, W*)). W
(14)
W
Here W = (E, Ey. By. x) denoting the parameter required in the interaction selection process, and vat denotes the validation set. We summarize the overall process of our sparsification-busee -bssed selection algorithm in Algorithm
Algorithm Sparsification-basese Selection Algorithm
Require: training and validation set Dira and D.al
Ensure: main model parameters W and feature interaction selection parameters W
1: for epoch = 1, ...,, do
2: while epoch finished do
3:
Sample mini-batch Baa and Bm from validation set Dira and training set Dval
4: Update selection parameter W using gradients V (BBall (W, W))
5: Update model parameter W using gradients C(B. (W, ( W))
6:
end while
7:
end for
3.3.3 Retraining with Optimal Selection Result
During the re-training stage, the optimal hybrid tensor determined as a* Ta following previous works [12
15 13
Hence,
obtain the optimal selection result
A' R a S(g0; (Ej) x
x 9op (E))) (1 S(Jo; (E.) x Σ (E.))
(15)
We freeze the selection parameters W and retrain the model from scratch following the existing works 7). This reason for introducing the retraining stage is remove the influence of sub- optimal selection results during the selection process over the model parameter. Also,
4 Experiment
4.1 Experiment Setup
Datasets To validate the effectiveness of our proposed method OptFeature, we conduct experiments on three benchmark datasets (Criteo, Avazu and KDD12 which are widely used in previous work on deep sparse networks for evaluation purposes 7 [1 The dataset statistics and preprocessing details are described in the Appendix
7

---

Baseline Models We compare our OptFeature with previous works in deep sparse networks. We choose the methods that are widely used and open-sourced:
Shallow network:
Logistic Regression(LR) 22 and Factorization Machine(FM) [21].
DSNs: FNN 27
DeepFM g DCN
IPNN
DSNs with feature interaction (DSSs with FIS): AutoFIs and PROFIT
For the shallow networks and deep sparse networks, we implement them following the commonly used library torchfm These baselines and our OptFeature are available here 8 For AutoFIS and PROFIT, we re-use their original implementations, respectively.
Evaluation Metries Following the previous works 8 30], we adopt the commonly-used perfor- mance evaluation metrics for click-through rate prediction, AUC (Area Under ROC curve) and Log loss (also known as cross-entropy). As for efficiency evaluation, we also use two standard metries, inference time and model size, which are widely adopted in previous works of deep sparse network 7 11.
Parameter Setup To ensure the reproducibility of experimental results, here we further introduce the implementation setting in detail. We implement methods using PyTorch. We adopt the Adam optimizer with mini batch size of 4096. We set the embedding sizes to 16 in all the models. We set the predictor as MLP model with [1024, 512, 256] for all methods, All the tuned on the validation set with learning rate from [lc-3, 3e-4, lc-4, 3c-5, Le-5] and weight decay from [lc-4, 3e-5, le-5, 3c-6, lc-6]. We also tune the learning ratio for the feature interaction selection parameters from [1e-4, 3e-5, le-5, 3e-6, le-6] and while weight decay from [le-4, 3e-5, le-5, 3e-6, 1e-6, 0]. The initialization parameters for the retraining stage are selected from the best-performed model parameters and randomly initialized ones.
are
Model Stability To make the results more reliable, we ran the repetitive experiments with different random seeds five times and reported the average value for each result.
Hardware Platform All experiments are conducted Linux server with one Nvidia-Tess Tesla V100-PCIe-32GB GPU, 128GB main memory and 8 Intel(R) Xeon(R) Gold 6140 CPU cores.
4.2 Benchmark Comparison
Model Performance We present the model performance across three benchmark datasets in Table and make the following observations. Firstly, OptFeature consistently outperforms other models on all three benchmark datasets, validating the effectiveness of our hybrid-grained selection space and selection algorithm. Secondly, DSNs generally yield better performances than shallow models, underscoring the importance of designing powerful sparse networks. Thirdly, DSNs with feature interaction selection tend perform superiorly compared to those incorporating possible interactions. This confirms the significance of conducting feature interaction selection within DSNs. Lastly, methods that decompose the selection space (e.g., OptFcature and PROFIT) consistently outperform AutoFIS, which directly navigates the original selection space.
Model Efficiency We also evaluate model efficiency and present the results in Figure 3 observe that OptFeature outperforms all other methods in terms of inference time, while its model size is comparable to other feature interaction selection methods. Both model accuracy and inference efficiency are critical factors when deploying DSNs. Our OptFeature not only achieves the best performance but also reduces inference time, indicating its practical value.
We can
"nanss//gihabb.com//hhheeee////
DU
7 http://ghhhnb.mmmiiiieeewwynnnnh btugs//ghhhbbboommmyyyyyyuuuuuuu
8

---

Table 1: Overall Performance Comparison
Dataset
Criteo
Avazu
KDD12
Category
Model
AUC
Logloss
AUC
Logloss
AUC
Logloss
LR
0.7882
0,4609
0.7563
0.3928
0.7411
0.1637
Shallow
FM
0.8047
0.4464
0.7839
0.3783
0.7786
0.1566
FNN
0.8101
0.4414
0.7891
0.3762
0.7947
0.1536
DeepFM
0.8097
0,4418
0.7896
0.3757
0.7969
0.1531
DSNs
DCN
0.8096
0,4419
0.7887
0.3767
0.7955
0.
IPNN
0.8103
0.4413
0.7896
0.3741
0.7945
0.1537
AutoFIS 0.8089
0.4428
0.7903
0.3749
0.7959
0.1533
DSNs with FIS
PROFIT 0.8112
0.4406
0.7906
0.3756
0.7964
0.1533
OptFeature 0.8116
0.4402
0.7925
0.3741
0.7982
0.1529
Here denotes statistically significant improvement (measured by a two-sided 1-test with p-value < 0.05)
over the best baseline. The best and second best performed results are marked in bold and underline format
KDD12
3.20
EXH
DoapFM
Doo
3.70
3.79
AutoFiB PROFIT
AutoFBB
Opti eature
warurre
Optif eature
K766
0.20
17
10
570
300
6.3
6900
ARh
cm
100
(MB)
0.70
3000
Die
Deepf'M
DiepTM
DCN
DCN
DCN
IPNN
0.20
Autofis
Autof
AutofIS
PROFFT
PRORT
PROFIT
Optocataro
OytFuuture
OptFostare
3.70
3/9
0780
3770
12
36
10
Time
(me)
(111)
Figure 3: Efficiency comparison between OptFeature and DSN baselines. Model size excludes the embedding layer. Inference time refers to the average inference time per batch across validation set.
4.3 Investigating the Selection Process
In this section, we delve into the selection process of OptFeature. We introduce two variants of OptFeature for this purpose: (i) OptFeature-f, which carries out field-level interaction selection, and (ii) OptFeature-v, which conducts value-level interaction selection. Both methods adopt the same selection algorithm outlined in Section 3.3.2] We will compare OptFeature and its two variants to other DSNs interaction selection baselines regarding effectiveness and efficiency.
Table 2: Performance Comparison over Different Granularity.
Dataset
Criteo
Avazu
KDD12
Model
Granularity
AUC
Logloss
AUC
Logloss
AUC
Logloss
AutoFIS
Field
0.8089
0.4428
0.7903
0.3749
0.7959
0.1533
PROFIT
Field
0.8112
0.4406
0.7906
0.3756
0.7964
0.1533
OptFeature-F
Field
0.8115
0.4404
0.7920
0.3744
0.7978
0.1530
OptFeature-v
Value
0.8116
0.4403
0.7920
0.3742
0.7981
0.
OptFeature
Hybrid
0.8116
0.4402
0.7925
0.3741
0.7982
0.1529
We evaluate the effectiveness of various DSN feature interaction methods in terms of comparable performance. Several observations can be made from the results presented in Table 2 Firstly. OptFeature consistently delivers superior results on all three datasets. This confirms that the grained while the selection algorithm effectively performs based on the decomposition. Secondly, the value-level selection method consistently surpasses field-level selection, suggesting that field- level selection might be too boarse-grained to leverage informative values within uninformative fields. Lastly, OptFeature-f outperforms both AutoFIS and PROFIT in terms of model performance.
Investigating the Selection Effectiveness
selection space allows OptFeature to explore finer-

---

The distinguishing factor among these three methods lies in the selection algorithm. AutoFIS directly optimizes the gating vector for each interaction field, rendering the space too large for efficient exploration. PROFIT, on the other hand, adopts progressive selection algorithm, leading to sub-optimal interaction selection. This finding further emphasizes the superiority of our sparsification- based selection algorithm in selecting informative interactions.
Investigating the Selection Efficiency Next, we per- form a comparison study on the efficiency of the selection methods. In Figure 4 we report the search cost, mea- sured in GPU hours. The search cost reflects the GPU resource consumption when selecting interactions from scratch. It's important to note that the cost of re-training is excluded as our focus is on the selection process. We ob- serve that the search cost of OptFeature is lower than other baselines on the Criteo and KDD12 datasets. OptFeature surpasses the other two variants as it typically converges in fewer epochs. These comparisons underscore the fea- sibility of our proposed method. However, on the Avazu dataset, OptFeature is slightly outperformed by its two variants, i.e. OptFeature-f and OptFeature-v, v, because this dataset usually converges in epoch
Model
Search Cosl (GPU hours)
AutoFII
PROFIT
OptFesture-
OptFeelure-/
OptFeature
Crito
Avazu
KDD12
Figure 4: Comparisons between OptFea- ture and other DSNs interaction selec- tion baselines the search cost.
(28). As result, all three methods converge within the same epoch. Furthermore, compared to AutoFIS and PROFIT, OptFeature-f incurs lower cost with the same complexity of selection space. This observation further highlights the superiority of our selection algorithm in terms of convergence speed.
5 Conclusions and Limitations
In this work, we introduce sebrid-grained selection approach targeting both feature field and feature value level. To explore the vast search space that extends from field value, we propose decomposition method to reduce space complexity and make it trainable. We then developed a selection algorithm named OptFeature that simultaneously selects interactions from field and value perspectives. Experiments conducted on three real-world benchmark datasets validate the effectiveness and efficiency of our method. Further ablation studies regarding search efficiency and selection granularity illustrate the superiority of our proposed OptFeature.
Limitations Despite OptFeature demonstrating superior performance over other baselines on model performance and inference time, requires larger model size than certain DSNs (8] 27]. Additionally, it lacks online experiments to validate its effectiveness in more complex and dynamic scenarios.
References
[1] Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, 3tss/1308.3333 2013.
[2] Alexander M. Chebotarev and Alexander Evgen' evich Teretenkov. Singular value decomposition for the takagi factorization of symmetric matrices. Appl. Math. Comput., 234:380-384, 2014.
[3] Jianbo Chen, Le Song. Martin J. Wainwright, and Michael I. Jordan. Learning to explain: An cooemattion-hhoretti perspective model interpretation. In Proceedings of the 35th International Conference on Machine Learning. ICML 2018, volume 80 of Proceedings of Machine Learning Research, pages 882 891, Stockholmsmässann. Stockholm, Sweden, 2018. PMLR.
[4] James Enouen and Yan Liu. Sparse interaction additive networks via feature interaction detection and sparse selection. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 13908-13920. Curran Associates, Inc., 2022.
10

---

[5] Jicong Fan. Multi-mode deep matrix and tensor factorization. In The Tenth International Conference on Learning Representations, ICLR 2022. Virtual Event, April 25-29, 2022. Open- Review.net, 2022.
[6] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, 2019. OpenReview.net.
[7] Chen Gao, Yinfeng Li, Quanming Yao, Depeng Jin, and Yong Li. Progressive feature interac- tion search for deep sparse network. In Marc Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2,21, December 6- 14, 2021, virtual, pages 392-403, 2021.
[8] Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: A cectorizationnmaallee machine based neural network for CTR prediction. In Proceedings of the Twenty- Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, pages 1725-1731, Melbourne, Australia, 2017. ijcai.org.
[9] Farhan Khawar, Xu Hang, Ruiming Tang, Bin Liu, Zhenguo Li, and Xiuqiang He. Autofeature: Searching for feature interactions and their architectures for click-through rate prediction. In CIKM, 2020.
[10] Tamara G. Kolda and Brett W. Bader. Tensor decompositions and applications. SIAM Rev., 5 51(3):455-500 2009.
[11] Bin Liu, Chenxu Zhu, Guilin Li, Weinan Zhang. Jincai Lai, Ruiming Tang, Xiuqiang He, Zhenguo Li, and Yong Yu. Autofis: Automatic feature interaction selection factorization models for click-through rate prediction. In KDD, 2020.
[12] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LAA, USA, May 6-9, 2019. OpenReview.net, 2019.
[13] Yaoxun Liu, Liangli Ma, and Muyuan Wang. GAIN: A gated adaptive feature interaction network for flick-througe rate prediction. Sensors, 22(19):7280, 2022.
[14] Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Yan Liu. Neural architecture opti- mization. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Sys- tems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 7827-7838, 2018.
[15] Fuyuan Lyu, Xing Tang, Huifeng Guo. Ruiming Tang. Xiuqiang He, Rui Zhang, and Xue Liu. Memorize, factorize, or be naive: Learning optimal feature interaction methods for CTR prediction. In 38th IEEE International Conference on Data Engineering, ICDE 2022, Kuala Lumpur, Malaysia, May 9-1 2022, pages 1450-1462, IEFE, 2022.
[16] Fuyuan Lyu, Xing Tang, Dugang Liu, Liang Chen, Xiuqiang He, and Xue Liu. Optimizing feature set for click-through rate prediction. In Ying Ding. Jie Tang, Juan F. Sequeda, Lora Aroyo, Carlos Castillo, and Geert-Jan Houben, editors, Proceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 4 May 2023, pages 3386-3395 ACM, 2023.
[17] Fuyuan Lyu, Xing Tang, Hong Zhu, Huifeng Guo, Yingxue Zhang, Ruiming Tang, and Xue Liu. Optembed: Learning optimal embedding table for thrkuhhhoough rate prediction. In Mohammad Al Hasan and Li Xiong, editors, Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA, October 17-21, 2022, pages 1399-1409, ACM, 2022.
[18] Yanru Qu Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, and Jun Wang. Product- based neural networks for user response prediction. IEEE 16th International Conference on Data Mining, ICDM 2016, pages 1149-1154, Barcelona, Spain, 2016. IEEE Computer Society.
11

---

[19] Yanru Qu, Bohui Fang, Weinan Zhang, Ruiming Tang. Minzhe Niu. Huifeng Guo, Yong Yu, and Xiuqiang He. Product-based neural networks for user response prediction over multi-field categorical data. ACM Trans. Inf. Syst., 37(1))::11553333 2019.
[20] Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka I. Leon-Suematsu, Jie Tan, Quoc V. Le. and Alexey Kurakin. Large-scale evolution of image classifiers. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 2902-291 1. PMLR, 2017.
[21] Steffen Rendle. Factorization machines. In ICDM 2010, The 10th IEEE International Confer- ence on Data Mining, pages 995-1000, Sydney, Australia, 2010. IEEE Computer Society.
[22] Matthew Richardson, Ewa Dominowska, and Robert Ragno. Predicting clicks: estimating the click-through rate for new ads. In Proceedings of the 16th International Conference on World Wide Web, WWW 2007. Banff, Alberta, Canada, May 8-12, 2007, pages 521-530, New York, NY, United State, 2007. ACM.
[23] David So, Quoe Le, and Chen Liang. The evolved transformer. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pages 5877-5886. PMLR, 09-15 Jun 2019.
[24] Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. Deep & cross network for ad click predictions. In Proceedings of the ADKDD 17, Halifax, NS, Canada, August 13 - 17, 2017, pages 12:1-12:7. ACM, 2017.
[25] Ruoxi Wang, Rakesh Shivanna, Derek Zhiyuan Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. DCN V2: improved deep & cross network and practical lessons for web-scale learning to rank systems. In WWW, 2021.
[26] Zhikun Wei, Xin Wang, and Wenwu Zhu. Autoias: Automatic integrated architecture searcher for click-trough rate prediction. In CIKM '21: The 30th ACM International Conference on Information and Knowledge Management, pages 2101-2110, Virtual Event, Queensland, Australia, 2021. ACM.
[27] Weinan Zhang, Tianming Du, and Jun Wang. Deep learning multi-field categorical data case study on user response prediction. In Advances in Information Retrieval 38th European Conference IR Research, ECIR 2016, volume 9626, pages 45-57. Springer, 2016
[28] Zhao- Yu Zhang, Stang-Rong Sheng, Yujing Zhang, Biye Jiang, Shuguang Han, Hongbo Deng. and Bo Zheng. Towards understanding the overfitting phenomenon of deep click-through rate models. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM '22, page 2671-2680, New York, NY, USA, 2022. Association for Computing Machinery.
[29] Guanghui Zhu, Feng Cheng, Defu Lian, Chunfeng Yuan, and Yihua Huang. NAS-CTR: efficient neural architecture search for click-through rate prediction. In SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 332-342, Madrid, Spain, 2022. ACM.
[30] Jieming Zhu, Jinyang Liu, Shuai Yang, Qi Zhang, and Xiuqiang He. Open benchmarking for tlick-through rate prediction. In Gianluca Demartini, Guido Zuccon, Shane Culpepper, Zi Huang, and Hanghang Tong, editors, CIKM '1: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 5, 2021, pages 2759-2769. ACM, 2021.
[31] Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017.
12

---

A General Form of Selection Tensor Decomposition
In this section, we further extend the selection tensor decomposition in Section 3.3.1 from a special
where t 2, to general case, where < t Σ n. The interaction selection tensor A for
t-th order features is also semi-positive and symmetric. By extending the Takagi factorization [2). we have:
A = E X1 u 222 X,M,
(16)
where En is d x.. x x diagonal tensor, X denotes the (-mode matrix multiplication S
ttimes
UE E Ruadd and d < m. Similar to the special case where / = 2, we adopt multi-mode tensor factorization [ replace U as output of neural network, denoted as:
u fo E..)
(17)
where fg. Rnndd - mm d < is neural network with parameter è and E m is an additional embedding table for generating feature interaction selection tensor. The element of architecture metric A A ] can be calculated given the following equation:
1 k., E. x1 fe. (Ey [K,, :I)
fe, (Eolka,
(18)
The original value-grained selection tensor A consists of O(m') elements. The trainable elements is reduced to ((md)) ) after the Takagi factorization [2] and to Odd(m d')) after the multi-mode
tensor factorization [5]
B Dataset and Preprocessing
We conduct our experiments on two public real-world benchmark datasets. The statistics of all datasets are given in Table We describe all these datasets and the pre-processing steps below.
Table 3: Dataset Statistics
Dataset
#samples
#field
#value
pos ratio
Criteo
4.6 x 10"
39
6.8
10
0.2562
Avazu
4.0 x 107
24
4.4
106
0.1698
KDD12
1.5 x 108
11
6.0 x 106
0.0445
Industrial
3.0 x 108
134
2498
0.1220
Note: #samples refers to the total samples in the dataset, #field refers to the number of feature fields for original features, #value refers the number of feature values for original features, pos ratio refers to the positive ratio.
Criteo dataset consists of ad click data over a week. It consists of 26 categorical feature fields and 13 numerical feature fields. Following the best practice [30), we discretize each numeric value æ to [log [z) if > 2; otherwise. We replace infrequent categorical features with a default "OOV" (i.e. out-of-vocabularg token, with min_coun/=2)
Avazu dataset contains 10 days of click logs. has 24 fields with categorical features. Following the best practice (30] we remove the instance field and transform the timestamp field into three new fields: hour, weekday and is weekend. We replace infrequent categorical features with the "OOV" token, with 2.in_count22.
KDD12 dataset contains training instances derived from search session logs. It has 11 categorical fields, and the click field is the number of times the user clicks the ad. We replace infrequent features with "OOV" token, with ciin_coun==10.
Industrial dataset is private large-scale commercial dataset. This dataset contains nearly million samples with 134 feature fields and 2498 feature values. Please notice that this dataset has more feature fields and fewer feature values, which differs from the previous benchmarks with fewer feature fields and larger feature values. The results of this dataset are shown in Appendix C.5
13

---

C Ablation Study
C.1 Feature Interaction Operation
In this section, we conduct ablation study the feature interaction operation, comparing the performance of the default setting, which uses the inner product, with the outer product operation. We evaluate these operations on OptFeature and its two variants: OptFeature-f and OptFeature-v. The results are summarized in Table
Table 4: Performance Comparison over Different Feature Interaction Operation.
Dataset
Criteo
Avazu
KDD12
Category
Model
AUC
Logloss
AUC
Logloss
AUC
Logloss
OptFeature-f
0.8115
0,4404
0.7920
0.3744
0.7978
0.1530
inner product
OptFeature-v
0.8116
0,4403
0.7920
0.3742
0.7981
0.1529
OptFeature
0,8116
0,4402
0.7925
0.3741
0.7982
0.1529
OptFeature-F
0.8114
0,4404
0.7896
0.3760
0.7957
0.1335
outer product
OptFeature-v
0.8113
0.4405
0.7902
0.3752
0.7961
0.1533
OptFeature 0.8115
0,4403
0.7899
0.3753
0.7961
0.1533
From the table, we observe that the inner product operation outperforms the outer product operation. This performance gap is particularly significant on the Avazu and KDD12 datasets, while it is relatively insignificant on the Criteo dataset. The drop in performance with the outer product operation is likely due to the introduction of significantly larger number of inputs into the final predictor. This makes it more challenging for the predictor to effectively balance the information from raw inputs and feature interactions.
C.2 Dimension Selection
In this section, we perform an ablation study on the feature interac- tion selection dimension d. We compare the AUC performance with the corresponding dimension d and present the results in Figure From the figure, we can observe that the dimension d increases, AUC performance remains relatively consistent over the Criteo dataset. This suggests that it is relatively easy to distinguish value- level selection on the Critco datasct. However, on the Avazu and KDD12 datasets, the AUC performance improves as the selection dimension d increases. This indicates that distinguishing informative values is comparatively more challenging on these two datasets.
Oriteo
Figure 5: Ablation over fea- ture interaction selection di- mension OptFeature.
C.3 Higher-Order Feature Interactions
In this section, we investigate the influence of higher-order feature interactions over the final results on the KDD12 dataset. We compare the default setting which only considers second-orde interactions with two other settings: (i) only third-order interactions and (ii) both second and third-order interactions. We visualize the result in F
From the figure, we can draw the following observations. First, only considering third-order interac- tions leads to the worst performance. This aligns with the common understanding that second-order interactions are typically considered the most informative in deep sparse prediction [15). Second, for field-level selection, the performance improves when both second and third-order interactions are incorporated into the model. This finding is consistent with previous studies [ as the inclusion of additional interactions introduces extra information that enhances the performance. In contrast, for value-level selection, the performance tends to decrease when both second and third-order interactions are included. This could be attributed to the fact that value-level selection operates at finer-grained level and might be more challenging to optimize directly. Finally, OptFeature constantly outperforms its two variants over settings. This indicates the feasibility and effectiveness of hybrid-graines selection, which combines both field-level and value-level interactions.
14

---

C.4 Selection Visualization Crtoo-OpiFeature
Orteo-OpfFaalurev
Criee-OOpFeelture
(a) Criteo Dataset
Avezl-OpfFoature-
Avszu--piFeesturee
Avazu-OpiFeeluue
(b) Avazu Dataset
KDD12 OptFecture.
KDD FF01
KDD12-OptFoatur
(c) KDD12 Dataset
Figure 7: Visualization of the Feature Interaction Selection Results.
In this section, we present the visualization of the interaction selection results for OptFeature and its two variants in Figure 7 OptFeature-f performs a binary selection for each interaction field, allowing for easy visualization through a heatmap representation where one indicates keep and zero indicates drop. On the other hand, OptFeature-v and OptFeature involve value- level interaction selection. Hence, we visualize them by setting each element as the percentage of being selected over the training set. The de- tailed equation for calculating the value for the interaction field (i, j) is shown in Equation 19
2nd
3rd
5/M
2nd
3nd
2hald3rd
Hoord
ViLue
AUC
(b) Logloss
Figure 6: Performance Comparison over Different Feature Interaction Orders.
# #Samples keeping interaction field (i. j)
P ((..)
(19)
#Training Samples
From the visualization, we can observe that OptFeature acts as a hybrid approach, exhibiting combination of both field-level and value-level interactions. Interestingly, we note significant differences between certain interaction fields in the KDD12 and Avazu datasets. OptFeature-f retains all of its interactions, while OptFeature-v only keeps a proportion of the value-level interactions. This observation further emphasizes the importance of exploring interactions at finer-grained level.
15

---

C.5 Experiment on Industrial Dataset
We explicitly conduct experiments on an industrial large- scale dataset, which contains more feature fields and fewer feature values compared with previous benchmarks. The following results in Table S can further prove the effectiveness of OptFeature. We can make the following observations. Firstly, OptFeature outperforms other models on the industrial dataset, validating the effectiveness of our -briidegainedd selection space and selection algorithm. This observation consistent with those on the other three public benchmarks, shown in Table Secondly, AutoFIS performs superiorly compared to other baselines. This confirms the significance of conducting feature interaction selection within DSNs. Lastly, PROFIT is surpassed by AutoFIS, unlike on previous public datasets. We believe this is likely due to the increasing number of feature fields. AutoFIS, which treats each field independently, can extend easily to datasets with more feature fields. AutoFIS, which decomposes the feature selection result on the field level, does not generalize well. With the increase of the feature fields, the complexity of such decomposition increases exponentially. OptFeature, on the contrary, can potentially extend to datasets containing more fields better due to its hybrid selection granularity. Table 5: Performance Comparison Industrial Dataset
Dataset
Industrial
Category
Model
AUC
Logloss
LR
0.7745
0.2189
Shallow
FM
0.7780
0.2181
FNN
0.7838
0.2168
DeepFM
0.7824
0.2179
DSNs
DCN
0.7844
0.2167
IPNN
0.7883
0.2147
AutoFIS
0.7889
0.2146
DSNs with FIS
PROFIT
0.7849
0.2161
OptFeature 0.7893
0.2142
The best and second best performed results marked in bold and underline format respectively.
D Broader Impact
Successfully identifying informative feature interactions could be a double-edged sword. On the one hand, by proving that introducing noisy features into the model could harm the performance, feature interaction selection could be used as supporting evidence preventing certain businesses, such as advertisement recommendations, from over-collecting users' information, thereby protecting user privacy. On the other hand, these tools, if acquired by malicious people, can be used for filtering out potential victims, such as individuals susceptible email fraud. As researchers, crucial for to remain vigilant and ensure that our work is directed towards noble causes and societal benefits.
16


Reviews:
[{"review_id": "Ru8X35F8Y7", "reviewer": "Reviewer_21ZP", "summary": "This paper introduces a hybrid-grained feature interaction selection approach that targets both feature field and feature value for deep sparse networks and decomposes the selection space using tensor factorization and calculating the corresponding parameters on the fly. \n", "questions": "See weakness", "limitations": "Yes, the authors discussed some limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Extending the selection granularity of feature interactions from the field to the value level.\n\nIntroduce a hybrid-grained feature interaction selection space, which explicitly considers the relation between field-level and value-level. \n\nThe tensor decomposition and the sparsification are combined to perform selection on the shrinking space.\n", "weaknesses": "1.The evaluation datasets are pretty small (the feature number is around 11-26). For recommendation systems optimization work, it is usually better to show the results in large-scale datasets like industrial datasets to demonstrate the scalability and performance.\n\n\n2.Missing several references:\n\nAutoFAS，\n\nNAS-CTR，\n\nAutoIAS,\n\nGAIN: A Gated Adaptive Feature Interaction Network for Click-Through Rate Prediction\n\nMaybe adding some discussions or comparisons to them is better. \n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hi Reviewer 21ZP:\n\nThanks for your effort in reviewing our paper and offering constructive suggestions. \n\n### **W1: small datasets**\nWe thank the reviewer's helpful suggestion in extending datasets with relatively different statistics. Our response to your concern is split into the following two parts.\n\nFirst, to ensure we are on the same page, we want to highlight the concepts of *feature field* and *feature value*. We include the following table regarding the statistics of our datasets (also detailedly described in Appendix B1). As we can observe, the number of feature values, which influence the maximum selection space, is around $\\sim 10^6$ level. \n\n| Dataset | \\#Samples | \\#field | \\#values | pos ratio |\n| --- | --- | --- | --- | --- |\n| Criteo | $4.6 \\times 10^7$ | 39 | $6.8 \\times 10^6$ | 0.2562 \n| Avazu  | $4.0 \\times 10^7$ | 24 | $4.4 \\times 10^6$ | 0.1698 \n| KDD12  | $1.5 \\times 10^8$ | 11 | $6.0 \\times 10^6$ | 0.0445 \n\nSecond, we add one additional experiment on a private large-scale industrial dataset. This dataset contains nearly 3.5 million samples with 134 feature fields and 2498 feature values. Please notice that this dataset has more feature fields and fewer feature values, which differs from the previous benchmarks with fewer feature fields and larger feature values. The following results can further prove the effectiveness of OptFeature. The observations are also consistent with those on the other three public benchmarks. We will include these results in the appendix later.\n\n| |  AUC | LogLoss |\n| --- | --- | --- |\n| LR         | 0.7745 | 0.2189 |\n| FM         | 0.7780 | 0.2181 |\n| FNN        | 0.7838 | 0.2168 |\n| DeepFM     | 0.7824 | 0.2179 |\n| DCN        | 0.7844 | 0.2167 |\n| IPNN       | 0.7883 | 0.2147 |\n| AutoFIS    | 0.7889 | 0.2146 |\n| PROFIT     | 0.7849 | 0.2161 |\n| OptFeature | 0.7893 | 0.2142 |\n\n\n### **W2: missing reference**\nThanks for referring to these relevant papers, as they are relevant and related. Please refer to the common response(CW2) for our response and corresponding changes.\n\n"}, {"review_id": "oqTIeRuWXo", "reviewer": "Reviewer_57Bi", "summary": "This paper tackles the problem of modeling fine-grained feature interactions in high-dimensional sparse features.\nA hybrid-grained feature interaction selection method is proposed, which operates on both field and value for deep sparse networks.\nTo handle the increase in computation, a decomposed form of the selection space is done, which greatly reduces the computational requirements of modeling.\nResults on deep sparse networks benchmarks show that the proposed method achieves SOTA results while being more computationally efficient.", "questions": "See weaknesses.", "limitations": "The limitations are discussed at the end of the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Strong results in terms of performance on established benchmarks and computational efficiency, demonstrating the effectiveness of the proposed method.\n- The proposed method seems generalizable and can be applied to other methods.\n- All experimental parameters are provided, making reproduction straightforward.\n- The writing is fairly clear and easy to understand.", "weaknesses": "- Experimental results:\n  - The proposed method is a simple tensor decomposition for improved efficiency and the additional consideration of more features. Such choices are generalizable to other architectures (as mentioned in lines 109-112) but this is not demonstrated in the paper. I would like to see the application of the proposed components to other existing approaches.\n- Significance of results:\n  - The AUC and Logloss scores differ by less than 0.001 between the proposed method and the previous SOTA. Is this significant? I suggest the authors add confidence intervals to Table 1 and 2 for easier comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hi Reviewer 57Bi:\n\nThanks for your effort in reviewing our paper and appreciating our effort. Below are responses to your questions.\n\n### **W1: generalization**\nWe deeply agree with the reviewer about the importance of generalization.\n\nTo investigate this aspect, we include ablation studies regarding various interaction operations, embedding dimensions and feature interaction orders in Appendix C1, C2 and C3, respectively. However, the reviewer's comment reminds us to be self-contained in the main text. We will append additional references to the appendix sections in the main text regarding the generalization issue. Hopefully, this can address the reviewer's concern.\n\nMore broadly speaking, we agree with the reviewer that it would be interesting to see how our method can be combined with more complex methods, such as embedding dimension search or feature interaction operation selection. However, the major difficulty lies in disentangling the influence between different methods, which may lead to sub-optimal results.  This, frankly speaking, is one of our future research projects. Kindly pay attention to our future papers regarding this aspect. \n\n### **W2: significance**\nThanks for pointing this out! Adding the statistical analysis can better validate our method. Please check the common weaknesses(Cw1) for our response.\n"}, {"review_id": "HM75AEDayZ", "reviewer": "Reviewer_Bzgv", "summary": "This work proposes a hybrid-grained feature interaction selection approach for deep sparse networks, which targets both feature field and feature value. The proposed approach uses a decomposed space that is calculated on the fly to explore the expansive space of feature interactions. The work also introduces a selection algorithm called OptFeature, which efficiently selects the feature interaction from both the feature field and the feature value simultaneously. The proposed approach is evaluated on three large real-world benchmark datasets, and the results demonstrate that the proposed approach performs well in terms of accuracy and efficiency. The work concludes that the proposed approach can effectively select feature interactions in deep sparse networks, and it has the potential to improve the performance of prediction tasks with high-dimensional sparse features.", "questions": "1. Experiments: What GPU was used in this work? How many were used? Were all the experiments conducted on the same GPU? Why formulate the hybrid-grained feature interaction selection as a binary selection according to Equation 6? Taking either 0 or 1 doesn’t seem to reflect ‘hybrid’. \n\n2. Writing: the introduction does not summarize the main contributions of this work, so readers cannot intuitively get the advantages of this work. In addition, the content of Section 2.1 introducing Neural Architecture Search seems to be not very relevant to this paper. Furthermore, Section 3.3.2 does not explain how to determine the parameter α in Equation 6, which makes one wonder how to choose between value-grained and field-grained.", "limitations": "See the weaknesses.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The hybrid-grained feature interaction selection approach goes beyond traditional field-level selection, and the decomposed space and sparsification-based selection algorithm make the work appear to be a cutting-edge method to some extent.\n2. This work ran the repetitive experiments with different random seeds five times and reported the average value for each result, and provides information about the parameter setup, metrics, datasets, baseline and parameter setup, so the experimental results appear to be reliable.", "weaknesses": "1. Novelty: The proposed approach that targets both feature field and feature value levels and introduced a decomposed space and a sparsification-based selection algorithm to explore the selection space, which appears to be a novel contribution to the field. But it does not provide a comprehensive review of related Feature Interaction Selection work in Section 2, and the novelty is not so obvious.\n\n2. Experiments: What GPU was used in this work? How many were used? Were all the experiments conducted on the same GPU? Why formulate the hybrid-grained feature interaction selection as a binary selection according to Equation 6? Taking either 0 or 1 doesn’t seem to reflect ‘hybrid’. \n\n3. Writing: the introduction does not summarize the main contributions of this work, so readers cannot intuitively get the advantages of this work. In addition, the content of Section 2.1 introducing Neural Architecture Search seems to be not very relevant to this paper. Furthermore, Section 3.3.2 does not explain how to determine the parameter α in Equation 6, which makes one wonder how to choose between value-grained and field-grained.\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hi Reviewer Bzgv:\n\nThanks for reviewing our paper and offering detailed comments. Below are responses to your questions.\n\n### **W1: Unclear expression of our novelty**\nThanks for recognizing the novelty of our method. This comment does remind us of the importance of constantly highlighting our contribution and clearly differing our method from related works. Please check the common review regarding our revision.\n\n### **W2 Part 1: Hardware platform**\nWe thank the reviewer for highlighting these details, as they can be important for reproducibility and can benefit the whole community. We listed these details below (also detailedly described in Appendix B3). Kindly check if they can address your question. However, the author's comment reminds us to be self-contained in the main paper. We will add corresponding references in the main paper later.\n\n| Hardware | Config |\n| --- | --- |\n| CPU | 8-core Intel(R) Xeon(R) Gold 6140 CPU |\n| GPU | Nvidia-Tesla V100-PCIe-32GB |\n| Memory | 128GB |\n| System | Ubuntu 18.04 LTS |\n\n### **W2 Part 2: Design of hybrid-grained feature interaction selection**\nBased on our humble opinion, the reviewer's confusion is about what part of the model is \"hybrid\". In our work, we define \"hybrid\" as one feature interaction conducts selection on the value level while another conducts selection on the field level. For instance, a 2nd-order feature interaction <*user id*, *City*> may conduct selection on the value level, as the user's preference may differ a lot. Meanwhile, another 2-nd order feature interaction <*date*, *city*> may conduct selection on the field level considering the selection efficiency. Hence, the \"hybrid\" refers to the different selection granularity between feature interactions.\n\n### **W3: Writing**\nWe appreciate the reviewer pointing out the confusing and unclear points in our writing. Accordingly, we address each point the reviewer mentioned as follows.\n\n#### *Introduction*\nPlease kindly allow us to highlight our papers' contributions here. We will revise the last paragraph of the introduction section accordingly.\nFirst and foremost, we extend the granularity of feature interaction selections from field level to value level. To the best of our knowledge, we are the first to propose such an extension.\nSecond, to tackle the size explosion problem brought by such an extension, we decompose the selection space via tensor factorization.\nThird, for the sake of selection efficiency, we propose a hybrid-grained selection approach (hybrid between value level and field level) named OptFeature.\nFinally, we conduct extensive experiments to validate the efficiency and effectiveness of OptFeature on three large-scale benchmark datasets. Multiple ablation studies are also conducted to investigate different aspects of our approach, which can help readers better understand and utilize it in their works.\n\n#### *Related Work (Section 2.1)* \nWe thank the reviewer for pointing this out. Generally speaking, the neural architecture search techniques are part of the techniques that inspired several previous works on feature interaction selection[1,2]. This is the same as continuous sparsification[3] in Section 2.2. To clarify the reviewer's concern, we will merge these two sections and highlight their relationship with our work at the beginning.\n\n#### *Section 3.3.2* \nWe explicitly want to thank the reviewer for pointing this out, which helps to make our paper more self-contained. We forget to mention the determination of $\\alpha$ in our paper. The reason is that determining $\\alpha$ makes no difference from DARTS[4] and previous work[5,6]. It is relaxed as a continuous vector (each element between 0 and 1) during searching and discretized as a deterministic vector (each element is either 0 or 1). Please check the following for determining $\\alpha$ in Section 3.3.2.\n- Inspired by DARTS[4] and its success in previous works[1,5,6], we relax the hybrid tensor $\\alpha$ into a continuous tensor $\\alpha_c \\in \\mathcal{R}^{n^t}$, which can be trained jointly with other parameters via gradient descent. To ensure convergence, we apply the sigmoid function over $\\alpha_c$. Hence, during training time, Eq 6 (original Eq 6 $\\mathbf{A} = \\alpha \\mathbf{A}_f + (1 - \\alpha) \\mathbf{A}_v$) can be rewritten as:\n$$\\mathbf{A} = \\text{sigmoid}(\\alpha_c) \\mathbf{A}_f + (1 - \\text{sigmoid}(\\alpha_c)) \\mathbf{A}_v$$\n\nWe will also add the following sentence to Section 3.3.3, which describes how we conduct retraining.\n- During the re-training stage, the optimal hybrid tensor is determined as $\\alpha^* = \\mathbb{1}_{\\alpha_c > 0}$ following previous works[4-6].\n\n\n\n### **Reference**\n1. Autofis: Automatic feature interaction selection in factorization models for click-through rate prediction (KDD2020) \n2. Autoias: Automatic integrated architecture searcher for click-through rate prediction (CIKM 2021)\n3. Progressive feature interaction search for deep sparse network (NeurIPS 2021)\n4. DARTS: Differentiable Architecture Search (ICLR 2019)\n5. Memorize, factorize, or be naive: Learning optimal feature interaction methods for CTR prediction (ICDE 2022)\n6. GAIN: A Gated Adaptive Feature Interaction Network for Click-Through Rate Prediction (Sensors 2022)\n\n*Please notice that we use the format paper \"title(venue)\" here due to the page limit. We fully appreciate all author's contributions to the community.*"}, {"review_id": "8PvvV673Ve", "reviewer": "Reviewer_eecX", "summary": "The authors propose a novel feature selection algorithm, aiming to detect interactions between features at instance-level, contrary to the usual feature selection algorithms that selects the same features for every sample. Initially, the authors propose a highly memory-demanding approach, requiring an $m \\times m$ matrix, being m the number of different values the features can take. Later, a less memory-demanding approach is presented, using matrix decomposition. The experimental results are slightly better to the state-of-the-art.", "questions": "See the weaknesses sec", "limitations": "Not applied.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **Quality:** Several DSN methods were included in the state-of-the-art section. The proposed algorithm is able to obtain very similar results.\n- **Clarity:** The paper is easy to follow and to understand. The decisions made are clearly motivated.\n- **Significance:** The idea of selecting, per each sample, the most important interactions between features is very interesting and it can provide a good explanation about the decision making.", "weaknesses": "- **Originality:** The algorithm is a combination of well-known techniques. The innovative part is focused on how to merge all of them.\n- **Quality:** There exists other field of methods that also address feature interaction: the so-called *dynamic feature selection'. Techniques like L2X [1] are focused on the same goal, without the need of using DSNs, which highly reduces the memory consumption. Some information regarding these techniques should be included in the paper.\n- **Clarity:** Fig. 3 is clearly misleading. Although it constantly suggest the proposed method outperforms the state-of-the-art, the granularity of the y-axis is almost non-existent. There are very little differences amongst all algorithms.\n- **Significance:** I have concerns regarding two critical aspects of the experimental results:\n    1. The experimental results show very little improvements against the baseline methods. An statistical analysis is mandatory, in order to establish whether the obtained results provide a real improvement over the state-of-the-art or not.\n    2. Although I agree with the authors that feature interaction selection can provide insightful information regarding the decisions provided by the network, the authors do not mention anything related to this in the experimental section.\n\n[1] Chen, J., Song, L., Wainwright, M., & Jordan, M. (2018, July). Learning to explain: An information-theoretic perspective on model interpretation. In International Conference on Machine Learning (pp. 883-892). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hi Reviewer eecX:\n\nThanks for reviewing our paper and offering helpful comments. Below are responses to your questions.\n\n### **W1: Originality**\nPlease kindly allow me to highlight our papers' originality here, as our writings may not be optimal and can confuse the reviewer. Our initial intuition and major contribution is to extend the granularity of feature interaction selections from field level to value level. To the best of our knowledge, we are the first to propose such an extension. To tackle the *size explosion* problem caused by such an extension, we factorize the selection space. For better *selection efficiency*, we proposed a hybrid-grained selection algorithm, which efficiently selects feature interaction concurrently from both feature field and feature value. \nHopefully, our response can address the reviewer's concern about our originality, which might be caused by our writing.\n\n### **W2 and Q1: comparison with techniques from other domains**\nWe explicitly want to thank the reviewer for mentioning L2X[3]. L2X is a classic yet inspiring work that utilizes mutual information scores as guidance to conduct *feature selection*. We are informed of this work before our project. However, we decide to exclude the discussion of these works mainly because L2X targets the *feature selection* problem(FS for short), while OptFeature targets the *feature interaction selection* problem(FIS for short).\n\nThe difference between FS and FIS generally lies in three aspects. First, the selection space is different. Given $m$ feature values, FS's selection space is smaller than $m$, while FIS's selection space is around $m^t$, with $t$ denoting the order ($t \\geq 2$). Second, FIS generally requires fixed features. Third, FIS is considered an important problem in the DSN community, given that feature interaction is generally believed to be an important factor for performance boosting[1,2]. Comparably, FS is a general machine-learning problem. \n\nAlthough the differences between FS and FIS do not mean we can not borrow intuition from works like L2X, directly borrowing off-the-shell methods might not be practical. Take L2X as an example. Even if we conduct a $t$-th order($t \\ge 2)$ field-level FIS using L2X(which is much smaller than value-level FIS), the computation cost for the cross-entropy can be as large as $O(n^t * S)$, with $S$ and $n$ denoting the number of data samples and the number of feature fields in the dataset. Considering the benchmark we adopted contains $n \\ge 10$ feature fields and $S \\ge 10^7$ data samples (Please see Appendix B1 for details), it can be unbearable for real-world systems. This suggests that we need further investigation into this issue.\n\n### **W3: Clarity in Fig. 3**\nWe thank the reviewer for pointing out the granularity issue of the y-axis in Fig. 3. However, as we can observe, the y-axis in Fig. 3 represents the AUC, which reflects the performance of models. So we think this point refers to the same issue as the following point, which is the significance of our result. Kindly check CW1 in the common response for a more detailed response.\n\n### **W4 Part 1 and Q2: significance analysis**\nWe deeply appreciate the reviewer's effort in keeping this rigid standard of significance analysis, as they are extremely important for the community. Kindly check CW1 in the common response for details. In short, we add a two-tailed t-test to statistically analyze the significance of our result.\n\n### **W4 Part 2 and Q3: insightful information**\nWe try to provide some insightful information in Appendix C4 and Figure 7 due to the page limit. Our message we can easily observe from Figure 7 is that: even though the field-level FIS methods retain certain feature interactions, the value-level FIS method still drops a significant proportion on the value level, resulting in better performance. Such an observation is consistent on all three benchmarks and serves as a justification of our initial intuition: field-level FIS is not good enough.\n\n### **Q4: benefits with other algorithms**\nWe think the major benefit lies in both effectiveness and efficiency. As mentioned, we are the first to bring feature interaction selection to finer-grained and propose a hybrid-selection approach. Such a design can help us to select informative interactions, which leads to better performance. As a side-effect, given the uninformative feature interactions are dropped by our approach, the inference time and model size are slightly improved compared with other SOTA baselines, like PROFIT.\n\n### Reference:\n1. Factorization machines (ICDM 2010)\n2. DeepFM: a factorization-machine based neural network for CTR prediction (IJCAI 2017)\n3. Learning to explain: An information-theoretic perspective on model interpretation (ICML 2018)\n\n*Please notice that we use the format paper \"title(venue)\" here due to the page limit. We fully appreciate all author's contributions to the community.*\n\n\n"}]

Author rebuttal:
Hi Reviewers and PCs:

We want to thank all your effort in helping us improve this paper. Below are some of the common concerns. Kindly notice that we try to use points to answer reviewers' questions, as some of the weaknesses and questions are repetitive or similar.

### **CW1: result significance**
Thanks to all reviewers for highlighting this point. Here we list a table summarizing the recent and relevant papers working on the datasets we adopted. We list their relative improvements compared with the highest baseline in the original paper(N/A means the corresponding result is not contained). Compared with our relative improvement(last line of the table), We can observe that the improvement achieved by our proposed method makes sense over all three datasets.

| | Criteo AUC | Criteo LogLoss | Avazu AUC | Avazu LogLoss | KDD12 AUC | KDD12 LogLoss |
| ---- | ---- | ---- | ---- | ---- | ---- | ---- |
| AutoInt[1]        | +0.0052 | +0.0053 | -0.0006 | +0.0005 | +0.0084 | +0.0020 |
| AutoFIS[2]        | +0.0001 | +0.0000 | +0.0016 | +0.0009 |   N/A   |   N/A   |
| PROFIT[3]         | +0.0001 | +0.0001 | +0.0027 | +0.0046 |   N/A   |   N/A   |
| AutoIAS[4]        | -0.0045 | -0.0943 | -0.0001 | -0.0003 |   N/A   |   N/A   |
| GAIN[5]           | +0.0005 | +0.0004 | +0.0001 | +0.0001 |   N/A   |   N/A   |
| NAS-CTR[6]        | +0.0009 | +0.0004 | +0.0073 | +0.0049 |   N/A   |   N/A   |
| OptFeature(ours)  | +0.0004 | +0.0004 | +0.0019 | +0.0005 | +0.0013 | +0.0002 |

We also conduct a two-tailed t-test over our OptFeature and the best-performed baseline, the corresponding $p$-value $<0.005$. This is usually denoted as statistically significant[2,6]. Corresponding changes will be made to the paper later.

### **CW2: missing discussion of some related works** and **unclear expression of novelty**

We thank reviewers 21ZP, Bzgv and eeCX for pointing out our flaws. We revise the 2nd paragraph in Section 2.3 to address these two issues. Kindly check the following:

```
With the advancement of neural architecture search[7-9] and continuous sparsification[10], various methods have been proposed to select the informative feature interactions[2,3] and reduce computational costs. AutoFis[2] employs a sparse optimizer to select suitable feature interactions at the field level. PROFIT[3] formulates field-level feature interaction selection within a distilled search space. It employs a progressive search for efficient exploration. AutoIAS[4] takes one step further to integrate the feature interaction selection as part of the search space and jointly conduct the search with other components like MLP architecture or embedding dimension. GAIN[5], on the other hand, focuses on the DCN[11]-like architectures and conducts the feature interaction selection jointly with the model training. However, all previous works conduct feature interaction selection on the field level. Our work builds on the existing approaches for modelling feature interactions in DSNs. More precisely, We extend the selection granularity to the value level and propose a hybrid-grained selection approach.

Furthermore, there also exists some works such as OptInter[12], AutoFeature[13], and NAS-CTR[6] take a different angle and search for suitable operations (such as inner product, outer product or element-wise sum) to model each feature interaction properly. These works are perpendicular to our study.
```

### **Reference**
1. Autoint: Automatic feature interaction learning via self-attentive neural networks (CIKM 2019)
2. Autofis: Automatic feature interaction selection in factorization models for click-through rate prediction (KDD2020) 
3. Progressive feature interaction search for deep sparse network (NeurIPS 2021)
4. Autoias: Automatic integrated architecture searcher for click-through rate prediction (CIKM 2021)
5. GAIN: A Gated Adaptive Feature Interaction Network for Click-Through Rate Prediction (Sensors 2022)
6. NAS-CTR: Efficient Neural Architecture Search for Click-Through Rate Prediction (SIGIR 2022)
7. Neural Architecture Search with Reinforcement Learning (ICLR 2017)
8. Neural architecture optimization (NeurIPS 2018)
9. DARTS: Differentiable Architecture Search (ICLR 2019)
10. Estimating or propagating gradients through stochastic neurons for conditional computation (CoRR 2013)
11. Deep & cross network for ad click predictions (ADKDD@KDD 2017)
12. Memorize, factorize, or be naive: Learning optimal feature interaction methods for CTR prediction (ICDE 2022)
13. Autofeature: Searching for feature interactions and their architectures for click-through rate prediction (CIKM 2020)

*Please notice that we use the format paper "title(venue)" here due to the page limit. We fully appreciate all author's contributions to the community.*


Meta-review: After a quite active discussion period, all the concerns of the reviewers are addressed and the promised paper revision is doable for before the camera deadline. Therefore, the ACs recommand accepting the paper.
Decision: Accept (poster)