Progressive Feature Interaction Search for Deep Sparse Network
Authors: Chen Gao
Yinfeng Li
quanming yao
Depeng Jin
Yong Li
Venue: NeurIPS 2021 Poster (2021)
Keywords: Deep Sparse Network
Neural Architecture Search
Automated Machine Learning

Abstract: Deep sparse networks (DSNs), of which the crux is exploring the high-order feature interactions, have become the state-of-the-art on the prediction task with high-sparsity features. However, these models suffer from low computation efficiency, including large model size and slow model inference, which largely limits these models' application value. In this work, we approach this problem with neural architecture search by automatically searching the critical component in DSNs, the feature-interaction layer. We propose a distilled search space to cover the desired architectures with fewer parameters. We then develop a progressive search algorithm for efficient search on the space and well capture the order-priority property in sparse prediction tasks. Experiments on three real-world benchmark datasets show promising results of PROFIT in both accuracy and efficiency. Further studies validate the feasibility of our designed search space and search algorithm.

Progressive Feature Interaction Search for Deep Sparse Network
Chen Gao', Yinfeng Li", Quanming Yao Depeng Jin', and Yong Li Beijing National Research Center for Information Science and Technology, Department of Electronic Engineering, Tsinghua University 24Paradigm Inc.
1iyong070tsinghuune edu. cn
Abstract
Deep sparse networks (DSNs), of which the crux is exploring the high-order feature interactions, have become the state-of-the-art on the prediction task with high- sparsity features. However, these models suffer from low computation efficiency, including large model size and slow model inference, which largely limits these models' application value. In this work, we approach this problem with neural architecture search by automatically searching the critical component in DSNs, the feattre-interrattion layer. We propose distilled search space to cover the desired architectures with fewer parameters. We then develop progressive search algorithm for efficient search on the space and well capture the order-prioric property in sparse prediction tasks. Experiments on three real-world benchmark datasets show promising results of PROFIT in both accuracy and efficiency. Further studies validate the feasibility of our designed search space and search algorithm.
Introduction
The deep sparse network (DSN) [9, 4] is a special type of deep networks, which targets learning from sparse and categorical features. Such a learning problem frequently appears in many industrial applications. Take the click-through rate (CTR) prediction as an example, features such as user id, user age, item id, and item category, are usually sparse and high- multidimensionall [4]. In advertisement recommendation [27]. there are sparse features such as category, brand, etc. In fraud detection [23]. there are sparse features such as income level, region, etc.
A general DSN framework (see Figure 1) consists of three layers. First, the embedding layer transforms the raw sparse features into dense embeddings. Then the feature-interaction layer can construct cross-features of raw features' embeddings via cnature-interactionn oper- ation. The cross-features are further fed into the output layer to calculate the prediction score. To achieve high prediction accuracy, researchers found that the crux the deep sparse networks is to design complex feature- interaction layers [9, 31, 5, 18, 25].
Prediction Score
Output Layer
Fealtrr-interaction Layer
ARC
B0D
Embedding Layer
Sparse Features
Illustration of deep sparse net- Figure 1:
works and feature-interaction layer.
For example, in Figure 1, the second-order cross-feature, AB, constructed by two raw features, A and via feature interaction. DeepFM [9] feed the two-order crossffeatures into multi-layer perceptions for implicitly generating high-order cross-features. Further advances replace the MLP in DeepFM with more complex neural networks, including deep cross network in XDeepFM [18], outer product network in PNN [25), self-attention network in AutoInt [31], and logarithmic transformation
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

network in AFN [5]. Since these deep neural networks work as black box, high-order cross- cross-features are learned in an implicit manner.
Neural architecture search (NAS) [8, 391, has become a powerful approach in replacing human experts in proposing both effective and lightweight models for various areas such as computer vision [? I. natural language processing [29], etc. More recently, there are some works that adopt neural architecture search for designing the feature-interactioon layer in DSN. For example, SIF [40] searches simple first order interaction function between users and items; AutoFIS [20] and AutoGroup [19] explicitly search for more powerful higher order cross-features. Although the searched feature- interaction layer is more lightweight to some extent, these methods are suffering from high search costs, resulting in limited prediction performance and application value. However, these works require extremely large search costs, limiting their application values on large datasets.
In general, the human-designed DSNs are faced with the efficiency issue since they mainly pursue prediction accuracy. In fact, efficiency has a very high impact on the application value of machine learning models in real-world scenarios [3, 33, 28]. Considering that today's machine learning systems are always deployed at diverse hardware platforms, inference time and model size are the two most significant factors. Specifically,, the inference time can largely affect the time delay of the user services, which largely affects the application value. The model size can determine whether the model can be deployed at specific devices, which sets the constraint to the model's application scenarios, especially for edge devices. Therefore, these complicated deep sparse networks are still limited due to their inferior efficiency.
In this work, to address the above-mentiones efficiency-accuracy dilemma of state-of-the-ard deep sparse networks, we propose utilize NAS to design lightweight and effective models. However, this problem suffers from two challenges. exponentially with the order of cross-features and searching in this space very hard and time-consuming. Second, in sparse prediction tasks, there exists the order-priority property, which is hard to handle. Specifically, the lower-order cross-features should be assigned higher priority compared with high-order ones. To tackle them, we propose a PROFIT method (short for PROgressive Feature InTeraction Search). We first design distilled search space, a low-rank approximation of the full search space, which can vastly reduce the difficulty of finding desired models. More precisely, the full search space, which uses a tensor to represent all possible architectures, is approximated by the composition of low-rank tensors. We then develop progressive search algorithm that searches cross-features from lower orders to higher orders sequentially. When searching for the higher orders, the corresponding architecture parameters of lower orders are fixed, helping distinguish the various importance of different orders. We conduct experiments on three real-world benchmark datasets and compare both accuracy and efficiency with state-of-the- models. Empirical results demonstrate that the model searched by our PROFIT can achieve high accuracy while keeping low computation cost and model size. Further studies verify the feasibility of each design, consisting of the search space and search algorithm.
2
Related Work
2.1 Neural architecture search (NAS)
Neural architecture search (NAS) [42, 1, 39] replaces human experts in designing neural networks architectures, which has been demonstrated a promising approach in many research areas, e.g., vision and natural language processing. There are two significant perspectives that determines its utility: search space and search algorithm. The search space is defined as the set of all possible architectures. It is task-specific and should be general enough to cover existing models but not too general to incur high search costs. The search algorithm attempts to find the desired architecture in given space. good search algorithm should well explore domain properties to find good architectures and achieve high search efficiency. More recently, some pioneer works [10, 2] propose NAS methods for automatically designing lightweight models for computer vision tasks, which can reduce the inference time on edge devices. Compared with the existing works, our work approach the problem of searching for deep sparse networks, of which there exist critical challenges of large search space and high search cost.

---

2.2 Deep Sparse Networks (DSNs)
As mentioned in Section 1, deep sparse networks (DSNs) serve as the sate-of-the--ae solution for the sparse prediction task. Its crux is to extract cross-features, which can have strong abilities in prediction. from raw input features.
Here, for the sparse prediction task, we recap the evolution from earlier shallow models to DSNs. Factorization machine [27] is pioneer work that assigned a linear matrix factorization operation all possible second-order cross-features (built with two raw features), which was extended by NFM [13] and AFM [35]. To overcome the shortcomings of these shallow feature-interaction operations, deep sparse networks [9. 18, 5, 31, 25] are proposed. In general, deep sparse networks adopt neural feature-interaction layers extract cross-features, as illustrated in Figure 1. Wide&Deey [4] and DeepFM [9] feed the low-order features into multi-layer perceptrons (MLPs) to implicitly reveal any order cross-features. More complex neural networks are proposed to serve as the cnature-interatiion layer, replacing MLPs, such as the compressed interaction network (CIN) in xDeepFM [18] and outer product network in PNN [25].
There are some recent works [31,5] 5] approaching the cross-feature search of DSNs. Specifically, in AutoInt [31], the feature-interactioon layer is designed to stack multi-layer self-attention architectures; AFN [5] used a logarithmic transformation layer to convert the order to a learnable parameter and then adopted a feature summation as the feature-interaction layer, which fuzzes the order of cross-features.
However, there no model that can achieve both promising efficiency and high accuracy. Thus, is important to solve the efficiency--ccurree some recent works [22, 41] that adopt AutoML in feature embeddings, which is the input of DSNs, for reducing the parameter number of embeddings.
dilemma in DSNs. It is worth mentioning that there
2.3 Deep Network Sparsification
Besides, it worth mentioning there is another research direction closely related to this work, network sparsification [26, 34, 11, 37], which aims to prune the network components or parameters to reduce the network size while keeping a little performance decrease. Different from them, our work searches for the architecture in an interpretable and explicit manner: finding architectures that reserve important cross-features and remove useless ones; for network pruning models, embedding parameters are dropped in an implicit manner the pruning masks are always implicitly learned from data, does not provide any interpretability. In deep sparse prediction tasks, it more significant to explicitly lighten the model at the level of cross- foas-feature, while existing works of network sparsification network pruning is at the level of latent dimension.
3
The Proposed Method
3.1 The Search Problem
Existing works of DSNs [18, 31, 5] have shown that the feature-interaction layer an important component. It can be defined as network module of which the input is embeddings of raw features, and the output is interactions of embeddings. The interactions have extracted features' co-relations and can then be fed to the output layer for obtaining the final prediction results. A feature-interaction layer that extracts o-th order cross-features can be formulated as
Layer, - {fe, @ feg-.. ... 3 fe, Ico E C.). s.t. |Col
of which denotes l-size vector of involved features' indexes of a o-th cross-feature (Cos C2 .... Co are indexes of raw features), f denotes the embeddings of raw features (i.e., the output of embedding layer), and C, denotes the set of o-th order cross- cross-features on which there is sparsity constraint of No. ensuring both the quality of cross-features and the model efficiency. The feature interaction is perturbation invariant, i.e., fi @ f- = fe 3 f. The element-win wise product the most used operation for feature interaction [18, 31, 5]. Let F (fi, f2, }.r). Then, in short, the general objective of DSN can be formulated as follows.
ming (ØFFF (C))):
(2)
where L denotes the loss function and & the prediction function, of which a linear predictor or MI.P can be used. The objective of the AutoDSN problem should be finding good sets of cnature-interaction
3

---

cells, (Co), with from to O, formulated as follows.
F arg [(Ø(F, {Cot): :): Dua), (3)
[Co) min [()(F", (C.)); Pral), | S No: =o= 1,2.......
(Co}
where Pmm and Vaal denote the training and validation datasets, respectively.
For the AutoDSN problem, is a challenging problem due to multiple aspects. First, the search space for C is relatively large, which burdens the search cost. Second, the desired sets discrete, and there are also sparse constraints, making it an NP-hard problem. Third, the desired cross- features closely co-related with each other, from lower orders to higher orders, which should be carefully considered.
3.1.1 Revisiting Existing Approaches
For the search problem in (3), intuitive NAS-based solution is to use parameterized vector to rep- resent all cross-features' importance. Such solution was recently proposed by AutoFIS [20], assigning different weights to cross-features and using the continuous relaxation, similar as DARTS [21]. For o-order cross-feature, a o-th order tensor A can be introduced, where Aes.c2......, represents the weight of a cross-feature involved with raw features {fey layer in (I) expressed as
feg. ,Cc. Then, a feature-interaction
Eeecus Ac,ezr..e, (fe, O O fe.),
(4)
where Crull denotes the set of all possible cross-features. Then the NAS problem with continuous space can be re-formulated as follows,
A min ( A)); Dra); s.t. F arg min (F, Dia):
where ei is the prediction function, which first reserves No cross-fcatures with the top-N, weights to meet the sparsity constraint and then feed reserved features to output layer. That is, it is similar with in Definition Such a continuous space can overcome the issue of discrete search space and the learned architecture parameters provide explicit meaning, i.e. the importance of crasuefeatures.
It is worth mentioning that AutoInt [31] learned the importance of cross-features implicitly by self- attention layers, which thus is stilled limited. Therefore we try to explicitly search for cross-features layer. Faced with the challenge of large search space, approach the problem with the idea of encoding the space into low-rank vectors for reducing the cost.
the
3.2 Proposed Search Method
First let us recap some recent works [20, 23, 19] adopting NAS to help explicitly search powerful cross-features as the (eature-interaatiion layer. Although the searched feature-interaction layer is more lightweight, these methods suffer from high search costs. Furthermore, it ignores the different orders of cross-features, from to O, and consider them as the same, which results in a very large-size weight tensor A, with length Eo=1 C%2- where M denotes the number of raw features. Such a NAS problem with continuous space still has two critical issues.
First, representation of the search space takes too many architecture parameters, which slows down the optimization procedure. With M fields of raw features and the desired feature order O, the size architecture parameter for the search space cross-feature is close to C which approximately equals to M considering M > O. Obviously, .M too large number in the gradient- NAS. For example, the widely-used Criteo benchmark dataset has M = 39 of fields of raw features. we choose the highest order as 4, then C3.y = 82,251. Such a parameter number is much larger compared with existing works of gradient-based NAS [21, 36]. of which the number would always be fewer than one hundred. Therefore, the challenge of large search space is still unresolved.
Second, such design still ignores the order-priority property in deep sparse prediction tasks. Specifically, the property consists two characteristics revealed by human practice: higher-order features' quality can be relevant to their de- de-generateed low-order ones, and lower-order cross-features are likely to be more vital compared with higher-order ones.
To address the above-mentioned two limitations in Section 3.1, we present our solution PROFIT here with 1) a distilled low-rank approximated search space address the space issue and 2) progressive search algorithm to take order- aeder-priortt property into consideration.

---

3.2.1 Distilling the Search Space
As above, the architecture parameter, i.e., tensor A, can be too large and ignore priorities on feature interactions. This can lead to high search costs and not being capable of finding desired architectures. Here, we study the property of A and attempt to approximate it with much fewer parameters.
We conducted an experiment on the Criteo dataset and set the max order as due to the computation cost. This dataset has 39 fields, and thus weights A for all second coddr interactions can be represented by a matrix of size 39 39. Singular values from A and a randomly generated matrix are compared in Figure 2 (a). We can observe that the parameter matrix has several dominant singular values, exhibits a low-rank property. In other words, skew singular values Figure (a) shows there redundancy in the architecture matrix for the second-order cross-feature. Besides, note that feature interaction is
perturbation invariant, i.e., fi @ f2 f2 @ fI. which means the tensor should be symmetric. Thus, we propose to distilled tensor A as
Σ,_, 9 repeats times.
A = ((18))
3
(6)
where each 9r E RIXM and R < M is the positive integer. Equation (6) can also be seen as symmetric CP decomposition [17] for the tensor A. In other words, 3 and 30 refer to the approximated tensor and decomposed tensor, respectively. Subsequently, the search problem in (3) is transformed as
{8"}" min ((V(F" .(((()))(()))))))))))))))))))))) Dea), s.t. F arg min (V ()); Pua)
Comparing with the original in (5), of which size is about M and grows exponentially with O { is of size M x O and can be much smaller. Thus, the number of architecture parameters is reduced significantly.
Rolative ranking (3rd-order)
Random Malia
Singular Values
0.1
0.0
10
20
30
40
1
5 6
10
Index Singular Values
Relative ranking (2nd-order)
Singular values for the second- order cross-features.
(b) Correlations between low-order and high-order cross-features.
Figure 2: Key observations search space (left) and search algorithm (right) for PROFIT.
Remark Here we discuss research topic related to our solution, search space compression [6. 38. 12]. Different from these works such as [6] which is based on compressed sensing, our method is based on low-rank approximation. Therefore, although both these works and our work conduct the space distillation, the fundamental assumptions (vector space matrix space) are different.
3.2.2 Progressive Search Algorithm
As mentioned above, different orders of cross-features are highly correlated. To confirm that higher- order cross-features that are constructed by important lower-order cross-features tend to be important, since the ML1M dataset has fewer fields of features and then full space of high-order features is easier to optimize, we conduct the experiment on the ML1M dataset. We still use A (4) and set the maximum order to two and three. The relative ranking is computed according to the importance weights all same-order cross-features. For example, second-order cross-feature with the relative ranking as one means it is the most important second-order cross-feature For given 2nd-order cross-feature, foy fog, we can also compute the average relative ranking third-order cross- ross-features that consists of C1 and c2. We study whether important second-order cross feature generate
5

---

important third-orded cross-features. The relative ranking is shown at Figure (b), and experimental details are Appendix A.S. We can easily observe that good 2nd-order cross-feature is more likely to generate good 3rd-order cross-feature. Note that these points are not monotonically increasing. with some special points such as the 5-th one. shows the necessity of utilizing auxiliary parameters to model the high-order cross-features.
The above motivates to adopt a progressive strategy search for the architecture parameters from lower orders to higher orders, to tackle the problem of modeling the order-priority property. The complete steps are in Algorithm 1. Specifically, when searching for o-th order cross-feature, step we propose to fix the architecture parameters with order lower than i.e., {/3'..... . following existing works [21. 20, 40], at step 4 and 5, we also use gradient descent to update both architecture and embedding parameters.
}. Then,
Algorithm Progressive gradient descent.
1: for 1..... do fix ,3 88, .3........00 30-1.
2:
3: for epoch = -,...., E. do
4:
Update architecture parameters 3 using gradients (FF Deal; 5: Update embedding parameters using gradients (F.. Pim);
6: end for
7:
end for
3.2.3 Retraining the Searched Lightweight Model
Finally, as the existing NAS works [21, 42] reveal, it is better to re-train the model with the obtained embedding parameters. In our task, to ensure both the prediction accuracy and model efficiency, we only reserve the cross-features with top-N, importance weights to meet the sparsity constraints. The reason in two folds. First, some features do not benefit the task, and removing them can benefit the training procedure to achieve better prediction accuracy. Second, more importantly, only reserving No cross-features can largely reduce the model size and inference times, ensuring the real-world deployment on various hardware platforms.
Experiments
4.1 Settings
Datasets. To validate the effectiveness of our proposed PROFIT, we conduct experiments three benchmark datasets (Criteo, Avazu and MLIM) widely used in existing works of deep sparse networks [5, 31, 18] to evaluate the performance, of which the details are provided in the Appendix.
Baselines. We compare PROFIT with state-of-the-art methods for sparse prediction. We choose methods that are widely used and open-source:
First-order method:
logistic regression [14] (denoted as "LR");
Second-order methods: we compared with the most widely-used second-order method factorization machine [27] (denoted as "FM"") and the cate-offtheearl second-order method attentive factorization machine [35]' (denoted as "AFM"). NEM [13] is a de-generated version of AFM, and thus we do not compare with it.
Higher-order methods with deep feature-interactio interactierrnnn layers: deep factorization machine DeepFM [9] extreme deep factorization machine [18]² (denoted "DeepFM")) extreme deep factorization machine [18]² (denoted as "XDeepPMM", product neural network [25]$ (denoted as "PNN"), automatic feature Interaction learning [31] (denoted "AutoInt")) and adaptive factorization
network [5] (denoted as "AFN").
oitprrttttiiiioilaaaerrriiitooon www.tariaaaionnmaaiieeeee m_cchin
httpss//ggtttt.commmooooooooooor DeepPP

---

Table 1: Testing performance on Loss and AUC over three datasets.
Dataset
Critco
Avazu
MLLM
Order
Model
Loss
AUC
Loss
AUC
Loss
AUC
First
LR
0.4568
0.7936
0.3934
0.7570
0.5880
0.7418
Second
FM
0,4552
0.7954
0,3809
0.7803
0,5387
0.7938
AFM
0.4538
0.7969
0.3832
0.7764
0,5530
0.7809
High
PNN
0,4508
0.8029
0.3798
0.7839
0.5254
0.8042
AutoInt
0.4443
0.8074
0.3786
0.7849
0.5285
0.8014
AFN
0.4428
0.8094
0,3803
0.7818
0.5294
0.8038
DeepFM
0,4469
0,8050
0.3787
0.7844
0,5318
0.8044
XDeepFM
0.4430
0.8093
0.3781
0.7856
0,5280
0.8069
AutoFIS
0.4537
0.7971
0.3815
0.7794
0,5353
0.7956
NAS
PROFIT
0.4427
0.8095
0.3735
0.7883
0.5281
0.8077
These human-designed baselines models are also selected as the representative DSN models for comparison in the AutoFIS paper. Besides, there is another recent work, i.e., AutoGroup [19], which similar to AutoFIS. Considering it has close prediction performance and search cost compared with AutoFIS, only compare AutoFIS in our experiments. There some recent works [30. 16] approaching the problem via automatically fusing human-designed models, which is still restrained by human-designed feature-interaction layers.
As mentioned above, there are some pioneer works that propose NAS methods for lightweight CNN models for computer vision tasks. Here we do not compare with them since it is hard to adapt them to the deep sparse networks.
Evaluation Metrics. We evalua te models from two perspectives, prediction performance and inference efficiency. For accuracy, we use two standard metrics, Logloss (denoted as "Loss") [24] and AUC [7]. following [5, 9, 18, 27, 25). For efficiency. we also use two standard metrics, inference time and model size, which are also widely used in existing works of deep sparse networks [31, 15].
Hyper-parameter Settings. To ensure the reproducibility of experimental results, here we introduce the implementation setting, of which more details are in Appendix. We implement PyTorch. We apply Adam with learning rate of 0.001 and a mini-batch size of 4096, a widely-use setting in existing works [5, 31]. We set the embedding sizes to 16 in all the models. We the same neural network structure ((400, 400, 400)) for all methods that adopt MLP for fair comparison, following 31J. All the other syper--arameters are tuned on the validation set. To make the results more reliable, for each instance, we run the repetitive experiments three times with different random seeds and report the average value.
methods using
4.2 Benchmark Comparison
Prediction accuracy. In Table 1, we present the prediction performance of all these methods. from which we have the following observations. First, models with higher-order cross-features can achieve better prediction performance than lower-order ones, which verifies the rationality of exploring high- order feature interactions. Second, with the carefully designed search space and search algorithm, PROFIT can achieve the best prediction performance three benchmark datasets (or very close to the best baseline). Third, the deep sparse networks achieve better performance compared with shallow models, which demonstrates the necessity of designing powerful deep sparse networks. Note that due to the limitations in search space and algorithm, which make the searching procedure very slow, AutoFIS does not achieve promising results compared with human- designed ones?.
Efficiency. We further compare the efficiency and present the results in Figure 3. We can observe that PROFIT is lightweight and fast-to-infes among all compared methods. In real-world applications, both the accuracy and the inference efficiency should be considered. Since our method not only achieves promising accuracy but also reduces the inference time, it has high application value in practice.
7 AAtoFIIS achieves better performance compared with numan-crafted baselines when setting max order of cross-feature as three, as shown by the original paper. However, during the experiments in our normal hardware platform, for the Criteo dataset and Avazu dataset, it cannot be well optimized due to too large search space and computation costs. Thus, we only report the results setting order two for these datasets.
7

---

Criteo
0.790
Avaza
0.510
ML.M
0.810
0.805
PMCF
Azant
Ale
0.785
PRO-11
Autchn
ALC
0.805
Aurciit
FROHT
A'
ATN
AFN
0.800
Rei
PAA
0.800
PNNN
iDDoPFM
KCeepFM
M
DespFM Ausrts
DocsFFM
0.795
Azr's is
0.780
Autofit
0.79%
(5)
Inference
Inferenee Time (6)
Critco
Avazu
MI.IM
0.810
0.805
PROFIT
PROFIT
0.805
PROTTT
Auport
0.785
Austhi
Austhi
APN
AEN
0.800
PNN
0.800
:Ceepr's M
XDoopPM
XEeouFM
Doep-F
0.796
Auser115 15
0.780
15
10
20 Model Size (MB)
30
(MI)
Model Size (MB)
Performance comparison of PROFIT and baseline models. ("Size" refers to the model size without feature embedding layer. "Time" refers the inference time for million samples.)
Figure 3:
0.900
1st
0.875
0.54
2nd 3rd
14l 3rd
0.850
Table 2: Performance of searched model with
Logloss
0.53
0.825
0
progressivelone-step/ramaae search. Due lack of space, only AUC is reported.
0.800
0.52
0.775
0.750
Search Algo.
Criteo
Avazu
ML.1M
0.51
0.725
Random with PS
0,7718
0.7499
0.7692
Ote-step Search
0.8077
0.7851 1
0.8053
0.50
Logloss
0.700
PROFIT
0.8095
0.7883
0.8077
Figure 4: Performance of utilizing different or- ders' cross-features on MLIM dataset.
4.3 Understanding the Search Strategy
Progressive search. In order to validate the effectiveness of the progressive manner in our search algorithm, we compared with the performance that utilizes one-step manner. can regarded the de-generated version of our PROFIT that replaces our progressive search with step, and the details of its searching procedure are shown in Appendix A.4. We also compare the random search with PS (parameter sharing), of which architecture parameters are randomly assigned, and the model parameters are shared during the search procedures. The final performance of the searched architectures is shown in Table 2. The curve of validation performance during the searching procedure shown in Figure 5.
From these results, can observe that random search achieves the worst performance since it does not explore any domain properties, as we have analyzed Section 3. 1. Also, with the simple one-step search algorithm, the performance is poorer compared with the progressive search algorithm. This validates the rationality of our design of progressive search, which can well capture the order- priooity property of cross-fcaturre in the deep sparse prediction task. In short, the effectiveness of our design of progressive search well validated by the results.
Distilled search space. First, we conduct experiments to study the searched results if different search space is used. We compared our distilled space, i.e., Eqn (7), with full space, i.e. Eqn (5). We choose the order of three. The curve of validation performance during the search procedure of two large-scale datasets, Avazu and Criteo, shown in Figure 6, and the final results are shown in Table 3. We can observe that searching in the distilled space can easily reach promising validation performance, However, when searching in the full space, requires higher computation cost, and result, it is hard to obtain good architecture parameters. As result, the distilled search space

---

0.8
Veldaton AUG
Validation AUC
0.7
0.6
0.50
0.6
5 6
8
9
10
11
5
8 10
Search Tima (Hour)
Search Time (Hour)
(a) Criteo.
(b) Avazu.
Figure 5: The validation performance during searching with progresiiveoon--ttt//raddooo search.
obtains better prediction performance. These results show that with a reduced budget for searching, using the distilled space is much better choice compared with the original full search space.
0.8
Validation AUC
0.7
Validation
0.8
0.50
0.50
Search Time (Hour)
Search Time (Hour)
(a) Criteo.
(b) Avazu.
Figure 6: The validation performance during searching with different search space (please note that we use validation performance illustrate the searching procedure following existing NAS works).
Table 3: Searched model's performance with different search space.
Dataset
Criteo
Avazu
MLIM
Model
Loss
AUC
Loss
AUC
Loss
AUC
Full Space
0.4533
0.7975
0.3802
0.7807
0.5353
0.7957
PROFIT's Space
0.4427
0.8095
0.3735
0.7883
0.5281
0.8077
5 Conclusion
In this work, we approached the problem of deep sparse prediction with neural architecture search. We propose distilled search space which is low-rank approximation of the full space, significantly reducing the search cost. We then propose a progressive differentiable search algorithm that can efficiently search for desired models in the space while being consistent with the order- order-priority property. Extensive experiments on three real-world benchmark datasets demonstrate that our proposed PROFIT can achieve both high accuracy and high efficiency. Further studies illustrate that the proposed search space and search algorithm of PROFIT is feasible. As for future work, we will explore more how dataset characteristies affect the searching results.
Acknowledgment
This work was supported by National Natural Science Foundation of China under U1936217, 61971267, 61972223, 61941117, 61861 136003.

---

References
[1] Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network archi- tectures using reinforcement learning. In International Conference on Learning Representations, 2017.
[2] Han Cai, Chuang Gan, Tianzhe Wang. Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations, 2019.
[3] Han Cai, Ligeng Zhu, and Song Han. Prokylessnas: Direct neural architecture search on target task and hardware. In International Conference on Learning Representations, 2019.
[4] Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye. Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recommender systems. In 1st workshop on deep learning for recommender systems, pages 7-10, 2016.
[5] Weiyu Cheng, Yanyan Shen, and Linpeng Huang, Adaptive factorization network: Learning adaptive-order feature interactions. In AAAI Conference on Artificial Intelligence, 2020.
[6] Minsu Cho, Mohammadreza Soltani, and Chinmay Hegde. One-shot neural architecture search via compressive sensing. arXiv preprint arXiv: 02069, 2019.
[7] Corinna Cortes and Mehryar Mohri. Auc optimization vs. error rate minimization. In Advances in Neural Information Processing Systems, pages 313-320, 2004.
[8] Thomas Elsken, Jan Hendrik Metzen, Frank Hutter, et al. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55): 2019.
[9] Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: factorization- machinee based neural network for prediction. In International Joint Con- ference on Artificial Intelligence, 2017.
[10] Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In European Conference on Computer Vision, pages 544-560 Springer, 2020.
[11] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruming, trained quantization and huffmman coding. In International Conference on Learning Representations, 2016.
[12] Elad Hazan, Adam Klivans, and Yang Yuan. Hyperparameter optimization: spectral approach. In International Conference on Learning Representations, 2018
[13] Xiangnan He and Tat-Seng Chua. Neural factorization machines for sparse predictive analytics. International ACM SIGIR conference on Research and Development Information Retrieval, pages 355-364, 2017.
[14] David W Hosmer Jr, Stanley Lemeshow, and Rodney X Sturdivant, Applied logistic regression, volume 398. John Wiley & Sons, 2013.
[15] Yuchin Juan, Yong Zhuang, Wei-Sheng Chin, and Chih-Jen Lin. Field-aware factorization machines for CTR prediction. In ACM Conference on Recommender Systems, pages 43-50, 2016.
[16] Farhan Khawar, Xu Hang, Ruiming Tang, Bin Liu, Zhenguo Li, and Xiuqiang He. Autofeature: Searching for feature interactions and their architectures for click-through rate prediction. In ACM International Conference on Information Knowledge Management, pages 625-634, 2020.
[17] Tamara G Kolda and Brett Bader. Tensor decompositions and applications. SIAM review, 51(3):455-50 2009.
10

---

[18] Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. xdeepfm: Combining explicit and implicit feature interactions for recommender systems. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1754-1763, 2018.
[19] Bin Liu, Niannan Xue, Huifeng Guo, Ruiming Tang, Stefanos Zafeiriou, Xiuqiang He, and Zhenguo Li. Autogroup: Automatic feature grouping for modelling explicit high-order feature interactions in CTR prediction. In International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 199 2099200,110 2020.
[20] Bin Liu, Chenxu Zhu, Guilin Li, Weinan Zhang. Jincai Lai, Ruiming Tang, Xiuqiang He, Zhenguo Li, and Yong Yu. AutoFIS: Automatic feature interaction selection in factorization models for click-through rate prediction. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2020.
[21] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. 2019.
[22] Siyi Liu, Chen Gao, Yihong Chen, Depeng Jin, and Yong Li. Learnable embedding sizes for recommender systems. In International Conference on Learning Representations, 2021.
[23] Yuanfei Luo, Mengshuo Wang, Hao Zhou, Quanming Yao, Wei-Wei Tu, Yuqiang Chen, Wenyuan Dai, and Qiang Yang. Autocross: Automatic feature crossing for tabular data in real-world applications. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1936-1945, 2019.
[24] Kevin P Murphy. Machine learning: a probabilistic perspective. MIT press, 2012.
[25] Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, and Jun Wang. Product-based neural networks for user response prediction. In International Conference Data Mining, pages 1149-1131 1154. IEEE, 2016.
[26] Russell Reed. Pruning algorithms-a survey. IEEE Transactions on Neural Networks, 4(5):740- 747, 1993.
[27] Steffen Rendle. Factorization machines. In IEEE International Conference on Data Mining, pages 995-1000. IEEE, 2010.
[28] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4510 2018.
[29] David So, Quoe Le, and Chen Liang. The evolved transformer. In International Conference Machine Learning, pages 5877-5886, 2019.
[30] Qingquan Song, Dehua Cheng, Hanning Zhou, Jiyan Yang, Yuandong Tian, and Xia Hu. Towards automated neural interaction discovery for click- rate prediction. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 945-955, 2020.
[31] Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang, Autoint: Automatic feature interaction learning via self-attentive neural networks. In ACM International Conference on Information and Knowledge Management, pages 1161-1170. 2019.
[32] Xiaoyuan Su and Taghi M Khoshgoftaar. A survey of collaborative filtering techniques. Ad- vances in Artificial Intelligence, 2009, 2009.
[33] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. Mnasnet: Platform-waan neural architecture search for mobile. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2820-2828, 2019.
[34] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in Neural Information Processing Systems, 2016.
11

---

[35] Jun Xiao, Hao Ye, Xiangnan He, Hanwang Zhang, Fei Wu, and Tat-Seng Chua. Attentional factorization machines: Learning the weight of feature interactions via attention networks. In International Joint Conference Artificial Intelligence, 2017.
[36] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. In International Conference on Learning Representations, 2019.
[37] Niannan Xue, Bin Liu, Huifeng Guo, Ruiming Tang, Fengwei Zhou, Stefanos P Zafeiriou, Yuzhou Zhang, Jun Wang, and Zhenguo Li. AutoHash: Learning higher-order feature inter- actions for deep CTR prediction. IEEE Transactions on Knowledge and Data Engineering, 2020.
[38] Yibo Yang, Hongyang Li, Shan You, Fei Wang, Chen Qian, and Zhouchen Lin. Ista-nas: Efficient and consistent neural architecture search by sparse coding. Advances in Neural Information Processing Systems, 33:10503-10513, 2020.
[39] Quanming Yao, Mengshuo Wang. Yuqiang Chen, Wenyuan Dai, Yu-Feng Li, Wei-Wei Tu, Qiang Yang, and Yang Yu. Taking human out of learning applications: A survey automated machine learning. Technical report, 2018
[40] Quamming Yao, Xu, Wei- Tu, and Zhanxing Zhu. Efficient neural architecture search via proximal iterations. In AAAI Conference Artificial Intelligence, volume 34, pages 6664-6671, 2020.
[41] Xiangyu Zhao, Haochen Liu, Hui Liu, Jiliang Tang, Weiwei Guo, Jun Shi, Sida Wang, Huiji Gao, and Bo Long. Autodim: Field-aware embedding dimension searchin recommender systems. In Web Conference, pages 3015-3022, 2021.
[42] Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2017.
12


Reviews:
[{"review_id": "p6AzvSoDgLc", "reviewer": "Reviewer_5u8P", "summary": "This paper proposes a neural architecture search approach, PROFIT,  for deep sparse networks.  PROFIT introduces a distilled low-rank search space and a progressive search algorithm from the lower orders. The proposed method is evaluated on three benchmark datasets and the authors conduct extensive ablation study and discussion on the method. The paper is well written and very easy to follow even for people without a NAS background.\n \n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Originality: Are the tasks or methods new? Is the work a novel combination of well-known techniques? (This can be valuable!) Is it clear how this work differs from previous contributions? Is related work adequately cited?\n\nNeural architecture search is no longer a new task. However, this work proposes a very reasonable solution to address the high cost of neural architecture search. It also puts the focus on interpretable sparse architecture search, which seems to be a novel and interesting perspective.  The discussion with the previous work is extensive and also easy to follow. \n\n2. Quality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?\n\nThe submission is technically sound.  The authors provide an extensive evaluation of three benchmark datasets. For example,  they provide motivating plots (e.g., Figure 2) to help the readers understand their approach, which is very helpful.  The method itself is kind of straightforward, but it's well-motivated and also quite effective. \n\n3. Clarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)\n\nThis paper is very well written and very easy to follow. Even readers outside the NAS community can understand it.  The paper structure is clear.  There are a few typos in the paper, though. For example, the Aycronm PROFIT directly pops up in the abstract without reference. Line 260 extra space in the word \"evaluate\".  \n\n4. Significance: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it p\nrovide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?\n\n\nThe results seem important and  it also achieves the state-of-the-art performance on three datasets.    It might be inspiring for others in the NAS field  and the deep  sparse networks. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "fxrGLFZzeL9", "reviewer": "Reviewer_Uczd", "summary": "This paper targets applying NAS to Deep Sparse Network (DSN) domain. To search for a good DSN, previous methods that apply DARTS based method to search an encoding vector, where each element represents the probability to select feature-interaction. Due to the interaction grows in a factorial way, the search parameters will soon become intractable. This work proposes to decompose this vector into multiple low-rank ones, to reduce the search space dramatically.\n\nExperiments on various datasets show a clear improvement comparing to the previous baseline AutoFIS. I would recommend accepting this one but since I am not really familiar with the DSN domain, I might change my mind based on other reviews.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strength\n\n+ Interesting idea to consider order-priotity property in DSN\n+ Reasonable toy-case demonstration, showing that the matrix of second-order interaction exploits a low-rank property comparing to randomly generated matrices, this makes the proposed low-rank method conceptually meaningful.\n+ This method reduce the search parameters from O(M^O) to O(MO).\n+ Results shows constant improvement over the previous baseline AutoFIS.\n\n# Weakness\n\n- Motivation\nThis work observes the previous baselines (AutoFIS and another one) that applies NAS technique to deep sparse network have extreme high search cost. However, I read the AutoFIS paper that uses DARTS as search strategy, where they only need maximum 12 min in their Table 1. To me this is not extremely high cost. What's the value of your work in terms of this? \n\nIn addition, in Line 40 you go back to discuss the human-designed DSNs that has poor computational cost accuracy trade-off. As your method is not the first one apply NAS into DSN domain, to me, what's more interesting is to discuss the methods with NAS. Are they also suffering from this? Could you provide concrete numbers rather than `inferior efficiency`?\n\n- Comparision with other methods\nIn the section 2.2, What's the difference in terms of methodology of other DSN with NAS baselines with your PROFIT? In other words, what makes your method search fast? If I understand correctly, it's your contribution to formulate the space encoding into low-rank PCA-like vectors to reduce the cost. Anyway, adding this info here might further enhance this section. And you probably want to emphasize this in the introduction as well.\n\n- order-priority property, any justification?\nThe results shown in fig 2 only shows comparing to random matrices. This may not be sufficient to show the real order-priority property isn't it? \n\n## minor issues\n- AutoDSN search space\nThe paper claims C is relatively large that burdens the search cost, isn't this true for any NAS problem? DARTs for example has a space with billions of architectures. I remotely sense that this search space is somehow different with the traditional one, as the feature interactions may not be simplified by the weight sharing approach. But this is not really clear to me, hope the authors can clarify further why this space is a NP-hard problem and why the traditional NAS approach cannot resolve this. I saw section 3.1 answers part of my problem. Please correct me if I understand anything wrong.\n\n\n- Writing issues\nPage 2 Line 38 - 39\n> However, these works require extremely large search cost.... \n\nThis line is redundant with the previous sentence \n> these methods are suffering from high search\n38 costs, resulting in limited prediction performance and application value\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Nothing special.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "NQ4nCk3stzz", "reviewer": "Reviewer_B6yE", "summary": "In this paper, the authors study the problem of deep sparse prediction based on NAS.\nIn order to reduce the search cost, they propose a distilled search space, which is a low-rank approximation of the full space.\nThey also propose a progressive differentiable search to capture the order-priority in sparse prediction.\nNumerous experiments are conducted to demonstrate the effectiveness of the proposed method.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, the paper is written well and easy to follow. However, I still have several concerns that the authors need to address:\n1. **Questions about distilling the search space.**\n(1) As Figure 2 shows, the authors only use the second-order interaction to conduct the experiment, so does the higher-order interaction also have similar properties? Please give some theoretical proofs or display the experimental results;\n(2) Equation 5 is very confusing. Please explain the following questions in detail: \n    - Why A can be approximately equal to this form? \n    - What conditions are needed to meet this approximation and how about the error? \n    - What is the specific meaning of R, and how to set the value of R in the experiments. \n    - Please give the detailed proof and deduction process of equation 5.\n\n2. **Questions about efficiency.**\n(1) In lines 213-214, the authors point that only N_0 important weights are retained. What is the basis for the selection of N0, what is the specific value, and how the selection of N_0 affects the accuracy and effectiveness? Please give more detailed experiments and explanations.\n(2) In lines 262-263, the authors point that the experiment only evaluated the inference time and the size of the models. I hope that the author should also give the computation cost during training and inferences, such as FLOPs or MACs or other indicators to illustrate the efficiency.\n(3) The author just compared only one NAS-based baseline in terms of efficiency. More challenging baselines are needed, such as the methods based on pruning [1,2].\n\n3. **Questions about experiments.**\n(1) The performance improvement in Table 1 seems limited. For example, the AUC in the Criteo dataset is only increased by 0.0001 compared with AFN.\n(2) In lines 112-113, the author mentioned that some similar NAS-based methods usually require a large search cost, but the author did not compare the search efficiency with these methods in the experiment.\n\n[1] Liu, Siyi, et al. \"Learnable Embedding Sizes for Recommender Systems.\" arXiv preprint arXiv:2101.07577 (2021).\n[2] Zhao, Xiangyu, et al. \"Memory-efficient embedding for recommendations.\" arXiv preprint arXiv:2006.14827 (2020).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please refer to the main review.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "FIzAuO2LIm", "reviewer": "Reviewer_ewUq", "summary": "This paper proposes a method for searching for the interaction of the feature columns in a CTR prediction application. The paper title suggests an application for deep sparse networks but it doesn’t seem to be explicitly tied with the requirement of the input being sparse. To this end, the paper proposes an iterative algorithm that incrementally learns the interaction tensor from lower rank to higher rank, and from lower order to higher order. Empirically it observes some benefits compared to current competitive methods on Criteo, Avazu and ML1M datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- The paper is clearly written.\n- The proposed method is clear and relatively simple.\n- It seems to outperform existing methods on several benchmarks.\n\n---------------------------------------------\n### Weaknesses\n- **Significance of results:** While the proposed method almost achieves the best result across the board, I am not sure how significant it is. If we look at the AUC score, then it only improves upon the second best by 0.0002, 0.0027, 0.0008. I am no expert on these benchmarks but it still seems too small of a difference. How much difference does this bring in practice?\n- **Limited scope and comparison to a more general architecture like Transformer:** As I mentioned in the summary, the paper focuses on the application of CTR prediction using a deep sparse network, but there is nothing in the method that actually relies on the input feature being sparse. In other words, the proposed method can be applied to anything that aims to look for high-order interactions among different dense features, but the scope is intentionally narrowed down for no reason. The compared methods are all densely populated in a very narrow domain. The author could also compare it with more general architecture like Transformers. One layer of the transformer allows pairwise interactions, and more layers can allow higher-order interactions. It is unintuitive what would be the benefit to directly learn an interaction tensor via the proposed approximation, as opposed to a more general method that involves attention.\n- **Entangling rank vs. order:** I found it strange that the paper entangles the two concepts of rank and order. According to Algorithm 1, it is unclear why we want to increase both the rank and the order by 1. Alternatively, you could increase the order by 1 but have a rank-3 approximation throughout. But the current algorithm learns one vector at a time, so it means that the rank is also the same as the order.\n- **Comparison to asymmetric low-rank space:** There can be an additional comparison in between “full space” and “PROFIT’s space,” where the space is low rank but asymmetric. In other words, it is unclear what benefit the assumption of symmetric tensors brings.\n\n---------------------------------------------\n### Conclusion\nIn conclusion, the contribution of this paper is clear. But I am not sure about its significance in terms of the results as well as the scope of the paper. Therefore, my initial rating is slightly below the bar.\n\nUpdate: the rebuttal addresses most of my concerns and I increased the score from 5 to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "0wl9d0pSxfg", "reviewer": "Reviewer_hemK", "summary": "The authors consider the problem of designing accurate and computationally efficient deep sparse networks (DSNs), which are an important problem in applications with sparse features such as click-through rate or movie recommendation. The authors propose a new approach based on neural architecture search (NAS) using two techniques: distillation of the search space (of cross-features), and a progressive search algorithm. They show that their approach leads to improvements in both accuracy and inference time, on three datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n- Machine learning problems with sparse features have popular applications, and so empirical improvements could have high impact.\n- The two parts of the algorithm, distillation of the search space and progressive search over feature orders, appears to be well-suited to the problem.\n- The results are strong. The authors pick a varied list of first-, second-, and high-order methods for comparison, as well as AutoFIS, the only other NAS-based DSN work that is open-source.  Table 1 and Figure 3 seem convincing.\n\n## Points that could be improved\n- Novelty/Impact. NAS has been applied to DSNs at least twice before, as cited by the authors. The ideas of distillation, and progressive search, have been known in the NAS community, although not used the same way as this work (e.g. [1] uses a technique to compress a DARTS-like search space). The main novelty/impact is combining and applying these ideas to the DSN setting, and showing that it leads to strong empirical performance. The technique is tailored to this application.\n- Reproducibility: the authors give specific implementation details in the appendix, which is good. However, they do not release their code? They mention in the checklist (twice) that the code is released, but I could not find it. Can somebody correct me if I am wrong? Reproducibility is an important consideration, both in the field of NAS where reproducibility is a problem, and also for papers in which the largest contributions are empirical and application-specific, such as this paper. For example, open-source code can lead to higher impact.\n- The authors conduct ablation studies for both the progressive search strategy and the distillation strategy, as well as a few more in the appendix (different ranks, output layer, embedding size), which is excellent. However, it would be helpful to put these ablation studies onto one plot, to better answer the question: which of the main techniques helped more (e.g. distillation or progressive search)?\n- The paper is written okay. There are some places that could be made clearer. Section 3.3. seems a bit out of place, and related work shows up in different places, and for example, the footnotes on page 7 could me moved to the appendix.\n\n## Overall\nThe paper is okay. The authors study DSNs using NAS, and the techniques are not extremely novel, but the execution of the techniques is well-done and leads to strong performance. Reproducibility could be improved. The decision could go either way, and currently I lean towards weak reject.\n\n[1] https://arxiv.org/abs/1906.02869\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not explicitly discuss limitations, as far as I could tell. Societal impact was addressed in the checklist.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical issues.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes using NAS for learning the feature interaction space for deep sparse networks that are used for high dimensional sparse networks commonly encountered in recommendation systems. The discovered architectures have higher efficiency compared to the baselines. 

During discussion period some points of confusion consistently came up regarding order-vs-rank and the fact that it was not explicitly clear that the search space in feature-interactions is several orders of magnitude bigger than that used in DARTS for CNNs due to the cardinality of the operators (7 or so in DARTs vs. 39 here).  This is a point that the authors can make clearer in the paper in the beginning itself. Otherwise the difficulty of the problem does not come about till later. 

Reviewers have also pointed out various ways to make presentation better like not using forward references to acronyms before definition. Since the main contribution of this paper is the search space and the progressive search over feature orders and not the NAS search technique the authors should emphasize that part explicitly and play to the strengths of the paper in presentation. 
Decision: Accept (Poster)