Automatically Learning Feature Crossing from Model Interpretation for Tabular Data
Authors: Zhaocheng Liu
Qiang Liu
Haoli Zhang
Venue:  (2020)
Keywords: AutoML
feature crossing
interpretation

Abstract: Automatically feature generation is a major topic of automated machine learning. Among various feature generation approaches, feature crossing, which takes cross-product of sparse features, is a promising way to effectively capture the interactions among categorical features in tabular data. Previous works on feature crossing try to search in the set of all the possible cross feature fields. This is obviously not efficient when the size of original feature fields is large. Meanwhile, some deep learning-based methods combines deep neural networks and various interaction components. However, due to the existing of Deep Neural Networks (DNN), only a few cross features can be explicitly generated by the interaction components. Recently, piece-wise interpretation of DNN has been widely studied, and the piece-wise interpretations are usually inconsistent in different samples. Inspired by this, we give a definition of interpretation inconsistency in DNN, and propose a novel method called CrossGO, which selects useful cross features according to the interpretation inconsistency. The whole process of learning feature crossing can be done via simply training a DNN model and a logistic regression (LR) model. CrossGO can generate compact candidate set of cross feature fields, and promote the efficiency of searching. Extensive experiments have been conducted on several real-world datasets. Cross features generated by CrossGO can empower a simple LR model achieving approximate or even better performances comparing with complex DNN models.

Under review as a conference paper at ICLR 2020
AUTOMATICALLY LEARNING FEATURE CROSSING FROM MODEL INTERPRETATION FOR TABULAR DATA
Anonymous authors
Paper under double-blind review
ABSTRACT
Automatically feature generation a major topic of automated machine learn- ing. Among various feature generation approaches, feature crossing, which takes cross-product of sparse features, is promising way to effectively capture the in- teractions among categorical features in tabular data. Previous works on feature crossing try to search in the set of all the possible cross feature fields. This is ob- viously not efficient when the size of original feature fields is large. Meanwhile, some deep learning-batt based methods combines deep neural networks and various interaction components. However, due to the existing of Deep Neural Networks (DNN), only few cross features can be explicitly generated by the interaction components. Recently, piece- wise interpretation of DNN has been widely studied, and the piece-wis wise interpretations are usually inconsistent in different samples. Inspired by this, we give a definition of interpretation inconsistency in DNN, and propose a novel method called CrossGO, which selects useful cross features cording to the interpretation inconsistency. The whole process of learning feature crossing can be done via simply training DNN model and logistic regression (LR) model. CrossGO can generate compact candidate set of cross feature fields, and promote the efficiency of searching. Extensive experiments have been con- ducted on several real-world datasets. Cross features generated by CrossGO can empower simple LR model achieving approximate or even better performances comparing with complex DNN models.
INTRODUCTION
Recently, automated machine learning (AutoML) has been widely studied, and is proven a practical and promising machine learning technique (Yao et al. 2018). AutoML aims provide a easy way to apply machine learning technique, and automate the procedure of machine learning partly or thoroughly. This reduces labour on data preprocessing, feature engineering, model selection, hyper- parameter tuning, model training and performance evaluation.
Among different components of machine learning, the quality of features plays extremely im- portant role (Liu & Motoda 1998; Domingos 2012). Accordingly, to improve the performances of machine learning tasks, various automatic feature generation methods have been proposed (Chapelle
ct al. 2015, Katz et al. 2016 Zhang et al. 2016;; Qu et al. 20166] Juan et al., 2016 Blondel et al. 2016 Guo al. 2017: Lian al. [2188] Luo et al.) 2019). Among these feature generation meth- ods, feature crossing, which takes cross product of sparse features, is a promising way to capture the interactions among categorical features and is widely used enhance the performance of machine
learning on tabular data in real-world applications (Chapelle et al. 2015 Cheng et al. 2016 Luo
et al. 2019).
Previous works on feature crossing mostly try to search in the set of all the possible cross features Rosales et al. 2012; Chapelle et al. 2015 Katz et al.) 2016). Based on these methods, AutoCross (Luo et al. 2019) employs engineering optimization tricks, and accelerates the searching process. However, search-based methods are not efficient when the size of original feature fields is large. Mcanwhile, some deep learning- based methods combine deep neural networks and various interac- tion components (Qu et al. 2016: Cheng et al.) 2016)] Wang et al.) 2017 Guo et al. 2017)] Lian et al.] 2018). However, due the existing of Deep Neural Networks (DNN), which capture variety of

---

Under review as a conference paper at ICLR 2020
interactions between original features, only few cross features can be explicitly generated by the interaction components.
As mentioned above. DNN is powerful method for capturing various feature interactions in its hidden layers (Guo et al, 2017]. This means, DNN can generate features implicitly, but can not provide interpretable cross features explicitly. Recently, the interpreration of deep models has drawn great attention in academia, and mostly focus on piece- wise interpretation, which means assigning a piece of local interpretation for each sample (Bastani et al., 2017: Alvarez-Melis & Jaakkola 2018) Chu et al. 2018). This can be done via the gradient backpropagation from the prediction layer to the feature layer. Usually, local interpretations of a specific feature are inconsistent different samples. And large interpretation inconsistency indicates that, the corresponding feature has interacted with other feature fields in the hidden layers of DNN. Therefore, the interpretations of DNN can help us find useful cross features.
Accordingly, in this paper, we propose a novel method called CrossGO, which can directly and automatically learn useful cross feature fields from the interpretation of DNN. (1) First, we train a common sparse DNN with the whole training set. (2) Then, for each sample in the validation set, we calculate the gradients of the output predictions with respect to the input features, i.e., local interpretations. We also calculate the global interpretation of specific feature as the average value of all the corresponding local interpretations. For a specific feature in specific sample, if the corre- sponding local interpretation is far from the corresponding global interpretation, i.e., interpretation inconsistency is large, regard this feature interacted with others by DNN in the sample. Thus, obtain a set of feasible features that works for feature crossing in each sample.
(3) Then, we employ greedy algorithm to generate a global candidate set containing both second-order and high-order cross feature fields. (4) With the the candidate set, we can train a simple LR model, therefore rank and select feature fields according to their contribution measured in the validation set. So far, we can obtain the final set of cross feature fields.
We have conducted experiments on several real-world datasets. The size of candidate set of cross feature fields is constrained to be 2N, where N is the number of original feature fields. To be noted, 2N is extremely small compared to the whole set of second-order and high-order cross feature fields, especially when N is large. And a simple LR model empowered by the final set of cross feature fields achieves approximate or even better performances comparing with DNN on all datasets. This whole process of learning feature crossing can be done via simply training DNN model and LR model. This shows that, our proposed CrossGO method can automatically generate a compact candidate set of cross feature fields, and efficiently find enough useful cross features.
2 MOTIVATION
In this section, we first summarize the advantages of feature crossing and the drawbacks of conven- tional methods. Then give brief analysis about why and how we learn feature crossing from the interpretation of DNN.
2.1 FEATURE CROSSING
According to the definition in previous works (Cheng et al. 2016 Luo et al. 2019), we can conduct feature crossing as
P0 " $t...... "Iy f & ز &... @ 8Ifn.m
where @ denotes cross-product, f2.......... is the corresponding generated cross feature, and y is binary feature associated with field e.g. feature "gender=male" associated with field " "gender". Via feature crossing, the cross feature of male English-speaking student can be denoted as ("gender=male" and "language=English" and "job=student"). Moreover, considering feature dis- cretization has been proven useful to improve capability of numerical features {Liu et al., 2002 Kotsiantis & Kanellopoulos 2006 Chapelle et al. 2015), we can conduct feature discretization numerical feature fields to generate associated binary features.
Via learning feature crossing, several advantages can be achieved: (1) new features can be generated;
(2) cross features, instead of latent embeddings in deep neural networks, are highly interpretable [uuo et al. 2019); (3) simple linear machine learning methods can take of cross features to
2

---

Under review as a conference paper at ICLR 2020
achieve approximate even better performances comparing with complex nonlinear methods (Luo 2019;; ; (4) it is flexible and suits for large-scale online tasks (Cheng et al. 2016).
iet al.
First attempts on feature generation mainly focus on generating second-order features (Rosales et al, Cheng et al.] 2014; Chapelle et al., 2015; J Juan et al.] 2016 Katz et al. 2016). In (Chapelle) 2015), the authors try to generate and select second-order cross features according to condi- tional mutual information. The main problem of conditional mutual information is that, once the mutual information of an original feature is high, the generated cross features containing it will also have high conditional mutual information. ExploreKit (Katz et al. (016) presents a framework for generating candidate features and ranking them. The proposed ranking and selecting methods are mainly based on the corresponding performances in classifiers, which is not efficient when the fea- ture set is large. Moreover, these conventional methods fail to capture powerful high-order features.
2012
et al.
Nowadays, deep learning- based prediction methods have shown their effectiveness. Among these methods, some try to generate and represent features implicitly or explicitly via designing various interaction components. The Wide & Deep model (Cheng et al.] (0016) directly learns parameters of manually designed cross features in the wide component. The Product-based Neural Network (PNN) (Qu et al., 2016) applies inner-product or outer-product to capture second-order features. The Deep & Cross Network (DCN) (Wang et al., 2017) designs a incremental cross structure to well as high-order features. Factorization Machine (FM) (Rendle 2010) 2012 Juan et al. 2016) is a suitable and success way to capture second-orde feature interactions, as well as high-order feature interactions (Blondel et al.) 2016). Thus, FM has been extended to deep architecture, e.g., DeepFM (Guo et al. 2017) and xDeepFM (Lian et al. 2018). As we know, these deep learning-based methods usually consists of DNN. And DNN able to capture variety of feature interactions in its hidden layers, while not able to generate interpretable cross features. Thus, only a few cross features can be generated by the interaction components in deep learning- based methods.
capture second-order
Another practical method for feature generation is to employ explicit search strategies to find useful features (Fan et al. 2010; Kanter& & Veeramachaneni 2015 Katz al. 2016 Luo al. 2019). As we can imagine, in search-based methods, the candidate set of cross feature fields is inevitable to be extremely large, thus the searching efficiency is usually low. To solve this problem, AutoCross (Luo) eet al. (219) presents framework to search in the large candidate set more efficiently, which is greedy and approximate alternative: (1) AutoCross iteratively searches in a set of cross feature fields, where the set is initialized all the second-orde feature fields, and the selected cross feature field is greedily used to generate new high-order cross feature fields in the next iteration; (2) AutoCross uniformly divides the dataset and validate the contribution of cross feature field on batch of data, which will face the problem of data lacking and random resulting when the candidate set is large. The authors apply the generated cross features, containing both second- and high-order cross features, in LR model, and it achieves approximate or even better performances comparing with the complex DNN model in 10 real-world datasets. However, AutoCross still searches for useful cross feature fields in large candidate set, whose size shows exponential relation with the number of the original feature fields. For example, when the size of original feature fields is 10, the number of second-order cross feature fields is 45. And when the size of original feature fields becomes 100, 200, 500 and 1000, the number of second-order cross feature fields becomes 4950, 19900, 124750 and 499500 respectively. When the candidate set is large, the searching efficiency will still be low, and data for verifying the contribution of a candidate cross feature field will be too little to produce reliable results. This strongly constrains the applicability of AutoCross in real-world applications.
2.2 LEARNING FROM THE INTERPRETATION OF DNN
To solve above problems of existing methods, we need to automatically generate a compact and accurate candidate set, so that the search procedure can be performed efficiently. The widely used DNN model has shown to be capable of capturing various feature interactions in its hidden layers (uoo et al.) 2017). And various works has done to conduct picce-wise interpretation of DNN, which means DNN model can be regarded as combination of an infinite number of linear classifiers (Bastani et al., 2017 Alvarez Melis & Jaakkola 2018 CCu al.) (018)). Via the gradient backprop- agation from the prediction layer to the feature layer, for a specific feature f with the feature field I and a specific sample in the dataset, we have
i Uk
w Wfa.k
(2)
3

---

Under review as a conference paper at ICLR 2020
Table The values of interpretation inconsistency features, where a E [0,1) and 8 E {0. 1), on four toy datasets, characterizing logical operations AND, OR, XNOR and XOR respectively. Large interpretation inconsistency values give hints about cross features.
sample
AND
OR
XNOR
XOR
Q
8
8
3
Q
3
a
B
0
0.0002
0.0002
0.0001
0.0000
0.0105
0.0152
0.0137
0.0T03
0 0.0006
0.0004
0.0003
0.0005
0.0000
0.0003
0.0411
0.0192
1
0.0000
0.0002
0,0002
0.0000
0.0126
0.0145
0.0189
0.0103
0.0000
0.0001
0.0002
0.0002
0.0308
0.0161
0.0190
0.0065
where 0x denotes the prediction made by DNN for the sample k. and wgih is the local weights computed via gradient backpropagation.
Definition 2.1 (Local Interpretation) Given a
f, and a specific sample k, the cor-
responding local interpretation
1z,ū, = ,f, j,.0
(3)
where f denotes the corresponding feature embedding in sparse DNN.
Usually, local interpretations a specific feature are inconsistent in different samples in the dataset.
Definition 2.2 (Global Interpretation) Given a specific feature Ty,s the corresponding global in- terpretation is
1 ej
(4)
where y is the averaged local weights of features associated with the feature field f in all samples, named as global weights.
Definition 2.3 (Interpretation Inconsistency) Given a specific feature f and specific sample k the corresponding interpretation inconsistency is
df.i,k = [1 w w e.sl-
(5)
When a specific feature has interacted with other features in the hidden layers of DNN, the corre- sponding local interpretations will be very inconsistent among different samples in a dataset, i.c., interpretation inconsistency is large. Thus, the interpretation inconsistency in DNN is able to lead us to generate compact and accurate candidate sets of cross feature fields. Accordingly, we can automatically learn feature crossing based on Assumption 2.1
Assumption 2.1 Once the interpretation inconsistency features in specific feature field exceeds a threshold, the corresponding feature field works for generating cross features.
To verify Assumption 2.1 we conduct empirical experiments on four toy datasets. The four datasets characterize four different logical operations: AND, OR, XNOR and XOR. And we have two input feature fields, where a E {0, }} and 3 E {0, 1}. Thus, for each toy dataset, we have four different samples, and the output lies in {0, 1}. As we know, the logical operations AND and OR easy and linearly separable, therefore no cross features are needed. In contrast, the logical operations linearly separable, therefore second-order cross feature field consisting of and 3 should be generated for promoting linear classifiers. We train two-layer DNN model on the four toy datasets until convergence, where the Area Under Curve (AUC) evaluation becomes 1.0. On all datasets, the gradients from the prediction layer to the feature layer is computed, and the interpretation inconsistency is obtained, as shown in Table [T] It is clear that, interpretation inconsistency values on AND and OR are extremely small, while those on XNOR and XOR are relatively large. This observation clearly shows that, a and / should XOR, while should not on AND and OR. This is clearly consistent with common sense, and strongly proves Assumption 2.1 According to the values Table [ 0.01 might be proper threshold for filtering feature fields to construct compact and accurate candidate sets of cross feature fields. So far, we can conclude that, it is proper to learn feature crossing from the interpretation inconsistency computed in DNN.
XNOR and XOR
crossed on XNOR and
4

---

Under review as a conference paper at ICLR 2020
Interpretability
Grodlenns
Condstincy
Anahyls
Local
Interpretability
Interprctaiion Inconsssency
Fozsible Feutures
DNN
Greedy Genenetoo
Hululd
Festures
Cross Festure Fiels
clate
Figure 1: Overview of our proposed CrossGO approach. 8, and fy indicate the i-th sample and the j-th original feature field respectively.
CROSSGO
In this section, we formally propose the CrossGO model. In general, CrossGO consists of two steps: (1) generating a compact and accurate candidate set of cross feature fields; (2) searching in the candidate set for the final cross feature fields. We f2: fa to denote the cross feature field generated by crossing fI. f2- ..., fn- Figure provides an overview of the proposed CrossGO approach, which will be described detail below.
3.1 CANDIDATE SET GENERATION
Firstly, as we rely on the piece-wise interpretation of DNN generate the compact and accurate candidate set of cross features, we need to train a DNN model. The input is the original categorical features and categorical features are field-wise high dimensional vectors. Thus, we use embedding layer (Zhang et al. 2016) transform the input features into low dimensional dense representations. Then the dense representations are passed through some linear transformation and nonlinear activa- tion obtain the predictions of samples, where sigmoid as the activation for the output layer to support binary classification tasks.
relu the activation for hidden layers, and
Based on the trained DNN model, according to Assumption 2.1 in every validation sample k, first compute the interpretation inconsistency dy,i.k of each feature Ty where df.i.a is defined in Definition 2.3 Thus, obtain matrix D, where D&f is the interpretation inconsistency of f-th feature in the k-th sample. Then we conduct element-wise filter on matrix D with a threshold which the threshold in Assumption 2.1 And r) is 0.01 for default. The filter operation can be formulated as
[1. Daf N D/y
0, otherwise"
where D' is a binary feasible feature matrix. According to Assumption 2.1 for each feasible feature D/2y = 1, the corresponding feature field ff can used to generate candidate cross feature fields.
Lastly, we perform greedy generation to obtain the candidate set of cross features fields. Pseudocode of greedy generation is given in Algorithm 3.1. Here, we introduce three new hyper-parameters 5, e and 7. is the maximum order of cross features, and is 4 for default, where higher-oder cross features rarely useful. denotes the threshold of the occurrence ratio of cross feature fields on validation set, and E is 0.01 for default. 7 is the number candidate cross feature fields returned by greedy generation, and r is 2N for default. To be noted, in general, 2N is extremely small comparing with the size of the whole set 2nd-order. 3th-order and 4th-order cross feature fields. For example, N is 100, the size of the whole set is 4087875 (4950 161700 + 3921225).
5

---

Under review as a conference paper at ICLR 2020
Algorithm Greedy Candidate Set Generation
Require: feasible feature matrix D', set original feature fields F = f2: JN maximum order â of cross features, threshold e of the occurrence ratio of cross feature fields on validation set, number r of candidate cross feature fields, the size of validation set is Malidd-
Ensure: compact and accurate candidate set
1: S=(}, 5'=(), set first-order features Si = {(fi , (f2h.... ifN H}, current order = 2,
count of occurrences counte 0 for every cross feature field
2: for < 8 do
S. P 0
4:
for sample & in validation set do
5:
set of feasible feature fields Qx = (' 1}; for ff in F. in S do
6:
7:
generate cross feature field p - (ʃ) U.: if C Qx then counlp count, + 1; + P.
8:
9:
10:
11:
end 12: end for
13:
end for
for p P do
15:
if couullp/ Muatid 2 e then
p D Soi end if end for
17:
19:
S' == ''=''USS.
0 0= 0=0+1;
21: end for
22: select top 7 cross feature fields with largest counts from s', and assign them to S:
23: return S.
3.2 SEARCHING FOR CROSS FEATURE FIELDS
After obtaining the candidate set of cross feature fields = fci, C2...., C2N }. where c; is a spe- cific cross feature field, we can search for the final useful cross feature fields according to their contribution measured in the validation set.
We first need to train a LR model. The input of LR model consists of both original features and candidate cross features. We use S the schema to process training data to generate training cross features, which are denoted as cross The shape of Xtrai is [Mtrain, 2N.. We also denote the original features of training set as original and the shape of it is [Mtroin: NJ. Thus, the number of model's input feature fields 3.V. Therefore, the time cost of training this LR model is in the same order of magnitude with directly training LR model with only original features.
Based on the trained LR model with model weights W. we conduct our searching process. Here, we use X ariggina and X denote the original features and the cross features in the validation set respectively. Pscudocode of searching process is given in Algorithm 3.2 It is important to note that the step 4-10 Algorithm 3.2 can be conveniently paralleled via multi-threading or multi- process implementation. Moreover, the inference process is much faster than the training process, and therefore the time cost of searching can be neglected.
This searching process is fast and effective. The reason why egy is, CrossGO can generate a compact and accurate candidate set of cross feature fields according to the interpretation inconsistency in DNN.
such easy searching strat-

---

Under review as a conference paper at ICLR 2020
Algorithm 2 Searching for Cross Feature Fields on Validation Set.
Require: candidate
{ci, ".,... C2N original features Xeeigimal with N feature fields and Meatid samples, candidate cross features X-aiid with 2N cross feature fields and Matid samples, labels Yvalid with Myalid samples, model weights W of LR trained with both orig- inal and cross features on training set, sigmoid function o() and AUC computing function compute. auc().
Ensure: final set of cross feature fields S*
5 = 0):
2: b(-1) voriginal [ N ---1, 1};
3: AUC(-1)
compute_auck valid, or (b(-1)))
4: for in [0, 2N - do
5: for in [0, 2N - 1]]d do
6:
if not in 5 then
7:
b(j) = b(-1)+ + Xeatdll [..] W[N + +J]; AUC()) = compute aY o (b(j)));
8:
9:
end if
10:
end for
11:
k argmax ArgmaxAUUC(]
12:
if 2 then
13:
Ck + S":
14:
b( x-1)==bAk)
15:
AUC(-1) AUC(K);
16:
else
17:
break;
18:
end if'
end for
20: return S*
Table 2: Statistics of the benchmark datasets.
#Samples
#Feature Fields
Datasets
Domain
Training
Testing
#Num.
#Cate.
Employee
29,494
3,277
0
9
Human Resource
Adult
32,562
16,282
8
Social
Allstate
131,823
56,497
115
Insurance
Prudential
41,567
17,816
22
104
Insurance
Movielens
588,79 799,
150,215
109
20
Entertainment
Criteo
41,256K
4.584K
13
26
Advertising
Anonl
233.123
58,282
178
15
Securities
Anon2
99,137
42,487
459
26
Banking
EXPERIMENTS
In this section, empirically test our proposed crossGO model on 8 datasets for thorough com- parisons. We first describe the datasets and settings of the experiments, then report and analyze the experimental results.
4.1 DATASETS
We evaluate the proposed crossGO model on six benchmark and two anonymous datasets. The two anonymous datasets are provided with sanitization and named as Anonl and Anon2. The statistic of these datasets are shown in Table 2 All the tasks are for binary classification. Employee, Adult and Criteo the same datasets in (Luo et al.) 2019). For other datasets except for the former three and Movielens, we use the first 70% data as the training data, while the rest 30% as testing. And we out the last 20% of the training data validation set.

---

Under review as a conference paper at ICLR 2020
Table Summary of results in terms of AUC (in percent) narrow datasets.
LR
CMI
FM
AutoCross
DNN
CrossGO
Employee 86.75
89.01
86.89
89.42
87.85
89,59
Adult
92.10
91.53
91.71
92.80
92.68
92.85
Criteo
78.55
78.52
79.27
80.34
79.85
80.41
Table 4: Summary of results in terms of AUC (in percent) on wide datasets.
LR
CMI
FM
AutoCross
DNN
CrossGO
Movielens
81.49
82.61
85.73
85.12
86.51
86.77
Prudential
84.56
84.78
84.88
84.82
84.95
84.97
Allstate
86.10
86.35
86.41
86.41
86.63
86.73
Anon
72.36
73.11
72.34
72.67
74.60
76.23
Anon
89.19
89.91
90.02
89.33
90.89
91.03
For Movielens, we regard reviews as behaviors, and sort the reviews from one user by time. For user u, we split u's behaviors into three parts: the first 50% is for training. following 20% for validation, while the rest is for testing. Meanwhile, we transform Movielens into a binary classi- fication data. Specifically, original user rating of the movies is continuous value ranging from to 5. We label the samples with rating of 4 and 5 to be positive and the samples with rating of and 2 to be negative. We extract a series statistical features based on each user's historical be- havior. Features include user_age, user gender, soccppccu,,, is_cate, recent_k. pecttt_k_cate.ae sitee_eennsseee and recent k_cate_cnt, where k E {10, 30, 50} and cate E {action, adventure, animation, children's ss comedy, crime, documentary, drama, fantasy, film-noir, horror, musical, mystery, romance, sci-fi, thriller, war, western]. The recent k cate ent is the count each category in movies that each user has reviewed in recent times. Accordingly, the recent_k _etnttt____ee____eeeeeeeeeeeeeee wate_aeeanna_aaaaeeaa mean__scoreeeeee is the average score of each category in movies that each user has reviewed in recent k times.
In summary, in terms of whether or the number of original feature fields is bigger than 100, test our approach on two types of datasets:
Narrow Datasets: Employee, Adult and Criteo;
Wide Datasets: Movielens, Allstate, Prudential, Anon and Anon2.
4.2 EXPERIMENTAL SET-UP
As our goal is to make a simple LR model empowered by generated cross features to achieve approx- imate or even better performances comparing with DNN, we compare against the strong basclines (LR, CMI+LR and DNN) and the state-of-the-ar feature crossing approach (AutoCross+LR) in (Luo et al. 2019). We simplify AutoCross+LR and CMI+LR to AutoCross and CMI. Accordingly, approach is named as CrossGO.
We use the same discretization method as in (Luo et al. 2019) for preprocessing data. For the DNN model, which we rely to compute the interpretation inconsistency of features: learning rate is 0.001, the dimensionality of embedding vector is 16, network structure 64-32. For the LR model, which we rely Adam with weight decay as zero as optimizer, the Area- Area-Under-Curve (AUC) as our experimental metric and the same early-stopping strategy. Specifically, if the validation AUC dose not increase in three epochs, the training process will be stopped.
search for eross features: learning ratee [0.0001, 0.001]. For all models,
4.3 PERFORMANCE ANALYSIS
Results of our experiments summarized in Table Table 4 Table 6 and Figure 2 According to Table 3 and Table all models except CMI significantly beat LR model on all datasets. CMI inferior to LR some datasets, which shows that adding cross features does not guarantee the improvement of performance. The set generated by CMI may contain many redundant and over-fitting cross features, which leads to poor performance some datasets.
8

---

Under review as a conference paper at ICLR 2020
Table 5: Comparison of cross feature generation time (in seconds) each dataset.
Employee
Adult
Criteo
Movielens
Prudential
Allstate
Anon
Anon2
AutoCross
182
128
11094
2829
452
1595
6136
8144
CrossGO
59
62
5922
367
86
196
161
288
Table 6: The number of second/high order cross feature fields generated on each dataset.
Empkoyee
Adult
Crieeo
Movielens
Prudential
Allstate
Anonl
Anon2
AutoCross
2nd-order
7
6
8
52
73
62
68
120
high-order
CrossGO
2nd-order
12
3
8
150
156
133
117
237
high-order
3
7
7
0
39
121
Obviously, proposed CrossGO model performs better than DNN all datasets. That is say, a simple LR model empowered by our generated cross features always achieves better performance comparing with DNN. Moreover, Figure 2 demonstrates that we only need small portion of our generated cross feature fields to make LR model achieving approximate DNN performance. And continuing adding our generated cross feature fields, the performances of LR with CrossGO increasing. Thus, user can decide the number of generated cross feature fields to add according to their expectation of testing performance. To be noted that as mentioned section 3 the number of generated cross feature fields has little impact on the time cost of CrossGO. In summary, these results strongly verifies the effectiveness, efficiency and flexibility of our approach.
Meanwhile, our proposed approach outperforms AutoCross on all datasets. On the narrow datasets, the results in Table demonstrate that AutoCross is inferior to our proposed CrossGO model. On the wide datasets, the candidate set of cross features is inevitable to be extremely large, and samples for evaluating each candidate cross feature field are too little. Thus, AutoCross performs poorly on wide datasets, and not achieve similar perfromances as DNN. Our porposed CrossGO model can significantly outperform AutoCross on wide datasets. Morcover, according to Table CrossGO is more efficient than AutoCross, especially on wide datasets. And wider the dataset, larger advantage.
4.4 INTERPRETABILITY OF CROSSGO
As CrossGO can explicitly generate cross features, CrossGO has good interpretability. In or- der to make the interpretability of CrossGO comprehensible, from dataset MovieLens in the process of testing. The five most important cross feature fields added to the final set are (is_horror, peccen_55_aaceee 50_action_entt) {is_action, dicent_50_crimee 3ee.0eemeeeeeeeeeeeeeeee _meee_ratee.. {is horror, recent 50_ 10-thriller maaanraaeeaaraaaaa fis-children's 's recent_30 30_fantasy_.oom cent 30 crime mean rate]. Obviously, these five cross feature fields are complete comprehensible to human. And from a logical point of view, they are highly likely to make data more linear-separable. For example, {is_horror,
choose representative
and fis_action, re-
ireett550aactionn _ntoon implies that for predicting current rating, it is good choice to consider both whether current film is horror and whether current user often watches action films. And {is_action, recent Drrent_50_crinnn 50_errme___..m.mm.eemeeeeeeeee both whether current film is action and whether current user likes crime films. In practice, adding these cross features significantly improves the performance of LR model. This demonstrates that proposed CrossGO model is interpretable and effective.
www.meeeeeeaammmmmemme mean meen implies that it is also a good choice to consider
CONCLUSION
In this paper, we define the interpretation inconsistency in DNN for the first time. According to the interpretation inconsistency, we propose novel CrossGO method. CrossGO can generate compact candidate set cross feature fields, with extremely small amount compared to the whole set of second-order and high-order cross feature fields in dataset. Based on the corresponding performances in the validation set, useful cross feature fields can be directly ranked and selected from our compact candidate set. The whole process of learning feature crossing simply training a DNN model and a LR model. Extensive experiments have been conducted on several real-world datasets. Cross features generated by CrossGO can empower simple LR model
be done via

---

Under review as a conference paper at ICLR 2020
89.5
92.0
92.7
92.6
88.5
92.5
92.4
92.3
87.5
CrossGO DNN
CrossGO
92.1
10
12
14
8
10
12
14
Cross Feature Fields
Cross Feature Fields
(a) Testing performances on Adult w.r.t. the number of (b) Testing performances on Employee w.r.t. the num- our generated cross features.
ber of our generated cross features.
Figure 2: Testing performances on Adult and Employee w.r.t the number of our generated cross features.
achieving approximate or even better performances comparing with complex DNN models, well as the state-of oftte-of-the-art the-art art feature crossing method, i.e., AutoCross, which greedily explores the whole of cross feature fields. Moreover, cross features generated by CrossGO have great interpretability.
REFERENCES
David Alvarez-Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 7786-7795, 2018.
Osbert Bastani, Carolyn Kim, and Hamsa Bastani. Interpreting blackbox models via model extrac- tion. arXiv preprint htt5:/705.08800 2017.
Mathieu Blondel, Akinori Fujino, Naonori Ueda, and Masakazu Ishihata. Higher-order factorization machines. In Advances in Neural Information Processing Systems, pp. 3351-3359, 2016.
Olivier Chapelle, Eren Manavoglu, and Romer Rosales. Simple and scalable response prediction for display advertising. ACM Transactions Intelligent Systems and Technology (TIST), 5(4): 61, 2015.
Chen Cheng, Fen Xia, Tong Zhang, Irwin King, and Michael R Lyu. Gradient boosting factorization machines. In Proceedings of the 8th ACM Conference on Recommender systems, pp. 265-272.
ACM, 2014.
Heng-Tze Cheng, Levent Koe, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recom- mender systems. In Proceedings of the 1st workshop on deep learning for recommender systems, pp. 7-10. ACM, 2016.
Lingyang Chu, Xia Hu, Juhua Hu, Lanjun Wang, and Jian Pei. Exact and consistent interpretation for piecewise linear neural networks: A closed form solution. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pp. 1244-1253.
ACM, 2018.
Pedro M Domingos. A few useful things to know about machine learning, Commun. acm, 55(10): 78- 2012.
Wei Fan, Erheng Zhong, Jing Peng, Olivier Verscheure. Kun Zhang, Jiangtao Ren, Rong Yan, and Qiang Yang. Generalized and heuristic-free feature construction for improved accuracy. In Pro- ceedings of the 2010 SIAM International Conference Data Mining, pp. 629 640. SIAM, 2010.
10

---

Under review as a conference paper at ICLR 2020
Huifeng Guo, Ruiming Tang. Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfin: factorization- machine based neural network for cur prediction. In Proceedings of the 26th International Joint Conference Artificial Intelligence, pp. 1725-1731, 2017.
Yuchin Juan, Yong Zhuang, Wei-Sheng Chin, and Chih-Jen Lin. Field-aware factorization machines for ctr prediction. In Proceedings of the 10th ACM Conference on Recommender Systems, pp- 43-50. ACM, 2016.
James Max Kanter and Kalyan Veeramachaneni. Deep feature synthesis: Towards automating data science endeavors. In 2015 IEEE International Conference Data Science and Advanced Ana- lytics (DSAA), pp. 1-10. IEEE, 2015.
Gilad Katz, Eui Chul Richard Shin, and Dawn Song. Explorekit: Automatic feature generation and selection. In 2016 IEEE 16th International Conference Data Mining (ICDM), pp. 979-984. IEEE, 2016.
Sotiris Kotsiantis and Dimitris Kanellopoulos. Discretization techniques: A recent survey. GESTS International Transactions on Computer Science and Engineering. 32(1):47-55) 2006.
Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. xdeepfm:: Combining explicit and implicit feature interactions for recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1754-1763. ACM, 2018.
Huan Liu and Hiroshi Motoda. Feauure extraction, construction and selection: A data mining per- spective, volume 453. Springer Science & Business Media, 1998.
Huan Liu, Farhad Hussain, Chew Lim Tan, and Manoranjan Dash. Discretization: An enabling technique. Data mining and knowledge discovery, 6(4):393-423) 2002.
Yuanfei Luo, Mengshuo Wang, Hao Zhou, Quanming Yao, WeiWei Tu, Yuqiang Chen, Qiang Yang. and Wenyuan Dai. Autocross: Automatic feature crossing for tabular data in real-world applica- 2019.
tions. arXiv preprint
Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, and Jun Wang. Product-based neural networks for user response prediction. 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 1154. IEEE, 2016.
Steffen Rendle. Factorization machines. In 2010 IEEE International Conference on Data Mining, pp. 995-1000, IEEE, 2010.
Steffen Rendle. Factorization machines with libfm. ACM Transactions on Intelligent Systems and Technology (TIST), 3(3):7,, 2012.
Rómer Rosales, Haibin Cheng, and Eren Manavoglu. Post-click conversion modeling and analysis for non-guaranteed delivery display advertising. In Proceedings of the fifth ACM international conference on Web search and data mining, pp. 293-302. ACM, 2012.
Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. Deep & cross network for ad click predictions. In Proceedings of the ADKDD' 17, pp. 12. ACM, 2017.
Quanming Yao, Mengshuo Wang. Jair Escalante Hugo, Guyon Isabelle, Yi-Qi Hu, Yu-Feng Li, Wei-Wei Tu, Qiang Yang. and Yang Yu. Taking human out of learning applications: A survey on automated machine learning. arXiv preprint arXiv: 13306, 2018.
Weinan Zhang, Tianming Du, and Jun Wang. Deep learning over multi-field categorical data. In European conference on information retrieval, pp. 45-57. Springer, 2016.
11


Reviews:
[{"review_id": "H1eu0MZ2tS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the paper, the authors proposed CrossGO, an algorithm for finding crossing features useful for prediction.\nIn CrossGO, one trains a neural network that captures feature crossing implicitly.\nThen, possible crossing features are estimated using the gradient-based saliency.\nThe idea here is that, if a feature has a crossing with some other features, its contribution in the saliency can vary across different inputs.\nThus, by looking at the variation of the saliency, one can find candidates features for feature crossing.\nCrossGO greedily selects candidate crossings based on the idea above.\nIn the last step, a simple logistic regression is trained using the candidate crossings, and the effective crossings are selected using a forward greedy feature selection.\n\nI found the paper well-written and the idea is easy to follow.\nMy concern, however, is the lack of Factorization Machines (FM) in the experiments.\nIn Introduction, the authors mention to the deep version of FM and stated \"(deep FMs are) not able to generate interpretable cross features\".\nBut, as the authors are aware of, non-deep FMs are able to handle feature crossings in a interpretable way.\nThus, it would be essential to adopt non-deep FMs as the baseline in the experiments.\nBecause the important baseline is missing, I found the results are not convincing enough to claim the effectiveness of the proposed method.\n\n\n### Updated after author response ###\nThe authors have partially addressed my concern by adding FM/HOFMs as the experiment baselines, which I greatly appreciate.\nHowever, I found the current paper misses some other possible baselines for high-order interaction models [Ref1,2].\nAs the authors mentioned in the response, FMs find the feature crossing as a kind of embedded representations, which may not be suitable for modeling sparse interactions.\nThus, the sparse interaction models need to be taken into consideration as well.\n\n[Ref1] Safe Feature Pruning for Sparse High-Order Interaction Models\n[Ref2] Selective Inference for Sparse High-Order Interaction Models", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByltxKWzYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper attempts to solve the cross feature generation problem efficiently. The state-of-the-art method AutoCross cannot control the size of the searching set of the candidates of cross feature fields. To narrow down this set, the authors provides a measurement called Interpretation Inconsistency. With an easy toy experiment, the authors conjecture that features with large Interpretation Inconsistency tend to interact with other features in the hidden layers of DNN. Therefore, based on this conjecture, the authors design an effective algorithm to discard those cross features with small Interpretation Inconsistency value, which finally narrows the set of cross features. With this narrowed set, the whole procedure can be accelerated largely. \n\nPros:\nThis work can be regarded as an accelerated version of AutoCross. By incorporating Interpretation Inconsistency, the set of cross features can be effectively narrowed. Although this is an incremental work, this idea is relatively novel. \n\nCons:\n1.\tThe setting of the threshold for filtering feature fields is somewhat heuristic. The authors should provide some explanations on its setting. If not, we cannot trust it and doubt that it may cause some unexpected results and thus will not be robust.\n2.\tThe experiments are somewhat not convincing. The main contribution is to accelerate AutoCross. Thus, I expect to see the time complexity comparison between them. However, I do not find it in this paper. Although the authors mention that on wide datasets, AutoCross simply cannot work, on narrow datasets, a time complexity comparison should be provided.\n3.\tIn Table 5, only the numbers of cross feature fields of the proposed method are provided. A comparison with baseline methods on the number would be better to show the advantage of the proposed method.\n\nMinor: In Section 2.1, the double quotation marks should be revised.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "ByejreosKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper presents a scheme to generate new features as cross-product\nof binary features to improve the performance of linear models while\nobtaining interpretable models. The candidate set of cross-features\ncan be exponential and is handled by the proposed scheme by utilizing\nthe gradient-based importances of the features in a (deep) neural\nnetwork. Features with large discrepancies in their local and global\ninterpretations are used as the seed set of candidate features for\ngenerating new cross-features, and the final step performs a feature\nselection to further reduce the final set of cross-features. The\nempirical evaluation demonstrates the utility of the proposed scheme\non 8 datasets. \n\nWhile the proposed scheme does present a way to improve the accuracy\nof interpretable models, I am currently recommending a reject for the\nfollowing reasons (given the higher standard recommended for papers\nover 8 pages):\n\n- While this paper does consider some baselines, it seems to be\n  missing some crucial baselines that address the same (or very\n  similar) problem. There are some papers [2,3] that learn boolean\n  conjunctions (that can be seen as cross-features) to generate\n  accurate interpretable models. Moreover, there are some search based\n  feature generation schemes [1,4] that significantly improve upon the\n  exhaustive feature generation scheme of Kanter & Veeramacheneni,\n  2015. This technique can easily be applicable in learning boolean\n  cross features with binary features. At the very least, it is important to \n  understand where this proposed scheme is positioned relative to the \n  aforementioned literature and why a comparison is not required.\n- It is very unintuitive (at least to me) to tie the candidate\n  generation scheme to a neural network especially given the\n  sensitivity of neural network training to different initializations\n  and other factors. For the same data and neural network, the local\n  vs. global discrepancies can change significantly, thereby changing\n  the candidate set of cross features. This can potentially make the\n  proposed feature generation scheme somewhat unstable, and the\n  interpretations from the subsequent models might not be as\n  interpretable as they seem. It would be good to understand what I am\n  missing here and why being tied to a neural network model is\n  essential and not an issue here. \n\n\nClarification:\n\n- Lines 8-10 in Algorithm 1 is not clearly explained.\n- The experiment to motivate Assumption 1 needs to be better\n  explained.\n\n\nMinor:\n\n- The notation in equation (1) needs to be clarified better.\n\n\n[1] Khurana, Udayan, et al. \"Cognito: Automated feature engineering\nfor supervised learning.\" 2016 IEEE 16th International Conference on\nData Mining Workshops (ICDMW). IEEE, 2016. \n[2] Dash, Sanjeeb, Oktay Gunluk, and Dennis Wei. \"Boolean decision\nrules via column generation.\" Advances in Neural Information\nProcessing Systems. 2018. \n[3] Wei, Dennis, et al. \"Generalized Linear Rule Models.\" Proceedings\nof the 36th International Conference on Machine Learning. 2019.\n[4] Khurana, Udayan, Horst Samulowitz, and Deepak Turaga. \"Feature\nengineering for predictive modeling using reinforcement learning.\"\nThirty-Second AAAI Conference on Artificial Intelligence. 2018. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a simple but effective method for feature crossing using interpretation inconsistency (as defined by the authors).

I think this is a good work and the authors as well as the reviewers participated well in the discussions. However, there is still disagreement about the positioning of the paper. In particular, all the reviewers  felt that additional baselines should be tried. While the authors have strongly rebutted the necessity of these baselines the reviewers are not convinced about it. Given the strong reservations of the all the 3 reviewers at this point I cannot recommend the acceptance of this paper. I strongly suggest that in subsequent submissions the authors should position their work better and perhaps compare with some of the related works recommended by the reviewers.
Decision: Reject