Automatically Learning Feature Crossing from Model Interpretation for Tabular Data

Automatically feature generation is a major topic of automated machine learning . Among various feature generation approaches , feature crossing , which takes cross-product of sparse features , is a promising way to effectively capture the interactions among categorical features in tabular data . Previous works on feature crossing try to search in the set of all the possible cross feature fields . This is obviously not efficient when the size of original feature fields is large . Meanwhile , some deep learning-based methods combines deep neural networks and various interaction components . However , due to the existing of Deep Neural Networks ( DNN ) , only a few cross features can be explicitly generated by the interaction components . Recently , piece-wise interpretation of DNN has been widely studied , and the piece-wise interpretations are usually inconsistent in different samples . Inspired by this , we give a definition of interpretation inconsistency in DNN , and propose a novel method called CrossGO , which selects useful cross features according to the interpretation inconsistency . The whole process of learning feature crossing can be done via simply training a DNN model and a logistic regression ( LR ) model . CrossGO can generate compact candidate set of cross feature fields , and promote the efficiency of searching . Extensive experiments have been conducted on several real-world datasets . Cross features generated by CrossGO can empower a simple LR model achieving approximate or even better performances comparing with complex DNN models . 1 INTRODUCTION . Recently , automated machine learning ( AutoML ) has been widely studied , and is proven a practical and promising machine learning technique ( Yao et al. , 2018 ) . AutoML aims to provide a easy way to apply machine learning technique , and automate the procedure of machine learning partly or thoroughly . This reduces labour on data preprocessing , feature engineering , model selection , hyperparameter tuning , model training and performance evaluation . Among different components of machine learning , the quality of features plays an extremely important role ( Liu & Motoda , 1998 ; Domingos , 2012 ) . Accordingly , to improve the performances of machine learning tasks , various automatic feature generation methods have been proposed ( Chapelle et al. , 2015 ; Katz et al. , 2016 ; Zhang et al. , 2016 ; Qu et al. , 2016 ; Juan et al. , 2016 ; Blondel et al. , 2016 ; Guo et al. , 2017 ; Lian et al. , 2018 ; Luo et al. , 2019 ) . Among these feature generation methods , feature crossing , which takes cross-product of sparse features , is a promising way to capture the interactions among categorical features and is widely used to enhance the performance of machine learning on tabular data in real-world applications ( Chapelle et al. , 2015 ; Cheng et al. , 2016 ; Luo et al. , 2019 ) . Previous works on feature crossing mostly try to search in the set of all the possible cross features ( Rosales et al. , 2012 ; Chapelle et al. , 2015 ; Katz et al. , 2016 ) . Based on these methods , AutoCross ( Luo et al. , 2019 ) employs engineering optimization tricks , and accelerates the searching process . However , search-based methods are not efficient when the size of original feature fields is large . Meanwhile , some deep learning-based methods combine deep neural networks and various interaction components ( Qu et al. , 2016 ; Cheng et al. , 2016 ; Wang et al. , 2017 ; Guo et al. , 2017 ; Lian et al. , 2018 ) . However , due to the existing of Deep Neural Networks ( DNN ) , which can capture variety of interactions between original features , only a few cross features can be explicitly generated by the interaction components . As mentioned above , DNN is a powerful method for capturing various feature interactions in its hidden layers ( Guo et al. , 2017 ) . This means , DNN can generate features implicitly , but can not provide interpretable cross features explicitly . Recently , the interpreration of deep models has drawn great attention in academia , and mostly focus on piece-wise interpretation , which means assigning a piece of local interpretation for each sample ( Bastani et al. , 2017 ; Alvarez-Melis & Jaakkola , 2018 ; Chu et al. , 2018 ) . This can be done via the gradient backpropagation from the prediction layer to the feature layer . Usually , local interpretations of a specific feature are inconsistent in different samples . And large interpretation inconsistency indicates that , the corresponding feature has interacted with other feature fields in the hidden layers of DNN . Therefore , the interpretations of DNN can help us find useful cross features . Accordingly , in this paper , we propose a novel method called CrossGO , which can directly and automatically learn useful cross feature fields from the interpretation of DNN . ( 1 ) First , we train a common sparse DNN with the whole training set . ( 2 ) Then , for each sample in the validation set , we calculate the gradients of the output predictions with respect to the input features , i.e. , local interpretations . We also calculate the global interpretation of a specific feature as the average value of all the corresponding local interpretations . For a specific feature in a specific sample , if the corresponding local interpretation is far from the corresponding global interpretation , i.e. , interpretation inconsistency is large , we regard this feature interacted with others by DNN in the sample . Thus , we obtain a set of feasible features that works for feature crossing in each sample . ( 3 ) Then , we employ a greedy algorithm to generate a global candidate set containing both second-order and high-order cross feature fields . ( 4 ) With the the candidate set , we can train a simple LR model , therefore rank and select feature fields according to their contribution measured in the validation set . So far , we can obtain the final set of cross feature fields . We have conducted experiments on several real-world datasets . The size of candidate set of cross feature fields is constrained to be 2N , where N is the number of original feature fields . To be noted , 2N is extremely small compared to the whole set of second-order and high-order cross feature fields , especially whenN is large . And a simple LR model empowered by the final set of cross feature fields achieves approximate or even better performances comparing with DNN on all datasets . This whole process of learning feature crossing can be done via simply training a DNN model and a LR model . This shows that , our proposed CrossGO method can automatically generate a compact candidate set of cross feature fields , and efficiently find enough useful cross features . 2 MOTIVATION . In this section , we first summarize the advantages of feature crossing and the drawbacks of conventional methods . Then we give a brief analysis about why and how we learn feature crossing from the interpretation of DNN . 2.1 FEATURE CROSSING . According to the definition in previous works ( Cheng et al. , 2016 ; Luo et al. , 2019 ) , we can conduct feature crossing as gxf1 , i , xf2 , j , ... , xfN , m = xf1 , i⊗xf2 , j ⊗ ... ⊗ xfN , m , ( 1 ) where ⊗ denotes cross-product , gxf1 , i , xf2 , j , ... , xfN , m is the corresponding generated cross feature , and xf , i is a binary feature associated with field f , e.g. , feature ” gender=male ” associated with field ” gender ” . Via feature crossing , the cross feature of a male English-speaking student can be denoted as ( ” gender=male ” and ” language=English ” and ” job=student ” ) . Moreover , considering feature discretization has been proven useful to improve capability of numerical features ( Liu et al. , 2002 ; Kotsiantis & Kanellopoulos , 2006 ; Chapelle et al. , 2015 ) , we can conduct feature discretization on numerical feature fields to generate associated binary features . Via learning feature crossing , several advantages can be achieved : ( 1 ) new features can be generated ; ( 2 ) cross features , instead of latent embeddings in deep neural networks , are highly interpretable ( Luo et al. , 2019 ) ; ( 3 ) simple linear machine learning methods can take use of cross features to achieve approximate or even better performances comparing with complex nonlinear methods ( Luo et al. , 2019 ) ; ( 4 ) it is flexible and suits for large-scale online tasks ( Cheng et al. , 2016 ) . First attempts on feature generation mainly focus on generating second-order features ( Rosales et al. , 2012 ; Cheng et al. , 2014 ; Chapelle et al. , 2015 ; Juan et al. , 2016 ; Katz et al. , 2016 ) . In ( Chapelle et al. , 2015 ) , the authors try to generate and select second-order cross features according to conditional mutual information . The main problem of conditional mutual information is that , once the mutual information of an original feature is high , the generated cross features containing it will also have high conditional mutual information . ExploreKit ( Katz et al. , 2016 ) presents a framework for generating candidate features and ranking them . The proposed ranking and selecting methods are mainly based on the corresponding performances in classifiers , which is not efficient when the feature set is large . Moreover , these conventional methods fail to capture powerful high-order features . Nowadays , deep learning-based prediction methods have shown their effectiveness . Among these methods , some try to generate and represent features implicitly or explicitly via designing various interaction components . The Wide & Deep model ( Cheng et al. , 2016 ) directly learns parameters of manually designed cross features in the wide component . The Product-based Neural Network ( PNN ) ( Qu et al. , 2016 ) applies inner-product or outer-product to capture second-order features . The Deep & Cross Network ( DCN ) ( Wang et al. , 2017 ) designs a incremental cross structure to capture second-order as well as high-order features . Factorization Machine ( FM ) ( Rendle , 2010 ; 2012 ; Juan et al. , 2016 ) is a suitable and success way to capture second-order feature interactions , as well as high-order feature interactions ( Blondel et al. , 2016 ) . Thus , FM has been extended to deep architecture , e.g. , DeepFM ( Guo et al. , 2017 ) and xDeepFM ( Lian et al. , 2018 ) . As we know , these deep learning-based methods usually consists of DNN . And DNN is able to capture variety of feature interactions in its hidden layers , while not able to generate interpretable cross features . Thus , only a few cross features can be generated by the interaction components in deep learning-based methods . Another practical method for feature generation is to employ explicit search strategies to find useful features ( Fan et al. , 2010 ; Kanter & Veeramachaneni , 2015 ; Katz et al. , 2016 ; Luo et al. , 2019 ) . As we can imagine , in search-based methods , the candidate set of cross feature fields is inevitable to be extremely large , thus the searching efficiency is usually low . To solve this problem , AutoCross ( Luo et al. , 2019 ) presents a framework to search in the large candidate set more efficiently , which is a greedy and approximate alternative : ( 1 ) AutoCross iteratively searches in a set of cross feature fields , where the set is initialized as all the second-order feature fields , and the selected cross feature field is greedily used to generate new high-order cross feature fields in the next iteration ; ( 2 ) AutoCross uniformly divides the dataset and validate the contribution of a cross feature field on a batch of data , which will face the problem of data lacking and random resulting when the candidate set is large . The authors apply the generated cross features , containing both second- and high-order cross features , in a LR model , and it achieves approximate or even better performances comparing with the complex DNN model in 10 real-world datasets . However , AutoCross still searches for useful cross feature fields in a large candidate set , whose size shows exponential relation with the number of the original feature fields . For example , when the size of original feature fields is 10 , the number of second-order cross feature fields is 45 . And when the size of original feature fields becomes 100 , 200 , 500 and 1000 , the number of second-order cross feature fields becomes 4950 , 19900 , 124750 and 499500 respectively . When the candidate set is large , the searching efficiency will still be low , and data for verifying the contribution of a candidate cross feature field will be too little to produce reliable results . This strongly constrains the applicability of AutoCross in real-world applications .

Summary: This paper attempts to solve the cross feature generation problem efficiently. The state-of-the-art method AutoCross cannot control the size of the searching set of the candidates of cross feature fields. To narrow down this set, the authors provides a measurement called Interpretation Inconsistency. With an easy toy experiment, the authors conjecture that features with large Interpretation Inconsistency tend to interact with other features in the hidden layers of DNN. Therefore, based on this conjecture, the authors design an effective algorithm to discard those cross features with small Interpretation Inconsistency value, which finally narrows the set of cross features. With this narrowed set, the whole procedure can be accelerated largely.