Feature Interaction Interpretability: A Case for Explaining Ad-Recommendation Systems via Neural Interaction Detection

1 INTRODUCTION . Despite their impact on users , state-of-the-art recommender systems are becoming increasingly inscrutable . For example , the models that predict if a user will click on an online advertisement are often based on function approximators that contain complex components in order to achieve optimal recommendation accuracy . The complex components come in the form of modules for better learning relationships among features , such as interactions between user and ad features ( Cheng et al. , 2016 ; Guo et al. , 2017 ; Wang et al. , 2017 ; Lian et al. , 2018 ; Song et al. , 2018 ) . Although efforts have been made to understand the feature relationships , there is still no method that can interpret the feature interactions learned by a generic recommender system , nor is there a strong commercial incentive to do so . In this work , we identify and leverage feature interactions that represent how a recommender system generally behaves . We propose a novel approach , Global Interaction Detection and Encoding for Recommendation ( GLIDER ) , which detects feature interactions that span globally across multiple data-instances from a source recommender model , then explicitly encodes the interactions in a target recommender model , both of which can be black-boxes . GLIDER achieves this by first utilizing our ongoing work on Neural Interaction Detection ( NID ) ( Tsang et al. , 2017 ) with a data-instance perturbation method called LIME ( Ribeiro et al. , 2016 ) over a batch of data samples . GLIDER then explicitly encodes the collected global interactions into a target model via sparse feature crossing . In our experiments on ad-click recommendation , we found that the interpretations generated by GLIDER are illuminating , and the detected global interactions can significantly improve the target model ’ s prediction performance . Because our interaction interpretation method is very general , we also show that the interpretations are informative in other domains : text , image , graph , and dna modeling . Code is available at : https : //github.com/mtsang/interaction_interpretability instance , denoted by the large green plus sign . 2 GLIDER identifies interactions that consistently appear over multiple data samples , then explicitly encodes these interactions in a target black-box recommender model frec . Our contributions are as follows : 1 . We propose feature interaction interpretations of general prediction models via interaction detection . 2 . Based on this approach , we propose GLIDER to detect and explicitly encode global feature interactions in black-box recommender systems . This process is a form of automatic feature engineering . 3 . Through experiments , we demonstrate the overall interpretability of detected feature interactions on a variety of domains and show that the interactions can be leveraged to improve recommendation accuracy . 2 NOTATIONS AND BACKGROUND . Notations : Vectors are represented by boldface lowercase letters , such as x or z . The i-th entry of a vector x is denoted by xi . For a set S , its cardinality is denoted by |S| . Let d be the number of features in a dataset . An interaction , I , is a subset of feature indices : I ⊆ { 1 , 2 , . . . , d } , where |I| is always ≥ 2 . A higher-order interaction always has |I| ≥ 3 . For a vector x ∈ Rd , let xI ∈ R|I| be restricted to the dimensions of x specified by I . Let a black-box model be f ( · ) : Rp → R. A black-box recommender model uses tabular feature types , as discussed later in this section . In classification tasks , we assume f is a class logit . p and d may be different depending on feature transformations . Feature Interactions : By definition , a model f learns a statistical ( non-additive ) feature interaction I if and only if f can not be decomposed into a sum of |I| arbitrary subfunctions fi , each excluding a corresponding interaction variable ( Friedman et al. , 2008 ; Sorokina et al. , 2008 ; Tsang et al. , 2017 ) , i.e. , f ( x ) 6= ∑ i∈I fi ( x { 1,2 , ... , d } \i ) . For example , a multiplication between two features , x1 and x2 , is a feature interaction because it can not be represented as an addition of univariate functions , i.e. , x1x2 6= f1 ( x2 ) + f2 ( x1 ) . Recommendation Systems : A recommender system , frec ( · ) , is a model of two feature types : dense numerical features and sparse categorical features . Since the one-hot encoding of categorical feature xc can be high-dimensional , it is commonly represented in a low-dimensional embedding ec = one hot ( xc ) Vc via embedding matrix Vc . 3 FEATURE INTERACTIONS IN BLACK-BOX MODELS . We start by explaining how to obtain a data-instance level ( local ) interpretation of feature interactions by utilizing interaction detection on feature perturbations . 3.1 FEATURE PERTURBATION AND INFERENCE . Given a data instance x ∈ Rp , LIME proposed to perturb the data instance by sampling a separate binary representation x̃ ∈ { 0 , 1 } d of the same data instance . Let ξ : { 0 , 1 } d → Rp be the map from the binary representation to the perturbed data instance . Starting from a binary vector of all ones that map to the original features values in the data instance , LIME uniformly samples the number of random features to switch to 0 or the “ off ” state . In the data instance , “ off ” could correspond to a 0 embedding vector for categorical features or mean value over a batch for numerical features . It is possible for d < p by grouping features in the data instance to correspond to single binary features in x̃ . An important step is getting black-box predictions of the perturbed data instances to create a dataset with binary inputs and prediction targets : D = { ( x̃i , yi ) | yi = f ( ξ ( x̃i ) ) , x̃i ∈ { 0 , 1 } d } . Though we use LIME ’ s approach , the next section is agnostic to the instance perturbation method . 3.2 FEATURE INTERACTION DETECTION . Feature interaction detection is concerned with identifying feature interactions in a dataset ( Bien et al. , 2013 ; Purushotham et al. , 2014 ; Lou et al. , 2013 ; Friedman et al. , 2008 ) . Typically , proper interaction detection requires a pre-processing step to remove correlated features that adversely affect detection performance ( Sorokina et al. , 2008 ) . As long as features in dataset D are generated in an uncorrelated fashion , e.g. , through random sampling , we can directly use D to detect feature interactions from black-box model f at data instance x . 3.2.1 NEURAL INTERACTION DETECTION . f can be an arbitrary function and can generate highly nonlinear targets in D , so we focus on detecting interactions that could have generic forms . In light of this , we leverage our method , Neural Interaction Detection ( NID ) ( Tsang et al. , 2017 ) , which accurately and efficiently detects generic non-additive and arbitrary-order statistical feature interactions . NID detects these interactions by training a lasso-regularized multilayer perceptron ( MLP ) on a dataset , then identifying the features that have high-magnitude weights to common hidden units . NID is efficient by greedily testing the top-interaction candidates of every order at each of h first-layer hidden units , enabling arbitraryorder interaction detection in O ( hd ) tests within one MLP . 3.2.2 GRADIENT-BASED NEURAL INTERACTION DETECTION . Besides the non-additive definition of statistical interaction , a gradient definition also exists based on mixed partial derivatives ( Friedman et al. , 2008 ) , i.e. , a function F ( · ) exhibits statistical interaction I among features zi indexed by i1 , i2 , . . . , i|I| ∈ I if Ez [ ∂|I|F ( z ) ∂zi1∂zi2 . . . ∂zi|I| ] 2 > 0 . The advantage of this definition is that it allows exact interaction detection from model gradients ( Ai & Norton , 2003 ) ; however , this definition contains a computationally expensive expectation , and typical neural networks with ReLU activation functions do not permit mixed partial derivatives . For the task of local interpretation , we only examine a single data instance x , which avoids the expectation . We turn F into an MLP g ( · ) with smooth , infinitely-differentiable activation functions such as softplus , which closely follows ReLU ( Glorot et al. , 2011 ) . We then train the MLP with the same purpose as §3.2.1 to faithfully capture interactions in perturbation dataset D. Given these conditions , we define an alternate gradient-based neural interaction detector ( GradientNID ) as : ω ( I ) = ( ∂|I|g ( x̃ ) ∂x̃i1∂x̃i2 . . . ∂x̃i|I| ) 2 , where ω is the strength of the interaction I , x̃ is the representation of x , and the MLP g is trained on D. While GradientNID exactly detects interactions from the explainer MLP , it needs to compute interaction strengths ω for feature combinations that grow exponentially in number as |I| increases . We recommend restricting GradientNID to low-order interactions . Algorithm 1 Global Interaction Detection in GLIDER Input : dataset B , recommender model frec Output : G = { ( Ii , ci ) } : global interactions Ii and their counts ci over the dataset 1 : G ← initialize occurrence dictionary for global interactions 2 : for each data sample x within dataset B do 3 : S ← MADEX ( frec , x ) 4 : G ← increment the occurrence count of Ij ∈ S , ∀j = 1 , 2 , . . . , |S| 5 : sort G by most frequently occurring interactions 6 : [ optional ] prune subset interactions in G within a target number of interactions K 3.3 SCOPE . Based on §3.1 and §3.2 , we define a function , MADEX ( f , x ) , that takes as inputs black-box f and data instance x , and outputs S = { Ii } ki=1 , a set of top-k detected feature interactions . MADEX stands for “ Model-Agnostic Dependency Explainer ” . In some cases , it is necessary to identify a k threshold . Because of the importance of speed for local interpretations , we simply use a linear regression with additional multiplicative terms to approximate the gains given by interactions in S , where k starts at 0 and is incremented until the linear model ’ s predictions stop improving . 4 GLIDER : GLOBAL INTERACTION DETECTION AND ENCODING FOR RECOMMENDATION We now discuss the different components of GLIDER : detecting global interactions in §4.1 , then encoding these interactions in recommender systems in §4.2 . Recommender systems are interesting because they have pervasive application in real-world systems , and their features are often very sparse . By sparse features , we mean features with many categories , e.g. , millions of user IDs . The sparsity makes interaction detection challenging especially when applied directly on raw data because the one-hot encoding of sparse features creates an extremely large space of potential feature combinations ( Fan et al. , 2015 ) . 4.1 GLOBAL INTERACTION DETECTION . In this section , we explain the first step of GLIDER . As defined in §3.3 , MADEX takes as input a blackbox model f and data instance x . In the context of this section , MADEX inputs a source recommender system frec and data instance x = [ x1 , x2 , . . . , xp ] . xi is the i-th feature field and is either a dense or sparse feature . p is both the total number of feature fields and the number of perturbation variables ( p = d ) . We define global interaction detection as repeatedly running MADEX over a batch of data instances , then counting the occurrences of the same detected interactions , shown in Algorithm 1 . The occurrence counts are not only a useful way to rank global interaction detections , but also a sanity check to rule out the chance that the detected feature combinations are random selections . One potential concern with Alg . 1 is that it could be slow depending on the speed of MADEX . In our experiments , the entire process took less than one hour when run in parallel over a batch of 1000 samples with ∼ 40 features on a 32-CPU server with 2 GPUs . This algorithm only needs to be run once to obtain the summary of global interactions .

Summary: This paper is focused on identifying/discovering feature interactions in blackbox models (with a focus on recommendations). Specifically the technique works by first corrupting datapoints and then using this "local dataset" to find interacting features via lasso-regularized multi-level perceptron approach (NID from Tsang et al). Using an expanded feature space and repeated calls to the above steps this is then used to find the most commonly occurring patterns.