SOInter: A Novel Deep Energy-Based Interpretation Method for Explaining Structured Output Models

We propose a novel interpretation technique to explain the behavior of structured output models , which learn mappings between an input vector to a set of output variables simultaneously . Because of the complex relationship between the computational path of output variables in structured models , a feature can affect the value of output through other ones . We focus on one of the outputs as the target and try to find the most important features utilized by the structured model to decide on the target in each locality of the input space . In this paper , we assume an arbitrary structured output model is available as a black box and argue how considering the correlations between output variables can improve the explanation performance . The goal is to train a function as an interpreter for the target output variable over the input space . We introduce an energy-based training process for the interpreter function , which effectively considers the structural information incorporated into the model to be explained . The effectiveness of the proposed method is confirmed using a variety of simulated and real data sets . 1 INTRODUCTION . The impressive prediction performance of novel machine learning methods has motivated researchers of different fields to apply these models in challenging problems . However , their complex and non-linear inherence limit the ability to explain what they have learned . Interpretation gets more attention when we want to discover the reasons behind the model ’ s decision and be sure about the trustworthiness and fairness of a trained machine learning model in areas such as medicine , finance , and judgment . Additionally , interpreting a model with a satisfying prediction accuracy in a scientific problem , which results in understanding relationships behind the data , leads to new knowledge about the problem domain.Murdoch et al . ( 2019 ) In many real-world applications , the goal is to map an input variable to a high-dimensional structured output , e.g. , image segmentation and sequence labeling . In such problems , the output space includes a set of statistically related random variables . As considering these dependencies can increase the prediction accuracy , many structured output models have been introduced . Many of these methods use graphical models , including random fields , to capture the structural relations between variables . Most define an energy function over these random fields , with a global minimum at the ground truth . Therefore , an inference is needed to find the best configuration of output variables for input by minimizing the energy function in the prediction step . Early efforts to utilize deep neural networks in structured output problems adopt deep networks to extract high-level features from the input vector to incorporate them in calculating the energy function Peng et al . ( 2009 ) ; Chen et al . ( 2015 ) ; Schwing & Urtasun ( 2015 ) . The computational complexity of the inference step in models that use random fields limits their ability to incorporate complex structures and interactions between output variables . Recent works in Belanger & McCallum ( 2016 ) ; Gygli et al . ( 2017 ) ; Belanger et al . ( 2017 ) ; Graber et al . ( 2018 ) propose to adopt deep networks instead of random fields to model the structure of the output space . Nevertheless , complex interactions between problem variables in such models make their interpretation too challenging , specifically when we focus on the model behavior in predicting a single output variable . This paper attempts to interpret a structured output model by focusing on each output variable separately . Our approach to model interpretation is based on instance-wise feature selection . Its goal is to find the relative importance of each input feature in predicting a single output variable . The subset of important features can vary across the input space . The complicated interactions between computational paths of output variables in structured output models cause critical challenges for finding a subset of important features associated with each output variable . A feature may not be used directly in the computational path of output but affects its value through relations with other outputs . To compute the importance of a feature for a target output variable , we should aggregate its effect on all output variables that are correlated to this target . Existing approaches of model interpretation can be divided into two groups , model-based and post hoc analysis Murdoch et al . ( 2019 ) . The model-based interpretation approach encourages machine learning methods that readily provide insight into what the model learned . However , it leads to simple models that are not sufficiently effective for complex structured output problems . Here we follow the post hoc analysis and try to explain the behavior of a trained , structured output model provided as a black box . Many interpretation techniques to find the importance of features as a post hoc analysis have been introduced . Works in Zhou & Troyanskaya ( 2015 ) ; Zeiler & Fergus ( 2013 ) ; Zintgraf et al . ( 2017 ) make perturbations to some features and observe their impact on the final prediction . These techniques are computationally inefficient in situations where we search for the most valuable features . Since we should perform a forward propagation for all possible perturbations , in another trend , works in Simonyan et al . ( 2013 ) ; Bach et al . ( 2015 ) back-propagate an importance signal from the target output through the network to calculate the critical signal of features by calculating the gradient of the target w.r.t the input features . These models are computationally more efficient than perturbation-based techniques because they need only one pass of propagating . However , they need the structure of the network to be known . As this approach may cause a saturation problem , DeepLIFT Shrikumar et al . ( 2017 ) proposes that instead of propagating a gradient signal , the difference of the output from a reference value in terms of the difference of features from a reference value to be considered . In addition to these approaches , other ideas have also been introduced in model interpretation . Authors in Ribeiro et al . ( 2016 ) introduce LIME which trains a local interpretable surrogate model to simulate the behavior of a black box model in the vicinity of a sample . It randomly selects a set of instances of the input space around that sample and obtains the black box prediction for them , and trains the surrogate model by this new dataset . Therefore this interpretable model is a good approximation of the black box around the locality of the selected sample . Shapley value , a concept from the game theory , explains how to distribute an obtained payout between coalition players fairly . The work in Lundberg & Lee ( 2017 ) proposes the kernel SHAP for approximating the shapely value for each feature as its importance for a prediction . As an information-theoretic perspective on interpretation , the work in Chen et al . ( 2018 ) proposes to find a subset of features with the highest mutual information with the output . This subset is expected to involve the most important features for the output . Existing interpretation techniques can be applied to explain the behavior of a structured model , w.r.t . a single output , by ignoring other output variables . However , none of these approaches consider possible correlations between output variables and only analyze the marginal behavior of the black box on the target . In this paper , we attempt to incorporate the structural information between output variables during training the interpreter . As our goal is to present a local interpreter , which is trained globally as Chen et al . ( 2018 ) , we train a function over the input space which returns the index of most important features for decision making about the target output . Since the value of other output variables affects the value of the target , incorporating them into the training procedure of an interpreter function may lead to higher performance and decrease our uncertainty about the black box behavior . To the best of our knowledge , this is the first time an interpreter is designed mainly for structured output models , and dependencies between output variables are considered during training the interpreter . 2 PRELIMINARIES AND MOTIVATION . Structured output prediction models map an arbitrary n-dimensional feature vector x ∈ X to the output y ∈ Y where y = [ y1 , y2 , . . . , yd ] includes a set of correlated variables with known and unknown complex relationships and Y shows a set of valid configurations . Now we explain our intuition about an interpreter , which explains the behavior of a structured output model in predicting a single output variable . We assume a structured model is available as a black box about which we do not know . Our goal is to find indices of k important features of x which affect the black box prediction about the target output yt . As for different localities of the input space these indices may vary , the proposed interpreter is a function IN t ( x ; α ) : X → { 0 , 1 } n over the input space with a set of parameters α which returns an n-dimensional k-hot vector . In this vector , the value of 1 shows the indices of selected k important features for target output yt . We define Θsb as the set of all parameters and hidden variables inside the structured black box . The probabilistic graphical model of Fig . 1 describes dependencies between problem variables . In this figure , x shows the input variable , ysbt and y sb −t = { ysbi |i 6= t } show black box predictions and αIN t is the set of parameters of IN t. The bidirectional edge between ysbt and ysb−t emphasizes the correlation between the outputs of a structured model . In fact αIN t is determined based on Θsb and the black box architecture , and the final prediction of the ysbt does not directly affect its value . However , here , Θsb is a latent variable which makes active paths between αIN t and output values ysbt and y sb −t . Therefore αIN t and y sb −t are dependent random variables and we have : H ( αIN t |x , ysbt ) > H ( αIN t |x , ysbt , ysb−t ) ( 1 ) where H ( .| . ) shows the conditional entropy . We use the strict inequality because αIN t and ysb−t are dependent random variables . The left term measures our uncertainty when we train the interpreter only by observing the target output ysbt . This inequality confirms that the uncertainty is decreased when we consider observed ysb−t during estimating αIN t . Motivated by this fact we propose a training procedure for an interpreter IN t which incorporates the structural information of the output space by observing the black box prediction on all output variables . We call our method SOInter as we propose it to train an Interpreter specifically for Structured Output models . 3 PROPOSED METHOD . We consider psb ( y|x ) as the distribution by which the structured black box predicts the output as follows , ysb = arg max y psb ( y|x ) . ( 2 ) Our goal is to train the interpreter IN t ( x ; α ) which explores a subset of most important features that affects the value of black box prediction on the target output yt in each locality of the input space . As the desired interpreter detects the subset of most important features , we expect perturbating other ones does not change the black box prediction of the target yt . Motivated by this statement , we are encouraged to compare the black box prediction for the target output when a sample and its perturbated version are passed through the black box . The interpreter IN t ( x ; α ) returns a k-hot vector in which the value of 1 shows the index of a selected feature . We define x̃ , a perturbated version of x , as follows , x̃ = x IN t ( x ; α ) ( 3 ) in which only features selected by the interpreter are included , and other ones are filled with zeros . Therefore by passing x and x̃ through the black box , we expect the value of the tth element of predictions to be the same , and we can define a penalty over the value of the target in these two situations . However , since the structure of the black box is unknown and the arg max operation in equation 2 is non-differentiable , a loss function that directly compares these two output values can not be used to find the optimal interpreter . Therefore , in the following subsection , we try to achieve a penalty according to the difference between these values for the target , which can transfer the gradient through an interpreter block .

Summary: The paper works on a novel problem of interpreting and explaining structured output models. The paper utilizes an energy based model to account for correlations between structured outputs and learns an interpretability block which given as input an image learns to mask it such that the energy based model would assign a similar score to the ground truth output and input as well as the ground truth output and perturbed input. In essence, the energy based model is a proxy for the actual deep neural network performing the structured prediction task. Results on a couple of datasets demonstrate that the work does better than baselines like LIME which do not utilize the correlations in the outputs modeled by the energy based model.