SOInter: A Novel Deep Energy-Based Interpretation Method for Explaining Structured Output Models

Abstract: We propose a novel interpretation technique to explain the behavior of structured output models, which learn mappings between an input vector to a set of output variables simultaneously. Because of the complex relationship between the computational path of output variables in structured models, a feature can affect the value of output through other ones. We focus on one of the outputs as the target and try to find the most important features utilized by the structured model to decide on the target in each locality of the input space. In this paper, we assume an arbitrary structured output model is available as a black-box and argue how considering the correlations between output variables can improve the explanation performance. The goal is to train a function as an interpreter for the target output variable over the input space. We introduce an energy-based training process for the interpreter function, which effectively considers the structural information incorporated into the model to be explained. The effectiveness of the proposed method is confirmed using a variety of simulated and real data sets.

Reviews and author responses:

The paper works on a novel problem of interpreting and explaining structured output models. The paper utilizes an energy based model to account for correlations between structured outputs and learns an interpretability block which given as input an image learns to mask it such that the energy based model would assign a similar score to the ground truth output and input as well as the ground truth output and perturbed input. In essence, the energy based model is a proxy for the actual deep neural network performing the structured prediction task. Results on a couple of datasets demonstrate that the work does better than baselines like LIME which do not utilize the correlations in the outputs modeled by the energy based model. 
 Positives
. + The proposed problem seems novel and definitely interesting to study
. + The general line of thinking with regard to the work and its motivation makes sense to me

Negatives

There are a number of items of improvement possible for this work. 

Writing: 

.1. The draft of the paper especially Sec. 3 is very hard to read and understand. Example, in Eqn. 5 left hand side says q( y | x) but right hand side only has \tilde{x}, Eqn. 8 is missing \tilde{x} on the LHS and says x instead (where ideally it would be good to include / clarify that \tilde{x} is a function of x and that is what is meant). Also the flow of Sec. 3 would be much more clear if one started with Eqn. 8 and 9 and then broke it down into what each of the terms does and then explained how to compute all of the terms and their intutions. I found the Sec. very hard to follow as it was currently written. 

.2. The paper is also currently lacking any text or motivation that suggests why interpreting / understanding one of the outputs in a structured output model is an interesting / useful problem to study, with examples. Perhaps a problem like human pose estimation on images is one where one wants to know why a key point is in a particular place and does not care about all the other joints or key points as long as the model gets them correctly. Without such scaffolding it is very hard to understand why the paper is important in terms of the problem it solves. 

Technical Correctness
.3. Eqn. 12 is claimed to be able to yield a “k” hot vector or something close to it, but I don’t see that to be the case. The simple way / use case where this would fail is if two of the c^i’s say for c^1 and c^2 sample something really high for the 0th index. In that case we would have a k-1 hot vector (assuming no further collisions). 

.4. Eqn. (8) is not making sense to me, in the sense that we are asking for a perturbation/ masking of the image which brings down the energy of the correct output y_t combined with all the wrong outputs for the other structured outputs (while accounting for a slack that says how far y_t predicted is from correct output y_t). There is no reason why such a perturbation should have a low energy and I fail to see how masking the image based on this criterion would lead to anything useful. 

Experiments
.5. One major issue is that experiments are done on very synthetic datasets currently such as a dataset with binary masks of horses, a synthetic energy based model and a dataset called bibtex which I have never heard of (which would have been fine, except these also do not appear to be cited). 

.6. The current results in Fig. 5 do not appear very interpretable to me and it is unclear what is the expected output. It would be useful to pick a task such as facial keypoint estimation or pose detection where it is more clear what constitutes an “undertstandable” or reasonable explanation (as a control). 

.7. The use of other datasets which are more complicated is also important since the paper in its current form does not explain the original deep learning black box model but a surrogate energy based model estimated based on its input output responses. A big part of understanding and stress testing this approach thus is in terms of how much this energy based approximation loses in terms of explaining the original black box structured prediction model. This is a crucial comparison for completeness of this paper which is missing. 
 The paper needs improvement in writing, experimentation and formulation. 

The paper proposes a technique for identifying what input variables are most relevant for determining the value of a single, given output variable in structured-output (MAP) inference.  The idea is to learn an energy model that predicts which input variables are relevant to a particular structured prediction (x, y).  The authors propose to implement the energy model using a neural network followed by a Gumbel-softmax activation, and to train it by maximizing a structured hinge loss.  The proposed approach is evaluated on three datasets and compared to standard attribution techniques (LIME, SHAP, L2X). The contribution itself tackles an interesting problem -- explaining structured-output models -- and it does so by extending ideas from perturbation-based attribution techniques to this setting.

However, I found the text confusing and hard to follow;  the related work to be incomplete;  and the empirical evaluation to be limited and not entirely convincing.


Issues
------

- The related work neglects all the literature on explaining structured models like Bayesian networks and other graphical models, for instance:

  Andy Shih, Arthur Choi, and Adnan Darwiche. "A Symbolic Approach to Explaining Bayesian Network Classifiers."

  Eric Wang, Pasha Khosravi, and Guy Van den Broeck. "Probabilistic Sufficient Explanations." arXiv preprint arXiv:2105.10118 (2021).

and related works.  Given that graphical models + MAP inference can be viewed as structured prediction, I don't think that it is fair to claim that this is the "first time that an interpreter is designed mainly for structured output models".  I suggest the authors to include these works in their overview of the literature and to clearly position their work against these works.

Furthermore, the authors should compare empirically against these techniques if close enough to their intended applications.

- Unfortunately, the text is neither very clear nor easy to follow.  There are some issues with grammar and sentence construction, see below, but this is not the main issue.  The text generally feels hastily written and not well organized.  For instance, even though all the experiments rely on structured prediction energy networks (SPENs), they are introduced twice in the first two experiments.  The acronym "SPEN" is only introduced in the second experiment.

- I did not find the motivation given in Section 2 to be particularly useful.  Eq. 1 seems to restate that, in structured-output prediction problems, the output variables depend on each other.  This property is obvious and actually *defines* structured-output tasks from non-structured ones.  I think that most of Section 2, including Figure 1, could be removed from the paper.

- The description of the method is unnecessarily complex.  There is no need to start from Eq. 5.  The distribution q_alpha introduced here is immediately deleted in Eq. 8, presumably because dealing with the partition function makes the learning problem intractable.  However, there is little use for defining a distribution q_alpha in the first place.  Indeed, q_alpha occurs nowhere else in the text, and does not provide additional insights.  It would be more clear to introduce the proposed method starting from the energy model and Eq. 8.

- Eq. 6 is not new.  It is the so-called structured hinge loss, and it has been used extensively in the context of structured-output support vector machines and related techniques.  A reference to this literature should be added.  It is true that this loss is used here in a different manner.

- The loss should is not described/illustrated sufficiently well;  adding an illustrated example would help enormously.

- Neither of the two optimization problems in Eq. 9 can be solved trivially, but no information is provided in the main text about how they are solved in practice.  The appendix clarifies that they are solved using SGD-like techniques.  This means that they are not solved to global optimality.  The fact that the examples are sub-optimal, however, is likely to affect the quality of the learned energy model.  This is what happens in regular structured-output prediction, for instance.  Regardless, I suggest the authors to discuss in the main text how Eq. 9 is solved and why it is okay to solve it approximately.

- For this reason, I don't think that \hat{y} optimizes q_\alpha, as written in p 4.

- I could not find a discussion of the computational cost of the proposed method.  My understanding is that it is *very* computationally intensive.  In order to fit the model, one has to sample a large set of perturbations and compute the MAP assignment *for each of them*.  This can be computationally prohibitive.  I urge the authors to discuss the computational requirements of their approach, whether it is possible to amortize them over different inputs x, and to report the run-times of their approach in the experiments (including the time required to generate the perturbed data set and to train the energy model).  The authors should especially clarify whether their approach is usable in practical applications of structured-output prediction with hundreds+ of variables.

- The first experiment is entirely synthetic.  This is fine.  However, I don't think that one needs to learn a SPEN that simulates Eq. 13.  Why not use as ground-truth Eq. 13 directly?  Fitting a SPEN on those equations introduces a possible mismatch between the formulas (which are used to evaluate the quality of the model's explanations) and the model (from which the explanations are extracted).  

- I did not find Figure 5 to be informative.  It is very, very hard to make out what the reported explanations say about the model at all.  Which makes it hard to claim that they are understandable in the first place.  I don't think that image segmentation is a good showcase for the proposed method.

- If the learned interpreter happens to be of poor-quality, the proposed technique could provide unfaithful explanations & hide bugs in learned models.  The authors do not address this issue.


MINOR ISSUES
------------

- The distribution q_alpha defined as in Eq. 4 has a very simple form: it is a uniform distribution over the solution set of the argmax.  I don't think that the continuous uniform distribution is part of the exponential family, hence the assumption made by the authors does not hold.  The authors should change the sentence "we assume q_alpha is from the exponential family" to "we opt to model q_alpha using an exponential family distribution, for simplicity" or similar.

- I could not find the definition of the \mathcal{L} term in Eq. 6 (i.e., the margin), so I am not sure why "the penalty value is zero" if \hat{y}_t = y^{sb}_t (right after Eq. 6).

- The distribution p(x) in Eq. 9 is undefined.  It is not clear to me whether it denotes the data distribution.  I suggest to define p(x) precisely.

- Please replace "locality" with "region" everywhere in the text.

- Please replace "perturbating" with "perturbing" and "perturbated" with "perturbed" everywhere in the text.

- p 4 "more simpler" -> "simpler"

- p 6 "information theoretic"

- p 6 "overall better perforamnce" -> "better *faithfulness*".

- p 7 "As the number of input features is increased, the performance of methods is generally degraded."  This hints at a general issue with this approach: that learning the surrogate becomes harder and harder for bigger problems.  This however manifests itself in uncontrollable performance rather than longer runtime requirements.  The authors should address how to detect this issue and how to avoid it, so to prevent users of their system from being fooled from unfaithful explanations. Interesting research direction but unconvincing text, technical contribution, and empirical evaluation

The authors propose an energy-based training method for achieving model interpretability, which performs instance-wise feature selection. The proposed model adopts a similar approach of one of the pre-existing interpretable methods by calculating feature-level importance score with regard to each instance. The authors validate their method on synthetic and public datasets. The tackled goal is interesting meaningful in fields of interpretable artificial intelligence and the authors achieved this by 
estimating feature-level score with an energy function-based approach. The proposed methodology is specifically described, but the writing needs to be elaborated. Introduced experimental settings are properly designed with synthetic and public datasets and the written results seem to be reasonable. However, more various experiments should be performed with qualitative and quantitative analysis.

Although interesting, It’s a bit hard to say the introduced technique is new. I can’t see the novel point for the proposed methods as the method seems to be a combination of pre-existing methods in model interpretatility. Also, the experimental results are relatively weak  Although interesting, I think more analysis is needed to make their method more interesting. 


This paper propose a method for interpreting structured output model. The key idea is to find an "interpretation" which explains an "target" output random variable based on subset of rest of the output variables. The training objective is on finding a small subset which keeps the target output random variable invariant. The proposed methodology is applied to explain a synthetic energy function and structured prediction energy networks.  In general, I found the paper quite hard to read and motivation of the paper was not very convincing. I think this paper could be improved by a careful rewriting of their motivation, methodology and terminology. 

1. Motivation of the interpretation task 
The overall motivation behind the task of explaining a structured prediction model as a top-k important feature is not very convincing to me. I would like to understand the real-world applications where interpretation of such a structured prediction model is needed. The authors provide interpretations of multi-label classification and image segmentation, but I do not understand their main purpose.

2. Definition of the structured output prediction model
The definition stated in the paper "structured output prediction models map a feature vector to the output including a set of correlated variables with known and unknown complex relationships" seems to describe any deep neural network with random vector as an output. I think this paper could use more precise description of the problem.

3. Objective & motivation of the training objective for the interpretor. 
The motivation behind the proposed training objective should be clarified (or strengthened). Equation (1) is stated as motivation behind the training objective, but it seems to be a trivial conditional entropy inequality in my perspective. Section 3.1 was very hard to follow because there were many equations that appears without any justification, e.g., Equation (6). 

 This paper seems to pursue an interesting direction of developing an explanation method for structured prediction model. However, motivation and methodology of the paper was very hard for me to understand and get convinced. I also think the proposed method lacks practical applications, i.e., any real-world situations where people require the explanation (provided from methodology of the paper). 

Meta-review: This paper proposes a method for interpreting structured output model. All the reviews are negative. The reviewers find the paper difficult to read, and lacking in novelty, technical contribution and empirical evaluation.

Decision: Reject