Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The author addresses the problem of post hoc explanation for the black box model. In this paper, the author discusses the task of explanation for two inputs, and the model provides a similarity score as output. The author proposes a model agonistic local explanation method for tabular and structure data. In the proposed method, the author uses feature attributes to explain the similarity between two inputs. Finally, the author proposed an analogy-based explanation to select diverse analogous pairs of examples for the same similarity. The author claims that using the proposed method can explain state-of-the-art models in healthcare utilization applications.
main_review: Strengths:
The main reason to accept this paper is empirical results, showing performance on the various methods. The author has done plenty of case studies to verify the explanation and with many examples of the proposed explanation method. 

Weaknesses: 
Generalizability of the proposed method. The author shows results on language tasks.  Can it be applied to other tasks? If yes, the author should show results on vision tasks and shows and compare results with state-of-the-art methods.

To find better similar and contrasting examples in the vision domain, people use exemplar [1,2] theory to find supporting and opposing examples.

What sort of feature attribute did the author consider for the explanation? Do you have a section discussing feature attributes?

“set of perturbations (x _i, y_ i ) in the neighborhood,” what kinds of perturbation is used in the text domain?

The author should compare results with the SHAPE[3], LOO[4], RISE, and Occlusion-based method for input perturbation and U-CAM method for logit perturbation.

How is it(analogies) different from paraphrasing a sentence? The author could motivate the paper on why analogies help to improve the explanation.


How did the author measure similarity between two inputs (word or phrase or sentence level)? Do you have an analysis of this?

However, the paper misses one of the core aspects of machine learning practice: readability and reproducibility of results. What are the various critical components in the proposed method? The author should provide an algorithm or pseudocode to reproduce the results missing in this paper.


Ref:
1. Jäkel, Frank, Bernhard Schölkopf, and Felix A. Wichmann. "Generalization and similarity in exemplar models of categorization: Insights from machine learning." Psychonomic Bulletin & Review 15, no. 2 (2008): 256-271.

2. Patro, Badri, and Vinay P. Namboodiri. "Differential attention for visual question answering." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7680-7688. 2018.

3. Lundberg, S. M.; and Lee, S.-I. 2017. A unified approach to interpreting model predictions. In Advances in neural information processing systems, 4765–4774.

4. Li, J.; Monroe, W.; and Jurafsky, D. 2016. Understanding neural networks through representation erasure. arXivpreprint arXiv:1612.08220.


summary_of_the_review: I have worked in this field and published related to this work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Goal: provide local explanations for black box (BB) models that assign similarity scores to two input examples. 

Approach: Two explanations generated: 1) feature attribution and 2) similar pair of examples that serve as analogies. 

Approach for 1) is:

Approximate the BB model on the instance as if it was a quadratic model of the pair of inputs and learn the weighting matrix A by sampling pairs of points around the input and solving the resulting SDP for the matrix A. The weights of A provide a way to assign value to each element of the inputs. 

Approach for 2 is:

Obtain K different pairs of examples that serve as analogies by solving a subset selection problem from the data that balances three terms:

1) the analogy pair must have similar score to the query according to the BB model

2) the analogy pair and query should have similar features highlighted (weighted by a HP $\lambda_1$

3) the K pairs should be diverse

Evaluation: The authors show three different kind of evaluations

Qualitative examples: on text data from STS, the authors show three examples with the results of their method

A user study on the STS dataset where participants have to predict if the BB predicted the sentences to be similar given the two kinds of explanations. They show that their method outperforms the baselines

Quantitative results on 3 datasets: STS, UCI Iris and MEPS where authors show that their method outperforms LIME and other baselines in terms of a metric called “generalized infidelity” 

main_review: Strengths:

- Novel formalization of objective function for finding analogies and feature attribution for BB similarity learners

- Diverse evaluation of approach using both objective metrics and a user study

Weaknesses/questions:

- On objective 1: how can one compare doing LIME over the concatenated (x,z) to having A be diagonal?

- On objective 2: Optimization over $\lambda_1, \lambda_2$ is unclear, how can one systematically search over them to get intuitive analogies? Furthermore, why is $\alpha$ set to 0 in all the experiments? What is the effect of $\alpha$ both quantitatively and qualitatively ?

- Figure 2: why are the words on the x and y axis shuffled from their original order? Also this kind of visualization is a bit hard to parse, is there a better way to visualize the cross weights (off diagonal elements of A) ?

- Issues with user study: 1) using google forms and non-paid participants raises questions on the effort each put into performing the user study.  2) showing participants the same example with different explanation methods: as I understand, there are 10 test examples, and you show each participant the same 10 with 3 different random methods to explain. Thus the participant has access to 3 explanations when they get to third time they see a given example, this clearly biases the results. A correct methodology is to assign each participant to a condition ( an explanation baseline) and only show them 10 examples with explanations from that condition. Then you compare between conditions. 3) no alignment between objective of user study (replicate BB model scores) and practical use cases: supposedly the explanations are to check if the BB is correct or not, why wasn’t that the use case for the user study? I expect it’s because humans are perfect at judging similarity, then it might have been more interesting to introduce an end task where judging similarity is used. 

 -  I am not sure why is fidelity the “metric” to compare things across for judging similarity.  Furthermore, it would have been helpful to understand the implications of a low generalized infidelity score and a high score. (Ramamurthy et al., 2020) also relies on comparing the feature importance weights, is it possible to do something here that is similar? 

summary_of_the_review: The paper presents a novel approach for obtaining explanations from a black box measure. The method appears sound, however, the evaluation is lacking in certain aspects. The user study has some flaws and the quantitative experiments rely on a single metric. My recommendation is a borderline reject that can be improved if authors can better argue their evaluation approach.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new technique for explaining models that predict the similarities of an input pair. The authors propose two forms of explanations for such models: feature and analogy-based. Feature-based explanations highlight the important features of a predicted similarity for an input pair. For the explained pair, analogy-based explanations provide a new input pair that has a similar relationship to one another. The proposed technique outperforms other similar techniques in human and functionally grounded empirical experiments.

main_review: I have summarized the main review into the following pros and cons: 

Pros: 

* The proposed technique is flexible as it can provide two forms of explanations: feature and analogy-based. Moreover, explanations in the form of analogies are intuitive for human users. 
* The study includes human and functionally grounded evaluation experiments to show the usefulness of the proposed explanation technique.

Cons:

* Many important design choices behind the proposed method in sections 4.1 and 4.2 are not well motivated.
* Some of the methods in functionally-grounded evaluation are not included in the human grounded evaluation experiments and vice versa. This makes it difficult to draw a general conclusion in favor of the proposed approach across both types of evaluation methods. 

summary_of_the_review: Overall, I vote for rejecting the paper. Although the proposed technique performs well in both human and functionally grounded evaluation experiments, many important design choices are not well motivated. Overall, I believe that the study needs some further refinements before it can be accepted to ICLR 2022. 


I have divided my detailed feedback into two categories: “major concerns” and “minor improvements”. I am willing to improve my current score in case the authors can address points raised in the major concerns section.

Major Concern

* What are the reasons that LIME and JSLIME are performing relatively similar in comparison to the proposed FBFull and FBDiag methods on MEPS dataset (Table 1)? Does that mean that the problem at hand can be solved with LIME and JSLIME formulation as well? If so, what are the benefits and limitations of the proposed explanation techniques in this paper? 

* How can the usefulness of the analogy-based explanations be argued for when the result of user studies show that users can get nearly similar accuracies using AbE or FBFull (Figure 3)? 

* Can authors provide explanations on the effect of each of five additive components in Equation 2? 

* What are the reasons for not performing the human and functionally grounded evaluations on the same set of techniques? In addition, how can this affect the generalized statements about which explanation techniques perform best across both evaluation experiments? (For example, LIME and JSLIME are missing in human studies in Figure 3 whereas PDash is missing in the functionally grounded evaluation in Table 1)

* Why lambdas and alphas are not tuned per example and what is the effect of this on the fidelity of “local” explanations (section 5.1 - AbE hyper-parameters)? 

Minor Improvement

* Can authors provide a more detailed explanation for the problems that hinder the extension or use the work of [Zheng et al., 2020; Plummer et al., 2020; Zhu et al., 2021] for the problem at hand? 

* I see a potential problem in the additive definition of w_{x_i, y_i} (section 4.1). In the current definition, the loss cannot differentiate between these two cases:  perturbations x_i s are close to x and many y_i points are further away from y and vice versa. This can be problematic since removing and adding terms to the explained pair of instances changes the Mahalanobis distance asymmetrically (see Example 1-3 in Figure 2). Can authors confirm this and provide an analysis on the possible effect this can have on the quality of explanations?
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a novel form of explanations for similarity-based models. The authors present two methods to generate explanations, called Feature-based similarity explanations and analogy-based similarity explanations, where the latter one is a novel type of explanations explicitly developed to explain similarity learners. However, the authors state that both can be used simultaneously, i.e. the latter can use the output of the first.
The authors conduct a user study as well as a quantitative evaluation of the proposed methods with a comparison to previous approaches. Since the proposed setting (analogies) is novel, previous approaches compared with needed adaptations to fit the setting. 
main_review: The paper’s proposed explanation form seems to be very interesting. After the problem and the explanations methods are well motivated and introduced, the authors first illustrate them on selected examples. This shows nicely the methods’ purpose. However, the STS dataset's task and the selected examples do not well support the quality of the generated explanations and the benefit of analogies-based explanations. I could imagine that the MEP dataset would be more relatable.

After this illustration, the methods are extensively evaluated with a designed user study and quantitative evaluations both taking (adapted) previous methods into account. The results demonstrate the purpose and the benefits of the proposed methods. Summarized, the approaches seem to be very interesting, especially since similarity learners became more and more popular in recent years, even beyond the text and tabular data domain. Applying, evaluating and extending these methods for e.g. the vision domain seems to be interesting.
Therefore, I'm tending to accept the present work.

Minor comments:

It took me a while to understand Figure 1 when it is first mentioned. Consider using shorter samples or/and expanding the description in the introduction. After reading section 5.1 it became more clear.

summary_of_the_review: I'm tending to accept this paper as it is well written and provides interesting and novel approaches to explain similarity-based methods.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper studies the problem of explanation for similarity prediction models. Given a pair of inputs (x1, x2), the model to be explained assigns a distance (or similarity) score. The task is then to explain the model prediction on individual inputs. Two methods are proposed. First, a feature-attribution style explanation is computed by learning a local distance approximation, similar to the LIME objective. Second, an analogy-based-explanation is used, in which a set of existing pairs of data are selected, and the pairs are encouraged to both be semantically diverse and share similar model reasoning process at the same time. In the experiments, qualitative and quantitative results are presented. Qualitative results are delivered mainly for the STS dataset. Quantitative results are provided via a user study demonstrating that the users can better predict the model prediction when given the proposed explanation compared to baselines, and an automatic evaluation showing that a global version of proposed method perform better than existing approaches on the fidelity metric. 
main_review: ## Strengths: 

This paper studies an under-studied problem of explanation for similarity models. Due to the particular natures of the similarity prediction task, methods that do not focus on interaction effects (i.e. pretty much all local explainers for the classification setting) understandably could not perform well. The authors solved this problem by the use of a learned local distance matrix, in which interaction effects are clearly shown. In addition, the proposed method of analogy-based-explanation seems novel. The explicit treatment of diversity sets it apart from other explanation methods that also use whole data point for explanation, such as counterfactuals. 

## Weaknesses: 

Despite the strengths, I do have serious concerns about the experimental evaluation, which fails to convince me of the quality of the explanation. 

### Qualitative analysis

For the qualitative explanation, and AbE in particular, none of the three examples in Sec. 5.1 are convincing, and they feel more like post-hoc over-explanation on the auhors' part based on the analogy pairs produced. As a concrete example, consider the (author-provided) explanation on the analogy-based explanation provided in example 1: 

"The first analogy is very similar except that hackysack is a sport rather than a musical instrument." -- Yes, but does this show that the model is recognizing the same-type-ness of the entity (musical instrument or sport) when making the similarity prediction? 

"The sentences in the second pair are more similar than the input pair as reflected in the corresponding BB distance." -- This statement is not about explanation, but rather merely about model prediction. 

"The third analogy is less related (both sentences are about cricket player selection) with a larger BB distance." -- Again, this statement is only about model prediction. 

For example 2, despite some Internet search, I could not understand what a "dolphin scheme" is. Is it a particular way for economic fraud/exploitation (like "pyramid scheme")? As a result, I could not understand the author-provided explanations for this example, and I do not think it is a good opening example for the same reason. 

For example 3, "Both sentences express the same idea (second half faster than first half) but in different ways, similar to the input pair." -- This is a very loose assertion. In fact, if both sentences truly express the same idea, I would expect the similarity to be much higher (i.e. distance much smaller), but this pair is actually the most dissimilar pair among the three. 

### Quantitative analysis

I have serious concerns about the simulatability user study in the paper. In summary, this design is easily game-able. Since the users have access to the explanation at "test time", a simple AbE explanation for achieving such correct prediction would simply be to produce pairs with similar predicted distances, regardless of any similarity in the reasoning process. To make it even worse, if the users could be "trained" for a bit, a "Trojan explanation" could easily lead to very high user performance, without the users understanding the model at all. For more details about the "game-ability" of this approach and the "Trojan explanation" definition, see https://arxiv.org/abs/2012.00893 and https://arxiv.org/pdf/2006.01067.pdf. The authors are suggested to consult an earlier proposal for user study design https://arxiv.org/abs/2006.14779, which (in my opinion) avoids this loophole. 

In addition, a synthetic task with known groundtruths could objectively evaluate various properties of the proposed methods, such as whether the highlighted words are indeed important, or whether the analogical pairs also use the same reasoning pattern. Some ideas are discussed in https://arxiv.org/abs/2104.14403 and https://arxiv.org/abs/2105.06506. 

Minor: 

The authors could use \citet in places such as "Smith (2019) proposed", rather than the current "(Smith, 2019) proposed". 

For gradient-based explanation, the authors cite GradCAM, but I view it more as an adaptation of the original CAM to non-fully-convolutional architectures, and GradCAM are fundamentally about visualizing maximally activating input regions for certain convolution layers/filters. Instead, I would recommend the original Gradient saliency paper by Simonyan et al. (2013) or its SmoothGrad/IntegratedGradient successor. 
summary_of_the_review: Unfortunately, I do not believe that this paper meets the standard for publication. While I like the proposed theory, I am really unconvinced by the experimental evaluations. In fact, the qualitative AbE examples raise more questions than they answer, and make me doubtful that the method is really working as intended. A more careful experimental investigation, perhaps with some revision to the theoretical approach based on the problems found, would make this paper a much better one.
review_title: null

Meta-review: This paper presents two novel approaches to provide explanations for the similarity between two samples based on 1) the importance measure of individual features and 2) some of the other pairs of examples used as analogies.  The proposed approach to explain similarity prediction is a relatively less explored area, which makes the problem addressed and the proposed method unique. However, reviewers expressed concerns about evaluation methods and there were some concerns about the design choices that were not well motivated. The major issue is, as pointed out by the majority of the reviewers, the evaluation methods. Given the paper, reviews, and responses of the authors and the reviewers, it appears that there is certainly room for improvement for more convincing evaluation methodologies to convince a cross-section of machine learning researchers that the proposed approach advances the field. Overall, this is a good paper, but appears to be borderline to marginally below the threshold for the acceptance.
Decision: Reject