Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary:
-------------
This work presents first of a kind logic-based framework that relates contrastive (minimally absent) and abductive (minimally present) explanations and shows that abductive explanations are essentially minimal hitting sets of contrastive explanations. 

+ve:
-----
A much needed formal framework and proofs that relate the different types of explanations. With many different types of explanations proposed in the XAI literature, this line of work improves are understanding of the overall space & taxonomy of explanations. Also the relationship between types of explanations may helps us enumerate/compute other types of explanations based on the computation of one type. 

Suggestions:
-------------------
It would be good to comment a bit on the overall steps needed to convert any ML problem into the proposed framework - costs of binarizing feature values, etc. before applying the proposed ideas for local or global explanations. 

review_title: A formal framework to understand contrastive and abductive explanations

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose to extract two types of explanations: abductive and contrastive explanations to address a gap in the literature of explainable AI. Indeed, that's a great point and often explainable models address the "why" and rarely the "why not" that can help identify the features guiding the change in the class. 
The ideas presented are compelling and it is good to see that we can re-use state of the art AI first order logic (FOL) statements in the field of explainable models. The references are excellent. 
However, the paper suffers from several drawbacks: 
1- The setting is limited to ML models that are expressed as a set of FOL sentences
2- Discretization of numerical features is required. We know static discretization can be problematic (large versus small interval); no discussion is in the paper to how to address this point
3- More details should be presented about SHAP since this is the main method the authors compare to;
4- Figure 1 part c) is not explained. There seems to be missing parentheses in the FOL statement;
5- The experimental section is rather weak.  The example on the real-vs-fake digit is not clear. Pointing out the brighter pixels as those responsible of the classification is not convincing to me. Comparison to Shap (using correlation) is not discussed.
The second experiment. provides some statistics on the time, number of abductive and contrastive explanations. Perhaps it would have been good to provide other examples of the importance of extracting both explanations and how pertinent they are. Overall, there is a need for some baseline to validate the explanations (both types); 
6- The methodology should be made clearer, notations introduced or re-introduced, many readers might not be familar with some notations like entailments; 

Overall, I feel there are good ideas in there, the authors should consider enlarging the spectrum of the applicability of their approach, may be rework the definitions and methodology section and design more solid experiments. 

Minor comments:
seeFigure


review_title: Compelling idea but limited setting and experiments

final_score: rating: 6: Marginally above acceptance threshold
review_content: Overview
This paper develops, what it calls, contrastive explanations using first order logic. Contrastive explanations answer the why not question, while standard explanations (abductive in this paper's language), answer why. This paper formulates the problem as a first order logic problem and then leverages what seems like classic algorithms from that domain to learn these explanations. First, I want to point out that I have no expertise in first order logic, so this review is perhaps an educated guess as to the quality of this work. 

Clarity/Writing
The writing is relatively clear and straightforward. One issue is with the citations: there is a missing bracket with the citations, which then makes them part of the sentence and ultimately distracting. 

Quality/Significance. 
Contrastive explanations could be useful for a variety of tasks and importantly provide actionable insights about how to change a sample to satisfy positive rating. In that regard, the goal of this work is a worthy one. I cannot judge the quality of this work since I donot have any expertise in formal logic. 

Questions
What is the difference between the contrastive explanations as defined here and what watcher et. al. and the literature calls counterfactual explanations? 

I am somewhat surprised at the computational performance from the results presented here; these seem much faster than I would have expected. It seems like the task of identifying a contrastive explanation, for the discretized data set, should be NP-Hard. Is this the case? 

In looking at figure 2, I can't really see the differences in attribution for CXp for the real data vs the fake one. How does the CXp help distinguish real from fake data? In the summary of results, the authors conclude that the CXps enumeration provide insight into the behavior of the classifier. However, the behavior is not discussed nor is any insight provided. What are the authors referring to? 

review_title: develops contrastive explanations using first order logic

final_score: rating: 5: Marginally below acceptance threshold
review_content: I thank the authors for their submission. I believe the investigated content is relevant and timely and would perhaps benefit from the discussion in a community such as the one of ICLR. Please find my comments below, as potential points of discussion.

High-level comments:
* generally, the paper does a good review of existing literature and aims to relate two important subfields (abductive vs contrastive explanations) using rules of logic, particularly using the minimal hitting set relationship 
* beyond showing that such a duality holds between abductors and contrastive explanations, I believe the experimental section should further explore comparisons with related work such as Dhurandhar et al. (as cited in earlier sections) and Rebeiro et al.
* furthermore, the idea of using FOL for generative contrastive explanations (also sometimes called counterfactual explanations) has been explored before (e.g., [Karimi et al.]);
* on the presentation of material, there seemed to be an underwritten requirement to be familiar and have a background in logic and verification, which makes me wonder whether ICLR is the right venue here (I also apologize for not being to provide much feedback on the technical front)

Minor comments and nits:
- the footnotes seemed to contain important details, but the number of footnotes seemed overwhelming and hurt the flow of reading
- [footnote 9] seems incorrect; e.g., non-linearities in MLPs or RBF kernels in SVMs cannot be encoded as first-order logic
- perhaps figure 2 can be redone to more visually demonstrate the benefit of using the proposed method; if the provided explanations aren’t visually appealing (relatively), then perhaps consider a non-image-based dataset?

[Ribeiro et al.] https://homes.cs.washington.edu/~marcotcr/aaai18.pdf
[Karimi et al.] https://arxiv.org/abs/1905.11190 
review_title: Review

Meta-review: The authors consider local 'why' or 'abductive' explanations for a model and a given class, which identify a minimal subset of features such that they're sufficient to imply that the model predicts the class; and 'why not' or 'contrastive' explanations, which identify a minimal subset s.t. they're sufficient to imply that the model predicts a different class. The two types of explanation are related using earlier work on minimal hitting sets going back to Reiter (1987). 

Reviewers were divided in their opinions. R4 was very positive but with little detail and only medium confidence, then did not participate in discussion. R2 was the only reviewer with high confidence, leaning against acceptance. The paper relies on FOL which was hard for reviewers to grapple with, and may make it challenging for readers. The presentation could be improved by clearly linking to existing work and demonstrating why the new approach is important.
Decision: Reject