Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a self-explaining model generating logic rules as explanations. It aims to learn explanations that (1) make accurate prediction, (2) are coherent with human's prior belief. The authors provides several experimental evidences supporting their claims using three benchmark datasets (two text datasets and a tabular dataset). The major contribution is on the quality of explanation where their method achieved a good performance compared to the others. But the evaluation of explanation quality is limited to human precision. It seems like the authors perform several other experiments (C.4.4, C.4.3) that are potentially related with explanation quality, but didn't discuss their experimental findings or relate with the explanation quality. I expect the paper can be improved during the rebuttal period, and am willing to adjust my score if it addresses several concerns listed below.
strengths_and_weaknesses: ### Quality
Strength
- If we assume Figure 1c is well designed process, the model component design itself has a good rationale. The model is composed of three parts (human prior, consequent estimation, antecedent generation) with corresponding loss terms. Each part takes a role to achieve the coherence, prediction, and generation. 
- The authors' main claims (prediction accuracy, and human precision) are supported by various experiments including human evaluation. 

Weakness
- The generative process (Figure 1c) should be provided with a good rationale, or in-depth discussion why it makes sense. [See Q1] 
- Explanation quality should be evaluated in different aspects. (It is important especially when the prediction performance are not substantially good enough.) [See Q4]

### Clarity
Weakness
- Lack of details about important experimental settings: "Base" model [See Q3]  


### Significance
Strength
- The authors' main claims (prediction accuracy, and human precision) are supported by various experiments including human evaluation. Especially, it achieves a good performance on the human precision evaluation.  
- In addition to the main experimental results, the paper provides several interesting experimental observations about relationships between model interpretability and robustness, and potentiality for debugging and refinement using the model explanations. Although those are not based on in-depth or wide-breadth experiments (probably because this is not directly related with the authors' main claims), those observations can fuel interesting future works in this direction. 

Weakness
- It seems like there are no significant improvement in the classification performance (Table 2). 

questions: ### Majors
(1)  Figure 1c: What are the assumptions you made for the generative process? For example, see the following questions:
- Do you assume that the antecedents (explanations) and corresponding 1st order logic rules are sufficient for making accurate prediction? 
- Do you assume that given $\alpha$, $\mathbf{x}$ and $b$ are independent? What is rationale for this?

(2) Provide AUCs for Table 2. 

(3) Provide details about the "Base" model. It is usually expected model performance degrades when an explanation mechanism is added to the model. It is important to compare with reasonably well designed baseline models to precisely evaluate classification performance of explainable models. (Btw, it is sometimes good enough to have comparable prediction performance with prediction-only baseline models, if it can provide substantially good quality of explanations in different aspects.)

(4) The following questions are related with the quality of explanations. 
- Are there any factors (e.g., rule length, number of atoms) impacting explanation quality? Do you expect trade-off between prediction performance and quality of explanation? 
- How unique the explanations are for an instance (sample)? (i.e., do the models trained on two different random seeds generate the same or similar logic rules for an instance?) Is there any factor controlling the uniqueness? -- I used the term "uniqueness" instead of "robustness" because the authors already used the term "robustness" for another purpose in the paper.

### Minors
- Provide standard error of the five runs for the classification performance in Table 2. (Figure 8 is unnecessary).
- Fix typo in the equation (1)
- No content in C.4.2 
limitations: First, it should discuss how likely the assumptions made for Figure 1c are correct in practice, and what we need to expect when they're not satisfied (see Q1). Second, it should discuss that the explanation quality can depend on several factors (see Q4), and careful designed experiments are required to get good enough explanation.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper describes a self-explainable model in which explanations consist of logical rules consistent with the data and decisions and decisions are consistent with these explanations (at least locally).  Both elements are implemented using neural networks.  The model also supports prior knowledge on admissible rules.  Results on a selection of textual and tabular data sets illustrate the performance and robustness of the proposed approach compared to black-box models, post-hoc explainers, and self-explainable alternatives.
strengths_and_weaknesses: **Originality**: The proposed approach combines familiar elements into a novel architecture.

**Clarity**: English is good, but the text - and especially the description of the method - is partially opaque.  Notation is not entirely clear.  This makes it hard to assess the properties of the proposed model.

In particular, Eq. 1 is confusing as it looks like the explanations are being marginalized over, which would mean that any one prediction could be the result of potentially exponentially many explanations (all those appearing in the sum and having probability > 0).  This would conflict with the goal of building an interpretable model.  To the best of my understanding, however, this is not the case, as both p(y|alpha) and p(alpha|y) are essentially discriminative, so no real marginalization takes place at inference time.  This is not deal-breaking, but it is an issue.

**Quality**: At a high-level, the proposed model is reasonable.  One issue I have with it is that it looks very complex - much more so than SENNs or concept bottleneck models.

Q1. Another element of confusion is the relationship between rules for different inputs:  is it possible for close inputs to have arbitrarily different explanations?  In other words, what is the ``radius of validity'' of the local explanations produced by the model?

Q2. More generally, is it possible that the rules predicted by the model are inconsistent with each other, i.e., that the rule predicted for an input x and that *should* apply to input x' is inconsistent with the rule predicted for input x'?

**UPDATE**: Q1 AND Q2 HAVE BEEN ANSWERED BY THE AUTHORS IN THEIR REBUTTAL.

Lack of explanation generalization across points would complicate global interpretability (and woese self consistency) of the model.

The numerical experiments look okay, with a reasonable selection of data sets and a number of competitors.  One issue is that the human evaluation is *very* small scale - it only involves three human subjects.  Platforms like Prolific however make it quite easy to setup larger scale experiments at a very reasonable cost, so the choice of restricting the experiment to only three people is puzzling.  I am not sure that the numbers reported have statistical significance.  See questions below.

**UPDATE**: THE AUTHORS HAVE REPORTED MORE EXTENSIVE EXPERIMENTS IN THEIR REBUTTAL.

As for related work, the authors write: "Collecting ground-truth labels of human decision processes are quite expensive, if not impossible."  But there exist a number of ML approaches that do exactly that - using post-hoc and self-explainable models.  See for instance:

  Kulesza et al. "Principles of Explanatory Debugging to Personalize Interactive Machine Learning" 2015.

  Schramowski et al. "Making deep neural networks right for the right scientific reasons by interacting with their explanations" 2020.

  Lertvittayakumjorn et al. "FIND: Human-in-the-loop Debugging Deep Text Classifiers" 2020.

  Ciravegna et al. "Human-driven FOL explanations of deep learning" 2020.

  Stammer et al. "Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanations" 2021.

  Bontempelli et al. "Toward a Unified Framework for Debugging Gray-box Models" 2022.

just to mention a few.  I would expect there to be also works on acquiring rules from human annotators in the context of logical and neuro-symbolic models, but I cannot provide pointers.  I would have expected better coverage of works on debugging neural networks and self-explainable models.


**Significance**: The model is of interest to research in XAI and especially for self-explainable models.

Other comments:

- l 121: "The deep model f is used to encode the sample x"  Wasn't f the complete model?  Also, at this point it is not clear what
  a "recursive generation layer" is.  I'd suggest to move lines 121-123 to a later section.

- l 125: "Our framework for deep logic rule reasoning bears an interesting resemblance to the human mind..."  All concept-based models and many neuro-symbolic models do.  But it's only a vague analogy and I'm not sure what it adds.  Moreover, notice that post-hoc models bear a striking resemblance to models of the mind in which decisions are taken subconsciously and then interpreted after the fact by an argumentative interpreter (see books by Michael Gazzaniga).  There is no conclusive evidence of how our minds really work in this regard.  Also, is really the subconscious mind "often thought to be more powerful"?

- l 183: square brackets, missing comma.

- l 191: organizes

- Eq 3.  The equality symbols should be replaced by \approx, especially the second one (as p is replaced by \hat{p}).
questions: - See Q1 and Q2 above.

- Are the results reported for the human study statistically significant?

- l 279: "The participants are allowed to choose multiple best explanations only if the chosen ones are the same"  Why?  Can't two explanations be equally good (or bad)?
limitations: Partially, in the conclusion.  Other limitations include lack of generalization of produced explanations across inputs and lack of self-consistency of explanations at different points.  I asked the authors to comment on these in my questions above.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper addresses the task of explaining deep models with logic form and proposes an explanation model applicable to text and tabular data, which generates latent antecedent and then makes a prediction based on the generated antecedent.
The experiments on review text and tabular datasets show that the proposed model outperforms the comparing methods in terms of classification performance and explainability.
strengths_and_weaknesses: ## Strengths
- The proposed model looks well designed, and the construction of each component of the proposed model, such as human prior, consequent estimation, and deep antecedent generation are reasonable.
-  As shown in Table 2, regardless of making the unexplainable models, such as BERT explainable, the proposed model can remain high classification performance.
- The paper is well written, and the performances of the proposed model in terms of classification performance and explainability are well analyzed with both numerical experiments and user study.

## Weaknesses
I don’t have any weaknesses to point out.
questions: - How do you make the candidates of atoms? Focusing on the word 'awesome', many candidates of atoms associated with 'awesome' can be made, such as 'awesome$\geq 1$', 'awesome$\geq 2$', and so on.
- Is there a possibility that atoms associated with the same word, such as 'awesome$\geq 1$' and 'awesome$\geq 2$' are generated for the explanation in an instance? 
limitations: NA
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes an interpretable-by-design framework for deep learning, leveraging logical rules in the form of propositional implications to explain model’s predictions. 
Specifically, the body of the rule (alternatively called antecedent in the article) is predicted by a carefully chosen deep learning architecture, whereas the head (alternatively called consequent) consists of the result of the implication rule, namely the label prediction. The deep learning component includes (i) a backbone encoding network to learn a representation from the raw input data and (ii) an autoregressive model transforming the latent representation into the predicting atoms (Boolean features) composing the body of the logical rule.
The training is performed in two stages. In particular, an auxiliary network is first trained to mimic the prediction and the confidence of random implication rules. The learnt network is then used as a prediction head for the above mentioned deep learning component to minimise the prediction error together with additional user specific constraints in an end-to-end differentiable fashion.

The proposed approach is compared against  two self-explainable and two post-hoc explanation solutions on two textual and one tabular datasets. The results show that the proposed model is able to outperform previous approaches in terms of explainability as measured by human precision, while retaining state-of-the-art predictive performance.

strengths_and_weaknesses: **Strengths**

- The paper is clearly written and the use of logical rules is well motivated.
- The considered problem is relevant and timely.
- Experiments show the superiority of the proposed solution over the baselines especially in terms of explainability. I liked the analysis of the predictions in Section 3.3, as providing convincing evidence on the utility of logical rules. This is an original part of the work.

**Weaknesses**

- The paper overlooks an important body of research in the emerging field of neuro-symbolic learning. Indeed, this is not the first time that rules are learnt in symbiosis with deep learning, see for example [1] and [2]. Experimental comparisons would be required against these strategies, but I realise that at this stage this could not be feasible. Authors should at least cite and discuss works in neuro-symbolic learning and highlight the point of novelties of their proposed solution.
- The work focuses on learning logical rules in the form of definite clause logic (a subclass of propositional logic), thus inheriting its limitations in terms of expressivity and also explainability. A discussion on these limitations is required.

[1] Learning Explanatory Rule From Noisy Data. JAIR 2018
[2] DeepProbLog: Neural Probabilistic Logic Programming. NeurIPS 2018

questions: 1. Can you please elaborate on the novelty of the proposed framework in relation to neuro-symbolic approaches?
2. What are the soft constraints and loss used in the experiments?

limitations: 1. Are there any limitations on the chosen class of logic? How would this influence the human precision?

MINOR MISTAKE

Please replace $p(\alpha|b)$ with $p(b|\alpha)$ in Eq. (1), line 115, line 131, multiple occurrences in line 132, line 133, line 142 and Eq. (6). Also, modify the generative process in Figure 1 (c) to take into account the correct dependence between $\alpha$ and $b$

-------------------------------------------------- 
POST - REBUTTAL

During the rebuttal, the authors have properly addressed all my questions. Also, they have provided concrete actions to take into account the weaknesses of the manuscript (i.e. missing related work on neuro-symbolic learning, discussion on language expressiveness). Overall, the work provides an interesting contribution for the application of neuro-symbolic learning in the area of explainable AI. I'm happy to recommend for a weak accept.
review_title: null

Meta-review: metareview: The paper deals with the important topic of devising accurate predictive models that are able to distill explainations that can be easily understood by humans. In particular, authors propose a deep neural network that predicts logical rules over which a human-specified prior can be imposed.

The reviewers agreed that the scope of the contribution is relevant and the contribution is timely. At the same time, they highlighted some shortcomings concerning the experimental setting (e.g. some metrics or baselines missing), the motivation and effect of certain assumptions over the explainations (e.g., are rules consistent). 
During the rebuttal, the authors managed to address the aforementioned concerns in a satisfactory way which saw some scores improve (kudos!)

The paper is accepted conditioned on the inclusion of all the experimental material and discussion that emerged during the rebuttal.
confidence: Less certain
Decision: Accept