Self-explaining deep models with logic rule reasoning
Authors: Seungeon Lee
Xiting Wang
Sungwon Han
Xiaoyuan Yi
Xing Xie
Meeyoung Cha
Venue: NeurIPS 2022 Accept (2022)
Keywords: Self-explaining
Reasoning
Deep Learning
Interpretability

Abstract: We present SELOR, a framework for integrating self-explaining capabilities into a given deep model to achieve both high prediction performance and human precision. By “human precision”, we refer to the degree to which humans agree with the reasons models provide for their predictions. Human precision affects user trust and allows users to collaborate closely with the model. We demonstrate that logic rule explanations naturally satisfy them with the expressive power required for good predictive performance. We then illustrate how to enable a deep model to predict and explain with logic rules. Our method does not require predefined logic rule sets or human annotations and can be learned efficiently and easily with widely-used deep learning modules in a differentiable way. Extensive experiments show that our method gives explanations closer to human decision logic than other methods while maintaining the performance of the deep learning model.

Self-explaining deep models with logic rule reasoning
Seungeon Lee"
Xiting Wang! Social Computing Group Microsoft Research Asia xi www.@miccoooofeee.....o. com
Sungwon Han'
KAIST School of Computing IBS Data Science Group archon159@kaistt act...c..e. kr
KAIST School of Computing IBS Data Science Group 1io44151@kain ac.kr
Xiaoyuan Yi
Xing Xie
Meeyoung Cha'
Social Computing Group
Social Computing Group Microsoft Research Asia xing. sieemmicoooott.... com
IBS Data Science Group KAIST School of Computing mcha@ibs.c re. kr
Microsoft Research Asia
xi siaoyuanyimiccrooooo www.ono.iiimrooooooooooooooooo.e com
Abstract
We present SELOR, a framework for integrating self-explaining capabilities into a given deep model to achieve both high prediction performance and human precision. By "human precision", we refer to the degree to which humans agree with the reasons models provide for their predictions. Human precision affects user trust and allows users to collaborate closely with the model. We demonstrate that logic rule explanations naturally satisfy human precision with the expressive power required for good predictive performance. We then illustrate how to enable a deep model to predict and explain with logic rules. Our method does not require predefined logic rule sets or human annotations and can be learned efficiently and easily with widely-used deep learning modules in a differentiable way. Extensive experiments show that our method gives explanations closer to human decision logic than other methods while maintaining the performance of deep learning models.
Introduction
Deep learning has shown high predictive accuracy in a wide range of tasks, but its inner working mechanisms are obscured by complex model designs. This raises important questions about whether deep model is ethical, trustworthy, or capable of performing as intended under various conditions [I].
Many approaches have been proposed to help humans assess and comprehend model decisions. Recent work on explainability has primarily focused on providing post-hoc explanations for
black-box models that have already been trained [2] 3
g 10 Post-hoc methods do
not change the model and hence preserve the predictive performance while providing the additional benefit of explainability. These methods have achieved considerable success in providing valuable insights for model understanding, but there are also known challenges such as computational cost (I]] and trust issues [12. For example, many popular post-hoc methods test the complex black-box model thousands times to obtain complete and faithful understanding of the model around a single in- generally required,
stance [ 13 [4] Subroutines such full optimization reverse propagation introducing approximations or heuristic assumptions that may lead to misinterpretation [14] [5].
Because there is guarantee that explanations are always faithful to the model [12), there exists a "general uneasiness" among practitioners about using and trusting post-hoc explanations [16]. Self- explaining models naturally solve these issues, making them ideal choice when interpretability
Work done during internship at Microsoft Research Asia t Corresponding Author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Antocedont a (Explanation)
Consequentt (Prediction)
Logic nule
108
Inpur sample
Antecident (Explonsiion)
Consequent (Prediction)
Rule Inser
AND fusty
Seatiment
Deep ancocedent
Consequent estimation
y
Education
PhD
OR
Age 49
Income 50K
60
generation
Human prior
Atom
Logical
5
Human belief
connective
Confidenncs rsst
(a) Rule example
(b) Human precision
(c) Our generative process
Figure I: Reasoning with logic rules: (a) examples of logic rule explanations; (b) human precision for
logic rule and linear regression explanations [11]
(c) generative process of our logic rule reasoning.
can be considered from the model design phase 17
18
19 20 Because the explanation mechanism
integrated inherently, these models can predict and explain simultancously with a single forward propagation without any approximations or heuristic assumptions that decrease the faithfulness of explanations. Self-explaining methods may also improve robustness [1] and provide actionable insights for directly refning model parameters without having to calibrate the dataset 21 22
Based on these observations, we regard self-explaining models as providing stronger link between humans and machine learning models, reducing misunderstanding and allowing direct control of the model based on human insights. The main challenge in achieving this new level of human-machin collaboration then becomes how to ensure self-explaining models' precision both in terms of predic- tive performance and human precision. Human precision refers to whether models' explanations of decision-makiin processes align with human decision logic. Existing approaches ensure explanations to be easy to read, for example, by requiring explanations be simple and smooth in local area [.]. However, there is little guarantee that a given explanation is correct rationale for prediction ac- cording to human perception. For example, the explanation "awesome22" (i.e., the word "awesome" appears twice in reviews) is good rationale for positive sentiment, while "is21" = positive sen- timent is easy to read but unreasonable humans. Without insurance for human precision, users may constantly find unreasonable explanations, which can significantly hamper user trust and prevent them from identifying actionable insights for model refinement, An interesting research question, then, is: how can self-explaining models generate explanations that are consistent with human decision logic?
To answer this question, we need decide what information models obtain from humans. Collecting fround-truth labels of human decision processes for every input instance [23 24 25 26
27 28] is
expensive and limits the method's scalability. Moreover, forcing the model to make decisions exactly like humans may be unwise since it could limit its data learning capability or even learn human biases that may significantly decrease the model's performance. To address this issue, is important that humans provide guidance at a higher level that allows the models to learn freely based on data. Accordingly, we propose two desirable properties for human precision. The first property, global coherency, restricts the explanation form to be consistent with human reasoning logic, thereby minimizing the probability of misinterpretation. The second property, local coherency, requires that each explanation naturally lead to the prediction according to human perception, thereby making explanations a correct rationale for the model output. As humans can hardly provide guidance for each explanation, it is more desirable that the models can automatically guarantee local coherency based on human guidance on global coherency.
A key to satisfying these two properties is logic rules. As shown in Fig. la logic rules can have flexible forms that meet human logic and preferences, making them easy to satisfy global coherency. For example, the logical connectives can be traditional (e.g.. AND, OR, and NOT) or self-defined (e.g., BEFORE). Moreover, the logic rules explicitly model whether an explanation can lead to a prediction by testing the hypothesis across the entire dataset. This ensures meaningful relationship between explanations and predictions that Icads to local coherency. Fig. Tb shows that logic rule explanations achieve even higher human precision than local stability, while providing a confidence score that correlates with human precision (more details in Appendix A. Lastly, logic rules of different logical connectives correspond to diverse set of feature interactions, providing the expressive power for good predictive performance.
explanations with
This paper proposes SELOR, a framework for upgrading a deep model with Self-Explainaal Explainable version with LOgic rule Reasoning capability. Our work is inspired by neuro-symbolic reasoning (2), which integrates deep learning with logic rule reasoning to inherit advantages from both. The most related works in this discipline are guided search that finds a global logic program that works for (most) input-output pairs [19 30 31 32,, or identifies local logic program and rule for

---

36 37]. We adopt the latter paradigm, as global explanations for deep models usually fails to possess the same predictive power that is comparable with the deep models [14] Existing works for generating local programs or rules have achieved promising results by effectively leveraging instance-level guidance about local programs or rules [3 35 strong external knowledge such as knowledge graphs [6 [37 38) and a small set of predefined rules [39] However, our scenario, there is no instance-level guidance about the ground-truth rules, and leveraging strong knowledge such as small set of predefined rules may introduce bias into the deep networks, as shown in our experiment results of RCN (3). To address this, we propose logic rule reasoning framework that leverages global level human priors about rules (e.g., desirable form and property of candidate atoms) and generate explanations by optimizing rule confidence, which can be automatically computed based on the training data. Moreover, we design neural consequent estimator that can accurately approximate the confidence even for rare rules and combine it with recursive Gumbel-Sofftax [40) search the solution space effectively, Codes are released at Github
the given instance 33 34
35
Our main contributions are as follows.
Our work suggests that human precision is key for self-explaining models to bridge human logic and model decision logic seamlessly. Logic rule-based explanations enable high human precision while allowing the expressive power to achieve high prediction performance.
We propose logic rule reasoning framework that upgrades a given deep model into self-explainable version by naturally integrating human priors, rule confidence modeling, and rule generation as an essential part of model prediction. Our method can achieve high human precision without depending
strong external knowledge, such as instance-level guidance about rules, knowledge graphs, or a small number of rule candidates.
Numerical experiments and user studies confirm key strengths of our framework in terms of human precision and robustness against noisy labels with maintenance of prediction performance.
Deep Logic Rule Reasoning
2.1 Formulation of Logic Rules
logic rule a > y. as shown in Fig. la consists of an antecedent a and consequent y. Meanings of symbols used in this paper are defined in Appendix B.1
explanation in a logic form. It is represented as a sequence a = -(11..,0r,, Or,,, where is either an atom or a logical connective. An atom is the smallest unit of explanation that corresponds given input (e.g., "awesome 22"). The interpretable features may be different from those in deep learning models. They could, for example, have a different granularity (e.g., words or phrases) than the model features (e.g., partial words), be statistical feature (e.g.. word frequency), or be derived using external tools (c.g., grammatical tagging of word). Mathematically, each atom is Boolean-valle
An antecedent
the condition apply the rule and corresponds single interpretable feature of
function, with o, (x) returning true if the i-th interpretable feature is present in input x and false, otherwise. More detail about atom selection is in Appendix C.2 A logical connective combines atoms to form explanation. Logical connectives can be traditional ones like AND, OR, NOT, or self-defined ones, as long as they take one or more Boolean values as the input and output single Boolean value.
We say that an input sample x satisfies an antecedent a, if (<() true.
The consequent y is the model's 'ss prediction output given the antecedent. For example, is the
predicted class in a classification task, whereas y is an explicit number in a regression task. We
mainly consider classification in the paper and extend the cases to regression in Appendix B.2
2.2 Framework for Deep Logic Rule Reasoning
Let us denote as a deep learning model that estimates probability p(y]x), where the input data sample and y is candidate class. We upgrade model f to self-explaining version by adding a latent variable a, which an explanation in the logic form. Then, we can reformulate p(y) x) as
p(y x b) Σ ply a, x, b)p((((xx)) = Σ p(yla))(aaxxxb s.,, a(a) < S
(1)
a
3

---

Here, b represents human's 's prior belief about the rules, e.g., the desirable form of atoms and logical connectives, ((aa) is the required number of logic rules explain given input x, and is the number of samples (logic rules chosen by the model). Eq. ① includes two constraints essential for ensuring explainability. The first constraint p(y ox, x, b) = p(ylax) requires that explanation a contains all information in the input and b that is useful to predict y. Without the constraint, the model may "cheat" by predicting y directly from the input instead of using the explanation (more details in Ap- pendix B.3). The second constraint ((a) I S requires that the model can be well explained by using only S explanations, where small enough to ensure readability (S = I in our implementation).
We can further decompose Eq. ① based on the independence between the input x and the human prior belief b, following the generative process in Fig. (prof and assumptions in Appendix B.3)
p(y|x, b) Σ p(yla)p(a/xx, b)
Σ
p(bla)
plyla)
p(a)x) s.t., 2(()) S (2)
Human
Consoquent
Deep antecedent
prior
estimation
generation
The three derived terms correspond to three main modules of the proposed framework, SELOR:
Human prior p(b(a) specifies human guidance regarding desirable forms for rules to minimize the probability of misunderstanding and ensure global coherency (Sec. 2.3
Consequent estimation p(y| (x) ensures meaningful and consistent relationship between the
explanation a and prediction y, so that each explanation naturally leads to the prediction according to human perception and satisfies local coherency (Sec. 224).
Deep antecedent generation p(a{[x) uses the deep representation of input x learned by the given deep model f to find an explanation a that maximizes global and local coherency (Sec. 2.5).
The sparsity constraint ((x)) S S for the explanations can be enforeed by sampling from p(axxx. In particular, we rewrite Eq. (2) as expectation and estimate it through sampling:
p(ylx,b) x E(blaa) p(yla) p(o(x) = E p(b)a))(yle)) Rs p(al))
S Σ p(bla(())) p(ylo(())
(3)
.ED.8] a(t))
where (+) is the s-th sample of a. For example, to maximize the approximation term with S 1, the explanation generator p(a{(2: must find single sample a(*) that yields the largest and it needs to assign high probability to the best a Otherwise, other samples with lower p(bla(()) ((yyla may be generated, thereby decreasing p(y/x, b). This ensures the sparsity of p(x())) and the model interpretability. If there are multiple best explanations that result in the exact same p(bla (*))(( (pyl)a)))) the explanation generator may find all them.
2.3 Human Prior pb((a.)
Human prior p(b a)= = Pm (b. (b (b)aa) consists of hard priors pk. (blaa and soft ones (b(b).).
Hard priors categorize the feasible solution space for the rules: Ps (blaa) if a is not feasible solution. Humans can easily define hard priors by choosing the atom types, such as whether the interpretable features are words, phrases, or statistics like word frequency. The logical connectives to be considered (e.g., AND, NOT) can also be chosen, as well as the antecedent's 's maximum length L. SELOR does not require predefined rule set. Nonetheless, we allow users enter one if is more desirable in some application scenarios. A large solution space increases the time cost for deep logic rule reasoning (Sec. 2.6) but also decreases the probability of introducing undesirable bias.
Soft priors model different levels of human preference for logic rules. For example, people may prefer shorter rules or high-coverage rules that satisfy many input samples. The energy function can parameterize such soft priors: p. (b)a)) x exp( (xp(-Lo(ed Lo (ox)). where La is the loss function for punishing undesirable logic rules. We do not include any soft priors in current implementation.
2.4 Consequent Estimation p(yla)
Consequent estimation ensures a meaningful and consistent relationship between an explanation a and prediction y, so each explanation naturally leads to the prediction according to human perception. This is achieved by testing the logie rule a + y across the entire training dataset to ensure that represents a global pattern that is typically consistent with human understanding.

---

Empirical estimation. A straightforward way to compute p(yla) is to first obtain all samples that satisfy antecedent a, and then calculate the percentage of them that have label y [8. For example. given explanation a "rawesmme2"" if we obtain all instances in which awesome appears more than twice and find that 90% of them have label y = positive sentiment, then p(yla) = 0.9. Large p(yla) a.) corresponds to global patterns that naturally align with human perception. Mathematically, this is equivalent to approximating ply (a) with the empirical probability pyy(a))
p(yla) = Ra.y/Ba
(4)
where Pa.y is the number of training samples that satisfy the antecedent a and has the consequent y, and no is the number of training samples that satisfy the antecedent a.
Directly setting ()) to p(yla)) can cause two problems. First, when na is not large enough, the empirical probability p(y(ox may be inaccurate, and the modeling of such uncertainty inherently missing in this formulation. Second, computing P(y)a) for every antecedent is intractable, since the number of feasible antecedents A increases exponentially with antecedent length L.
Neural estimation of categorical distribution. To address the aforementioned problems, we jointly model p(y a)) and the uncertainty caused by low-coverage antecedents with the categorical distribution and use neural network to generalize to similar rules and better handle noise.
Assume that given antecedent a. y follows categorical distribution, with each category correspond- ing to a class. Then, according to the posterior predictive distribution, y takes one of potential classes, and may compute the probability of new observation y given existing observations:
plyla)na + B
p(yla) = p(y|Da B)
(5)
1a + K33
Here, Va2 denotes na observations of class label y obtained by checking the training data, and is the concentration hyperparameter of the categorical distribution that we automatically learn with backpropagation. Eq. (5) becomes Eq. 4 when na increases to co, and becomes a uniform distribution when na goes to O. Thus, low-coverage antecedent with small n is considered uncertain (i.e., close to uniform distribution). By optimizing Eq. (5) our method automatically balance the empirical probability p(y ox) and the number of observations Tla Probability p(ylax) also serves as the confidence score for the logic rule a + y.
We then employ a neural model predict p(y)a) and n better manage noise, generalize to similar rules, and improve efficiency. In particular, we obtain A' samples of a and compute p(yla) and naw by checking the training data. Here A' is significantly smaller than the total number of feasible antecedents A (Sec. [.6). We use the multi-task learning framework in [41] to train the neural network with these samples. In particular, we minimize the loss in following equation.
C.
(((y)a)) p(yla) 20
- IIna na 1 2 log TpTn
20
P(ylox), na are the predicted empirical probability and the coverage given by the neural model, and Up and on are standard deviations of ground truth prohability and coverage. More details for training the neural network are described in Appendix B.4 and Appendix. B.5 and effectiveness of the neural consequent estimator is shown in Appendix C.5.2
2.5 Deep Antecedent Generation p(a(xx)
Deep antecedent generation finds explanation by reshaping the given deep model f. Specifically, we replace the prediction layer in f with an explanation generator, so that the latent representation z of input is mapped to explanation, instead of directly mapping
prediction (c.g.. class label).
Given z, which is the representation of input x in the last hidden layer of f, we generate explanation a (oI.., o1) with recursive formulation to ensure that the complexity is linear with L (Sec. 2.6 Formally, given z and 01, 1.. we obtain og by
I[ E C.)xpp(hFo o:)
h Facoder (2:00::;00::
(1
En; I(a E C,)exp(h o.)
where o; is the embedding of o; and Encoder a neural sequence encoder such as GRU 42 or Trans- former [43] 1 is the indicator function, and C2 is the set of candidates for o, Every candidate should
5

---

satisfy both global and local constraints. The hard priors in Sec. 2.3 provide the global constraint and ensure that a has human deean- logic form. The local constraint requires that satisfies antecedent cx An atom "awesome" 2", for example, is sampled only if x mentions "awesome" more than once.
We then sample 0; from p( x, o1.., 0i-1) in differentiable way ensure easy end-to-end training:
0j = Guo E @ (1-x001)--11))). p(o[))= = II
p(o:) x, 01-1) +E[1.]]
Gumbel Straight-Through Gumbel-Softmax [40]. differentiable function for sampling discrete values. oj is represented as a one-hot vector with a dimension of |O) and is multiplied with the embedding matrix of atoms and logical connectives to derive the embedding o,.
2.6 Optimization and Complexity Analysis
Optimization. A deep logic rule reasoning model is learned two steps. The first step optimizes the neural consequent estimator by minimizing loss L. in Eq. (6. The neural consequent estimator only needs to be trained once for each dataset, and then can be used for various deep models and hyperparameters. The second step converts deep model f to explainable version by maximizing p(y|x, b) in Eq. (3) with a cross cross-entropy loss. This is equivalent to minimizing loss La = - La (+)) - log p(y" (a()),) where (c) punishes explanations that do not fit human's 's prior preference for rules (global coherency), and log p(y" (x*)) finds explanation (+) that leads to the ground-trute truth class y with large confidence (prediction accuracy), in which the confidence is measured by testing rule (*)) = y² in all training data (local coherency).
Complexity analysis. Time complexity is com- pared in Table The complexity for antecedent generation corresponds to the time added for generating the antecedents during model training compared to the time required for training the base deep model ʃ. Here, N is the number of training samples, and C' the time complexity for computing the consequent of each antecedent. As shown in the table, removing the recursive antecedent generator (RG) or the neural conse-
Table 1:
Time complexity analysis. -RG and -NE denote our method without recursive antecedent generation and neural consequent estimator.
Consequent
Antecedent
Estimator
Generator
SELOR -RG -RG-NE
O + A'L O(ACC
ONN NL W O(NA)
O(AC)
quent estimator (NE) brings an additional linear complexity with the number of feasible antecedents A, which is much larger than A'. For example, in our experiment, setting A' to 10* is good enough to train an accurate neural consequent estimator, while the number of all possible antecedents is A = 6.25 12 Here, do not include the analysis for sampling A' rules before training the consequent estimator. See Appendix B.4 for more details.
3 Experiment
3.1 Experimental Settings
We conduct experiments on three datasets. The first two are textual, and the third is tabular. Yelp classifies reviews of local businesses into positive or negative sentiment [44), and Clickbait News Detection from Kaggle labels whether a news article is a clickbait (45). Adult from the UCI machine learning repository [46), is an imbalanced tabular dataset that provides labels about whether the annual income of an adult is more than S50K/yr or not. For Yelp, we use a down-sampled subset (10%) for training, as per existing work (39 More details about the datasets are in Appendix C.I
Datasets.
Baselines. We compare our model to four basclines. Two self-explainable models, SENN and RCN 33, are compared in accuracy, robustness, explainability, and efficiency. Two post-hoc explainable methods, LIME and Anchor [14] are compared in explainability and efficiency.
Implementation details. To match with baselines, we use the AND operation by default in explanations. The impact of using other logical connectives is presented in Appendix C.5.3 The atoms, or interpretable features, are the same as in the majority of baselines, i.e., the existence of words for the textual dataset (e-g., "amazing"), and categorical and numerical features for the tabular data (c.g., "age<28"). More details including selection of atom candidates are in Appendix C.2

---

Table 2: Comparison of classification performance measured in AUC. The average results from five runs are shown. "Base" refers to the performance of unexplainable vanilla backbones. The best results among self eelf-exnlanning models are marked in bold, and the highlighted cells indicate a similar or better result compared with the unexplainable base model. The numbers in subscript indicates the standard error of the result.
Yelp
Clickbait
Adult
Average
BERT
RoBERTa
BERT
RoBERTa
DNN
Base
97.39 0.0859
97.90 0.0577
62.27 1.0400
63.72 0.8722
68.62 0.2317
77.98
SENN
96.00 0.1087
96.97 0.0841
55.64 1.0118
57.93 0.7779
67.39 0.0854
63.20
RCN
97.31 0.0274
98.03 0.0086
59.91 0,2024
59.37 0.2259
70.06 0.0411
76,94
SELOR
97.28 0.0335
97.78 0833
60.31 0,8198
64.14 0.5906
70.36 0,0892
77.97
Ours
Base
SENSEN
RCN
100
Classification performance (F1)
75
Classification performance (FI)
80
Classiflcation performance (F1)
95
70
65
75
80
55
70
0.00
0.05
0.10
0.20
0.00
0.05
0.10
0.20
0.00
0.05
0.10
0.20
Noisy label ratio
Noisy label ratio
Noisy label ratio
(a) Yelp
(h) Clickbait
(c) Adult
Figure 2: Experimental results on robustness under different ratios of noisy labels.
3.2
Classification Performance and Robustness
Classification performance. Table shows the classification performance of SELOR and baselines. Here, we evaluate the PR AUC instead of the ROC AUC because the latter is less suitable for imbalanced datasets (47. BERT 48 and RoBERTa 49 are used as the backbone networks in the NLP datasets, while 3-Layer DNN is used for the tabular dataset. The base method is the vanilla backbone network that does not support explainability (Appendix C.2 The prediction performance of post-hoc methods, LIME and Anchor, is the same as the base model as they utilize the trained model without any extra optimization. Comparison with a tully-transparent model is presented in Appendix C.5.1 Our method achieves comparable average performance with the unexplainable base model and outperforms other self-explaining models by 1.3%. Moreover, our method achieves the best comparable results various datasets against backbone models, demonstrating the expressive power of logic rules for high prediction performance. RCN cannot perform as well on challenging textual datasets like Clickbait because it computes soft attention over predefined rule set. This indicates that (potentially biased) predefined rule sets will limit the model's capability.
we assess the robustness of SELOR against randomly corrupted labels. We hypothesize that the effect of the noisy label is alleviated by consequent estimation term p(y] a)), where model verifies its decision by testing the logic rule over the entire dataset. For experiments, symmetric noise is introduced by randomly flipping the labels for a subset of the training data. Fig. 2 shows the results over Yelp, Clickbait, and Adult datasets with multiple levels of noise ratio from 5 5% to 20%. Our model outperforms other models across all scenarios. The improvement is substantial even with high noise ratio (i.e., 20%).. For noise ratio above 10%, our method consistently outperforms the unexplainable base models (2.4% to 13.7%).
Robustness to noisy labels. Following the literature 550 51
Sensitivity analysis. Due to space limitations, we show that the prediction performance of SELOR is stable under different hyper-parameter settings in Appendix C.5.4
3.3 Explainability
User study on human precision. To evaluate human precision, we recruited nine native English speakers through vendor company [52]. Each participant was provided with randomly selected 50 Yelp reviews and 50 Adult samples. Five explanations obtained from different methods were provided for each sample, and the participants reviewed whether the explanations offered reasonable rationales. Participants provided two labels for each explanation, indicating whether it good
7

---

Table 3: User study results on human precision. We show the average (Avg.) and inter- unter-particiaan agreement (Agr.) on the percentage of explanations that are considered good (a, b) or best (c, d). One star (+) means p-value is less than 0.05. Best results are highlighted bold.
(a) Percentage of good (Yelp)
(b) Percentage of good (Adult)
Avg.
Agr.
P-value
Avg.
Agr.
P-value
Lime
89.8
84,4
8.68 E-04*
Lime
42.7
57.1
6.09 E-54*
Anchor
84.4
87.7
1.12 E-07*
Anchor
52.7
59.9
5.56 E-18*
SENN
34.4
72.3
1.40 E-51*
SENN
46.0
51.5
1.18 E-41*
RCN
64.0
77.6
7.26 E-13"
RCN
60.9
53.2
2.83 E-27*
SELOR
94.4
93.9
SELOR
90.7
85.7
(c) Percentage of best (Yelp)
(d) Percentage of best (Adult)
Avg.
Agr.
P-value
Avg.
Agr.
P-value
Lime
34.2
67.6
8.87 E-03*
Lime
1.3
96.7
1.72 E-64*
Anchor
18.0
83.6
5.63 E-18*
Anchor
13.8
82.9
1.23 E-35*
SENN
2.4
96.3
6.84
E-40*
SENN
9.6
83.3
2.30 B-36"
RCN
2.0
96.3
6.84 E-40*
RCN
10.2
82.9
1.23 E-35*
SELOR
46.7
64.8
SELOR
65.1
58.4
or if it was the best one. A good explanation should naturally lead to the prediction, but it can contain noisy features. For example, "amazing, are" is a good explanation for positive sentiment. The best explanation is the one that contains the most important and least noisy features. The participants were allowed to choose multiple best explanations only if the chosen ones were the same. For a fair comparison, we showed explanations the same form: list of features each method considers important for prediction. Example explanations generated by our method and the baselines are shown in Fig. Note that in RCN means negation. More details about the explanation generation, labeling guidelines, and participants results are given in Appendix C.4
Table 3 shows that SELOR marks the highest percentage of good explanations, with average ratio of 94.4% on Yelp and 90.7% on Adult. Our method is also most frequently chosen as the best explanation. All results statistically significant according to the p-values from the t-tests. Although logic rules are promising, choosing from a small set of predefined rules may be insufficient due the potential bias in the rule set. For example, RCN uses rules extracted with traditional machine learning methods that meet the global data distribution but frequently fail to adequately represent each sample, particularly on datasets with many features like Yelp. As a result, RCN is rarely chosen as the best explanation, especially for Yelp text data. Post-hoc methods also tend to offer good human precision. The best ratio of LIME and Anchor, however, is substantially lower than ours, indicating that the base model may rely on more noisy features for prediction. In contrast, our method can verify its decision by testing the logic rule across the entire dataset.
Case study on model debugging and refinement. What useful insights can SELOR provide on performance? In study of 20,000 sampled Yelp reviews, clustered the generated explanations into 10 clusters by applying K-Means on the antecedent embeddings. [able/sshows five clusters with the lowest training accuracy to illustrate potential reasons for bad performance. Here, NULL is an empty atom when the model generated explanations that were shorter than the predefined length L.
We make the following observations. First, low training accuracy in cluster 5 is due to non-English reviews, which accounted for 0.31% % and led to underfitting. Second, performance degradation also
The staff was amazing with him and mode him, feel comfortable that he actually sot through an entire cleaning without being upset once...The dentist ond the assistants were made work children and that made oil very relaxing and enjoyable for him.
Adult
LIME Anchar SENN
RCN
Ours
age education educational-num marital-status
27
<32.5
<28
HS-grad 9
<12.5
<10
Separated
LIME
amazing, comfortable, enjoyable, more
occupation relationship race capital-gain capital-loss
Croft-repair
Anchor
amazing, so
Own-child White
SENN
about
amazing, "went, "seeing, "did
amazing, relaxing, comfortable, enjoyable
Figure 3: Example explanations produced by five methods on Yelp (left) and Adult (right).
8

---

Table 4: Case study on Yelp. We cluster the explanations for the training samples and show the five clusters with the lowest training accuracy. Num is the number of explanations in the cluster, and Len the average text length of the reviews. Potential reasons for bad performance are marked in brown.
Cluster
Acc
Label
Num
Len
Atoms the explanations (ordered by frequency)
99.1%
1,763
not(290) bad(2333) no( 185) mediocre(153) bland( never
99.2
643
Neg
8.82%
again( worst( 19) ok(1 disappointing( 15) terrible(107)
99.2%
2,730
great(667) delicious(508) best(330) love(294) fresh(285) tasty(255)
2
99.2
584
Pos
13.7%
definitely(254) friendly(2322) perfect(199 amazing( (aazzing(198) favorite( (194)
98.6%
2,686
great(793) friendly(329) always(3 15) best(304) 1ove(295) fun(223)
3
98.6
548
Pos
13.4%
definitely(218) helpful( (63) awesome( (1 (59) amazing( (36) vegas( (17)
57.8%
848
NULL(1738) great( 133) not(67) best (3) service(39) love(34)
4
93.2
119
Pos
4.24%
friendly(29) 266d(26) awesome(22) (22) f2s)(22) overpriced(20)
58.1%
62
NULL((4)) 113 un (1) (ine(9) service(8) pas(8) d8r(8) die(7)
5
83.9
682
Neg
0.31 %
und(7) um(6) den(5) pour(5) de(5) das(4) prix(4) je(4) zu(3) im(3)
Accuracy: 0.8994 0,9053
Example 1
Confidence: 0.8974 H 0.7989
This Jubby (lay geou for ore ching, repocious pricing! food Only eastelessons unesceptiona, they are charging 25% more at los Veras Restourant than they at all there other locations Go play the slots, you will have a better return on your money. (Lučel: Nogi
Generated ator change
20
Explanation: vegas, locations, bobby (Pred: 0. 0.7228) - tasteless, bobby, locations (Pred: O. 0.2960)
Example 2
-10
NULL
worth
huge
nice
stip
cool
must
Late a siice of pizz, then all remember's putting my head down and I was KO'd Oh Veras.. (labnl: Pos! Explanation: vegas, pizza (Pred: 0,5842) H+ putting, remember (Pred: O 0.3666)
Figure 4: Steering the model without re- re-training. SELOR allows users to exclude noisy features from explanations during testing, which may simultaneously improve the explanation quality and prediction accuracy. This figure shows the performance before and after removing "vegas".
happens when the model does not have enough evidence. For example, reviews in cluster were short (average length of 119 words) and contained an overwhelming number of NULL atoms (on average 2 per explanation). Third, cluster contained 13.4% samples with positive sentiment, and its training accuracy (98.6%) is higher than cluster 4. However, the cluster often included "vegas" in the explanation, which does not seem directly related to sentiment classification. Fourth, clusters 1 and 2 have reasonable atoms, which seem consistent with high training accuracy (99.2%)
SELOR allows us to steer the model directly. For example, after identifying the potentially noisy feature like "vegas", we can prevent the model from including the term by removing it from the candidate atom list C.. This type of refinement can be easily achieved during testing, unlike the efforts-taking dataset calibration model retraining. Fig. shows the performance change of the 169 samples that previously included "vegas" in their explanations. The histogram shows which atoms that are generated more after removing "vegas". The model sometimes relies on similar atoms such as "las" or does not find a good candidate (e.g.. choosing NULL), which may lead to decreased confidence. However, the chance of including more meaningful atoms also increases (e.g.. "worth" in the histogram, and "tasteless" in Example 1). One may also verify assumptions by checking the samples whose prediction score changes. For instance, after removing "vegas", the model can no longer predict Example 2 correctly. The example contains no obvious indication of sentiment, and "vegas" may be the most helpful feature. This contradicts our previous assumption that "vegas" seems not critical for sentiment classification. Instead it can provide new insights and guidance for further improvement (c.g., punishing "vegas" with a soft prior instead of directly removing it).
Explanation stability and sensitivity analysis. We discuss the stability of our explanations in Appendix C.6 Our quantitative experiment demonstrates that the explanations generated in different runs are consistent. We also present a case study in that SELOR gives similar explanations for similar inputs. Moreover, we discuss user study results that the human precision of the explanations is good across different parammtrr settings in Appendix C.5.4
3.4 Efficiency
Table 5) shows that post-hoc explanation methods like LIME and Anchor require a longer time to generate an explanation. RCN has the largest complexity among the self-explaining methods since

---

Table 5: Time costs in seconds on Yelp (BERT) and Adult.
Consequent
Deep model training (1 epuch)
Explanation generation sample)
estimator training
Base
SENN
RCN
SELOR
LIME
Anchor
SENN
RCN
SELOR
Yelp
2041.4
571.9
224.5
503.7
665.6
55.0
2854.2
0.037
0.071
0.055
Adult
1502.8
12.8
9.1
953.9
98.2
2.5
1.18
0.02
0.17
0.015
it enumerates all possible rules and combines them with soft attention. To alleviate this problem, RCN uses predefined rule set; hence, its efficiency becomes dependent on the size and quality of the rule set. In contrast, SELOR is trained within acceptable time even for large solution space, and humans only need to define the types of atoms and logical connectives. Our model generated each explanation with a linear complexity with length L, while RCN goes over all possible rules and has exponential complexity with L. Our method required additional time for the neural consequent estimator, taking 35 minutes on Yelp and 25 minutes on Adult. This step is only required once for each dataset and hence is acceptable. The consequent estimator can also be reused.
Conclusion and Future Work
This work presented a new framework, SELOR, which incorporates self-explanatory capabilities into a deep model to provide high human precision by explaining logic rules while also maintaining high prediction performance. Our method does not require predefined rule sets and can be learned in differentiable way. Extensive tests involving human evaluation show that our method achieves high prediction performance and human precision while being resistant to noisy labels. Although our method brings substantial advantages, there remain multiple aspects for improvement in the future:
Stability. desirable property for self-explaining models is stability, which requires that similar inputs lead to similar explanations. Unlike SENN ([]] which proposes robustness loss to ensure stable explanations against adversarial inputs, SELOR does not employ such a constraint and cannot guarantee the stability of explanations for inputs with similar raw features. However, our framework theoretically ensures stability is modeled in the selected feature space (see Appendix [.6 for more details), which is partially evaluated by case study in Appendix C.6
Applicability. While we explored text and tabular data, our model is applicable to other data types like images and graphs. We can treat cluster of images or superpixels an atom [II or extract atoms with CAV (Concept Activation Vector), feature that indicates the concept of humans (e.g., striped, red) (3 End-to-end feature learning is possible in our framework if the number of candidate atoms is small (c.g., around 100 object classes or concepts (35).
Level of insight. SELOR cannot explicitly model higher-level properties of atoms (e.g., learn that "awesome" is positive sentiment word and make a rule based on positive sentiment word) since we do not directly consider predicates. We can only find rules constructed with bottom-leve atoms instead of summarizing important high-level patterns, which also leads low coverage of rules (e.g., rule "awesome AND tasty" => positive sentiment only covers 0.37% of the input instances). If re-designed as the first-order logic, the model may directly find high-level patterns such as "a negation word AND positive sentiment word"' => negative sentiment, instead of listing many specific rules such as "not great" => negative sentiment and "no good" => negative sentiment. This could save human cognitive budget and improve the reasoning capability of deep models. Morcover, we may automatically compose high-level concepts such as "strong positive phrase" and build rules with them. The concept "strong positive phrase" may be composed by detecting two consecutive positive sentiment words (" ("amazingly comfortable" and "perfectly enjoyable") with predicate invention in 19
Acknowledgments and Disclosure of Funding
We thank Fangzhao Wu, Sundong Kim, and Eunji Lee for their insightful feedback on our work. We appreciate the reviewers of this paper for their valuable suggestions that improved the paper significantly. This research was supported by Microsoft Research Asia, the Institute for Basic Science (IBS-R029-C2) in Korea, and the Potential Individuals Global Training Program (2021-0-00 (001--000666) by the Ministry of Science and ICT in Korea.
10

---

References
[1] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should ''rust You?" explaining the predictions of any classifier. In KDD, 2016.
[2] Sebastian Thrun. Extracting rules from artificial neural networks with sepress In NewrlPS, 1994.
[3] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Nothing else matters: Model-agnostic explanations by identifying prediction invariance. stat, 2016.
[4] Tao Lei, Regina Barzilay, and Tommi Jaakkola. Rationalizing neural predictions. In EMNLP, 2016.
[5] Dimitrios Alikaniotis, Helen Yannakoudakis, and Marek Rei. Automatic text scoring using neural networks. In ACI., 2016.
[6] Hendrik Strobelt, Sebastian Gehrmann, Hanspeter Pfister, and Alexander M Rush. Lstmvis: tool for visual analysis of hidden state dynamics in recurrent neural networks. IEEE TVCG, 2017.
[7] W James Murdoch, Peter J Liu, and Bin Yu. Beyond word importance: Contextual decomposi- tion extract interactions from Istms. In ICLR, 2018.
[8] Georgina Peake and Jun Wang. Explanation mining: Post hoc interpretability of latent factor models for recommendation systems. In KDD, 2018.
[9] Jian Liang, Bing Bai, Yuren Cao, Kun Bai, and Fei Wang. Adversarial infidelity learning for model interpretation. In KDD, 2020.
[10] Jingyue Gao, Xiting Wang. Yasha Wang, Yulan Yan, and Xing Xie. Learning groupwise explanations for black-box models. In IJCAI, 2021.
[11] David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. In NeurIPS, 2018.
[12] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 2019.
[13] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In NeurIPS, 2017.
[14] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: High-precision model- agnostic explanations. AAAI, 2018
[15] Chaoyu Guan, Xiting Wang. Quanshi Zhang, Runjin Chen, Di He, and Xing Xic. Towards a deep and unified understanding of deep neural models in nlp. In ICML, 2019.
[16] Sungsoo Ray Hong, Jessica Hullman, and Enrico Bertini. Human factors in model interpretabil- ity: Industry practices, challenges, and needs. PACM HCI, 2020.
[17] Benjamin Letham, Cynthia Rudin, Tyler H McCormick, and David Madigan. Interpretable classifiers using rules and bayesian analysis: Building a better stroke prediction model. AOAS, 2015.
[18] Hongyu Yang, Cynthia Rudin, and Margo Seltzer. Scalable bayesian rule lists. In ICML, 2017.
[19] Richard Evans and Edward Grefenstette. Learning explanatory rules from noisy data. Journal of Artificial Intelligence Research, 2018.
[20] Elaine Angelino, Nicholas Larus-Stone, Daniel Alabi, Margo Seltzer, and Cynthia Rudin. Learning certifiably optimal rule lists. In KDD, 2017.
[21] Yao Ming, Panpan Xu, Huamin Qu, and Liu Ren. Interpretable and steerable sequence learning via prototypes. In KDD, 2019.
[22] Zhongxia Chen, Xiting Wang, Xing Xie, Mehul Parsana, Akshay Soni, Xiang Ao, and Enhong Chen. Towards explainable conversational recommendation. In IJCAI, 2020.
[23] Todd Kulesza, Margaret Burnett, Weng-Keen Wong, and Simone Stumpf. Principles of explana- tory debugging to personalize interactive machine learning. In IUI, 2015.
11

---

[24] Patrick Schramowski. Wolfgang Stammer, Stefano Teso, Anna Brugger, Franziska Herbert, Xiaoting Shao, Hans-Georg Luigs, Anne-Katrin Mahlein, and Kristian Kersting. Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nature Machine Intelligence, 2020.
[25]
Piyawat Lertvittayakumjorn Lucia Specia, and Francesca Toni, Find: Human-in-the-loop debugging deep text classifiers. In EMNLP, 2020.
[26] Gabriele Ciravegna, Francesco Giannini, Marco Gori, Marco Maggini, and Stefano Melacci. Human-driven fol explanations of deep learning. IJCAI, 2021.
[27] Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting, Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations. In CVPR, 2021.
[28] Andrea Bontempelli, Fausto Giunchiglia, Andrea Passerini, and Stefano Teso. Toward a unified framework for debugging gray-box models. arXiv preprint arXiv:2
2021.
[29] Luc De Raedt, Sebastijan Dumaneic, Robin Manhaeve, and Giuseppe Marra. From statistical relational to neuro-symbolic artificial intelligence. In IJCAI, 2020.
[30] Lazar Valkov, Dipak Chaudhari, Akash Srivastava, Charles Sutton, and Swarat Chaudhuri. Houdini: lifelong learning as program synthesis. In NeurIPS, 2018
[31] Kevin Ellis, Lucas Morales, Mathias Sablé-Meyer, Armando Solar-Lezama, and Josh Tenen- baum. Learning libraries of subroutines for neurally-guided bayesian program induction. In NeurIPS, 2018.
[32] Ashwin Kalyan, Abhishek Mohta, Oleksandr Polozov, Dhruv Batra, Prateek Jain, and Sumit Gulwani. Neural-guided deductive search for real-time program synthesis from examples. In ICLR, 2018.
[33] Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Josh Tenenbaum. Learning to infer graphics programs from hand-drawn images. In NeurIPS, 2018
[34] Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In ICLR, 2019.
[35] Dongran Yu, Bo Yang, Qianhao Wei, Anchen Li, and Shirui Pan. A probabilistic graphical model based on neural-symbolic reasoning for visual relationship detection. In CVPR, 2022.
[36] Xiting Wang, Kunpeng Liu, Dongjie Wang, Le Wu, Yanjie Fu, and Xing Xie. Multi-level recommendation reasoning over knowledge graphs with reinforcement learning. In WebConf, 2022.
[37] Fan Yang, Zhilin Yang, and William W Cohen. Differentiable learning of logical rules for knowledge base reasoning. In NeurIPS, 2017.
[38] Kangzhi Zhao, Xiting Wang, Yuren Zhang, Li Zhao, Zheng Liu, Chunxiao Xing, and Xing Xie. Leveraging demonstrations for reinforcement recommendation reasoning over knowledge graphs. In SIGIR, 2020.
[39] Yuzuru Okajima and Kunihiko Sadamasa. Deep neural networks constrained by decision rules. In AAAI, 2019.
[40] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with sombbel-softmx. stat, 2017.
[41] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi- task learning using uncertainty to weigh losses for scene geometry and semanties. In CVPR, 2018
[42] Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoderr approaches. In SSST, 2014.
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
[44] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NeurIPS, 2015.
[45] Open Data Science (ODS. ai). Kaggle clickbait news detection. https: : www. http://www.kggl.e... com/ c/clickbait news- wwwpstt-nssseeeeeeeeeeeee detection 2020.
12

---

[46] Dheeru Dua and Casey Graf. UCI machine learning repository. http://archive. ics uci
edu/m1 2017.
[47]
Takaya Saito and Mare Rehmsmeier. The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets. PloS one, 10(3):e01 18432, 2015.
[48] Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
[49] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /692, 2019.
[50] Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In ICLR, 2019.
[51] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. Mixup: Beyond empirical risk minimization. In ICLR, 2018.
[52] Speechocean.
spw.choceccc.....ee .cm.ioocccaccccoomcooo..
[53] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In ICML, 2018.
[54] Eyal Peer, David Rothschild, Andrew Gordon, Zak Evernden, and Ekaterina Damer. Data quality of platforms and panels for online behavioral research. Behavior Research Methods, 14444:1643-1662, 2022.
[55] Beth Sagar-Fenton and Lizzy McNeill. How many words you need to speak language. BBC, 2018.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sec.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Sec.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes) See Sec. 2.1
Sec. 2.2 Sec. 2.3 Appendix [B.3
(b) Did you include complete proofs of all theoretical results? Section and Appendix B.3
[Yes] See Section 2.2
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material
URL)? [Yes] Codes
released at Github /https:/ggithub.c (https://github..o//rrrrul com//rcchon/19///LLR
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Appendix C.T and Appendix C.2
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See Vableeaand Table
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix C.2
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Sec. 3.1 and
Appendix C.2
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [Yes] See checklist 3-(a).
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes] See Sec. [3.3] Appendix Appendix C.4 and additionally attached guideline files and screenshots.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent participant compensation? [Yes] See Appendix A and Appendix C.4
14


Reviews:
[{"review_id": "ajryj3fbfQ0", "reviewer": "Reviewer_pby1", "summary": "The paper proposes an interpretable-by-design framework for deep learning, leveraging logical rules in the form of propositional implications to explain model’s predictions. \nSpecifically, the body of the rule (alternatively called antecedent in the article) is predicted by a carefully chosen deep learning architecture, whereas the head (alternatively called consequent) consists of the result of the implication rule, namely the label prediction. The deep learning component includes (i) a backbone encoding network to learn a representation from the raw input data and (ii) an autoregressive model transforming the latent representation into the predicting atoms (Boolean features) composing the body of the logical rule.\nThe training is performed in two stages. In particular, an auxiliary network is first trained to mimic the prediction and the confidence of random implication rules. The learnt network is then used as a prediction head for the above mentioned deep learning component to minimise the prediction error together with additional user specific constraints in an end-to-end differentiable fashion.\n\nThe proposed approach is compared against  two self-explainable and two post-hoc explanation solutions on two textual and one tabular datasets. The results show that the proposed model is able to outperform previous approaches in terms of explainability as measured by human precision, while retaining state-of-the-art predictive performance.\n", "questions": "1. Can you please elaborate on the novelty of the proposed framework in relation to neuro-symbolic approaches?\n2. What are the soft constraints and loss used in the experiments?\n", "limitations": "1. Are there any limitations on the chosen class of logic? How would this influence the human precision?\n\nMINOR MISTAKE\n\nPlease replace $p(\\alpha|b)$ with $p(b|\\alpha)$ in Eq. (1), line 115, line 131, multiple occurrences in line 132, line 133, line 142 and Eq. (6). Also, modify the generative process in Figure 1 (c) to take into account the correct dependence between $\\alpha$ and $b$\n\n-------------------------------------------------- \nPOST - REBUTTAL\n\nDuring the rebuttal, the authors have properly addressed all my questions. Also, they have provided concrete actions to take into account the weaknesses of the manuscript (i.e. missing related work on neuro-symbolic learning, discussion on language expressiveness). Overall, the work provides an interesting contribution for the application of neuro-symbolic learning in the area of explainable AI. I'm happy to recommend for a weak accept.\n", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is clearly written and the use of logical rules is well motivated.\n- The considered problem is relevant and timely.\n- Experiments show the superiority of the proposed solution over the baselines especially in terms of explainability. I liked the analysis of the predictions in Section 3.3, as providing convincing evidence on the utility of logical rules. This is an original part of the work.\n\n**Weaknesses**\n\n- The paper overlooks an important body of research in the emerging field of neuro-symbolic learning. Indeed, this is not the first time that rules are learnt in symbiosis with deep learning, see for example [1] and [2]. Experimental comparisons would be required against these strategies, but I realise that at this stage this could not be feasible. Authors should at least cite and discuss works in neuro-symbolic learning and highlight the point of novelties of their proposed solution.\n- The work focuses on learning logical rules in the form of definite clause logic (a subclass of propositional logic), thus inheriting its limitations in terms of expressivity and also explainability. A discussion on these limitations is required.\n\n[1] Learning Explanatory Rule From Noisy Data. JAIR 2018\n[2] DeepProbLog: Neural Probabilistic Logic Programming. NeurIPS 2018\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WZm_UfWQWkT", "reviewer": "Reviewer_YUk5", "summary": "The paper describes a self-explainable model in which explanations consist of logical rules consistent with the data and decisions and decisions are consistent with these explanations (at least locally).  Both elements are implemented using neural networks.  The model also supports prior knowledge on admissible rules.  Results on a selection of textual and tabular data sets illustrate the performance and robustness of the proposed approach compared to black-box models, post-hoc explainers, and self-explainable alternatives.", "questions": "- See Q1 and Q2 above.\n\n- Are the results reported for the human study statistically significant?\n\n- l 279: \"The participants are allowed to choose multiple best explanations only if the chosen ones are the same\"  Why?  Can't two explanations be equally good (or bad)?", "limitations": "Partially, in the conclusion.  Other limitations include lack of generalization of produced explanations across inputs and lack of self-consistency of explanations at different points.  I asked the authors to comment on these in my questions above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality**: The proposed approach combines familiar elements into a novel architecture.\n\n**Clarity**: English is good, but the text - and especially the description of the method - is partially opaque.  Notation is not entirely clear.  This makes it hard to assess the properties of the proposed model.\n\nIn particular, Eq. 1 is confusing as it looks like the explanations are being marginalized over, which would mean that any one prediction could be the result of potentially exponentially many explanations (all those appearing in the sum and having probability > 0).  This would conflict with the goal of building an interpretable model.  To the best of my understanding, however, this is not the case, as both p(y|alpha) and p(alpha|y) are essentially discriminative, so no real marginalization takes place at inference time.  This is not deal-breaking, but it is an issue.\n\n**Quality**: At a high-level, the proposed model is reasonable.  One issue I have with it is that it looks very complex - much more so than SENNs or concept bottleneck models.\n\nQ1. Another element of confusion is the relationship between rules for different inputs:  is it possible for close inputs to have arbitrarily different explanations?  In other words, what is the ``radius of validity'' of the local explanations produced by the model?\n\nQ2. More generally, is it possible that the rules predicted by the model are inconsistent with each other, i.e., that the rule predicted for an input x and that *should* apply to input x' is inconsistent with the rule predicted for input x'?\n\n**UPDATE**: Q1 AND Q2 HAVE BEEN ANSWERED BY THE AUTHORS IN THEIR REBUTTAL.\n\nLack of explanation generalization across points would complicate global interpretability (and woese self consistency) of the model.\n\nThe numerical experiments look okay, with a reasonable selection of data sets and a number of competitors.  One issue is that the human evaluation is *very* small scale - it only involves three human subjects.  Platforms like Prolific however make it quite easy to setup larger scale experiments at a very reasonable cost, so the choice of restricting the experiment to only three people is puzzling.  I am not sure that the numbers reported have statistical significance.  See questions below.\n\n**UPDATE**: THE AUTHORS HAVE REPORTED MORE EXTENSIVE EXPERIMENTS IN THEIR REBUTTAL.\n\nAs for related work, the authors write: \"Collecting ground-truth labels of human decision processes are quite expensive, if not impossible.\"  But there exist a number of ML approaches that do exactly that - using post-hoc and self-explainable models.  See for instance:\n\n  Kulesza et al. \"Principles of Explanatory Debugging to Personalize Interactive Machine Learning\" 2015.\n\n  Schramowski et al. \"Making deep neural networks right for the right scientific reasons by interacting with their explanations\" 2020.\n\n  Lertvittayakumjorn et al. \"FIND: Human-in-the-loop Debugging Deep Text Classifiers\" 2020.\n\n  Ciravegna et al. \"Human-driven FOL explanations of deep learning\" 2020.\n\n  Stammer et al. \"Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanations\" 2021.\n\n  Bontempelli et al. \"Toward a Unified Framework for Debugging Gray-box Models\" 2022.\n\njust to mention a few.  I would expect there to be also works on acquiring rules from human annotators in the context of logical and neuro-symbolic models, but I cannot provide pointers.  I would have expected better coverage of works on debugging neural networks and self-explainable models.\n\n\n**Significance**: The model is of interest to research in XAI and especially for self-explainable models.\n\nOther comments:\n\n- l 121: \"The deep model f is used to encode the sample x\"  Wasn't f the complete model?  Also, at this point it is not clear what\n  a \"recursive generation layer\" is.  I'd suggest to move lines 121-123 to a later section.\n\n- l 125: \"Our framework for deep logic rule reasoning bears an interesting resemblance to the human mind...\"  All concept-based models and many neuro-symbolic models do.  But it's only a vague analogy and I'm not sure what it adds.  Moreover, notice that post-hoc models bear a striking resemblance to models of the mind in which decisions are taken subconsciously and then interpreted after the fact by an argumentative interpreter (see books by Michael Gazzaniga).  There is no conclusive evidence of how our minds really work in this regard.  Also, is really the subconscious mind \"often thought to be more powerful\"?\n\n- l 183: square brackets, missing comma.\n\n- l 191: organizes\n\n- Eq 3.  The equality symbols should be replaced by \\approx, especially the second one (as p is replaced by \\hat{p}).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HJcjHjKDNzg", "reviewer": "Reviewer_oPY7", "summary": "This paper addresses the task of explaining deep models with logic form and proposes an explanation model applicable to text and tabular data, which generates latent antecedent and then makes a prediction based on the generated antecedent.\nThe experiments on review text and tabular datasets show that the proposed model outperforms the comparing methods in terms of classification performance and explainability.", "questions": "- How do you make the candidates of atoms? Focusing on the word 'awesome', many candidates of atoms associated with 'awesome' can be made, such as 'awesome$\\geq 1$', 'awesome$\\geq 2$', and so on.\n- Is there a possibility that atoms associated with the same word, such as 'awesome$\\geq 1$' and 'awesome$\\geq 2$' are generated for the explanation in an instance? ", "limitations": "NA", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The proposed model looks well designed, and the construction of each component of the proposed model, such as human prior, consequent estimation, and deep antecedent generation are reasonable.\n-  As shown in Table 2, regardless of making the unexplainable models, such as BERT explainable, the proposed model can remain high classification performance.\n- The paper is well written, and the performances of the proposed model in terms of classification performance and explainability are well analyzed with both numerical experiments and user study.\n\n## Weaknesses\nI don’t have any weaknesses to point out.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4eeFDrLVdHl", "reviewer": "Reviewer_zwRN", "summary": "This paper proposes a self-explaining model generating logic rules as explanations. It aims to learn explanations that (1) make accurate prediction, (2) are coherent with human's prior belief. The authors provides several experimental evidences supporting their claims using three benchmark datasets (two text datasets and a tabular dataset). The major contribution is on the quality of explanation where their method achieved a good performance compared to the others. But the evaluation of explanation quality is limited to human precision. It seems like the authors perform several other experiments (C.4.4, C.4.3) that are potentially related with explanation quality, but didn't discuss their experimental findings or relate with the explanation quality. I expect the paper can be improved during the rebuttal period, and am willing to adjust my score if it addresses several concerns listed below.", "questions": "### Majors\n(1)  Figure 1c: What are the assumptions you made for the generative process? For example, see the following questions:\n- Do you assume that the antecedents (explanations) and corresponding 1st order logic rules are sufficient for making accurate prediction? \n- Do you assume that given $\\alpha$, $\\mathbf{x}$ and $b$ are independent? What is rationale for this?\n\n(2) Provide AUCs for Table 2. \n\n(3) Provide details about the \"Base\" model. It is usually expected model performance degrades when an explanation mechanism is added to the model. It is important to compare with reasonably well designed baseline models to precisely evaluate classification performance of explainable models. (Btw, it is sometimes good enough to have comparable prediction performance with prediction-only baseline models, if it can provide substantially good quality of explanations in different aspects.)\n\n(4) The following questions are related with the quality of explanations. \n- Are there any factors (e.g., rule length, number of atoms) impacting explanation quality? Do you expect trade-off between prediction performance and quality of explanation? \n- How unique the explanations are for an instance (sample)? (i.e., do the models trained on two different random seeds generate the same or similar logic rules for an instance?) Is there any factor controlling the uniqueness? -- I used the term \"uniqueness\" instead of \"robustness\" because the authors already used the term \"robustness\" for another purpose in the paper.\n\n### Minors\n- Provide standard error of the five runs for the classification performance in Table 2. (Figure 8 is unnecessary).\n- Fix typo in the equation (1)\n- No content in C.4.2 ", "limitations": "First, it should discuss how likely the assumptions made for Figure 1c are correct in practice, and what we need to expect when they're not satisfied (see Q1). Second, it should discuss that the explanation quality can depend on several factors (see Q4), and careful designed experiments are required to get good enough explanation. \n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Quality\nStrength\n- If we assume Figure 1c is well designed process, the model component design itself has a good rationale. The model is composed of three parts (human prior, consequent estimation, antecedent generation) with corresponding loss terms. Each part takes a role to achieve the coherence, prediction, and generation. \n- The authors' main claims (prediction accuracy, and human precision) are supported by various experiments including human evaluation. \n\nWeakness\n- The generative process (Figure 1c) should be provided with a good rationale, or in-depth discussion why it makes sense. [See Q1] \n- Explanation quality should be evaluated in different aspects. (It is important especially when the prediction performance are not substantially good enough.) [See Q4]\n\n### Clarity\nWeakness\n- Lack of details about important experimental settings: \"Base\" model [See Q3]  \n\n\n### Significance\nStrength\n- The authors' main claims (prediction accuracy, and human precision) are supported by various experiments including human evaluation. Especially, it achieves a good performance on the human precision evaluation.  \n- In addition to the main experimental results, the paper provides several interesting experimental observations about relationships between model interpretability and robustness, and potentiality for debugging and refinement using the model explanations. Although those are not based on in-depth or wide-breadth experiments (probably because this is not directly related with the authors' main claims), those observations can fuel interesting future works in this direction. \n\nWeakness\n- It seems like there are no significant improvement in the classification performance (Table 2). \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept