Interpretable Concept-Based Memory Reasoning
Authors: David Debot
Pietro Barbiero
Francesco Giannini
Gabriele Ciravegna
Michelangelo Diligenti
Giuseppe Marra
Venue: NeurIPS 2024 poster (2024)
Keywords: Concept-based models
explainable AI
neurosymbolic
Primary area: interpretability_and_explainability
TL;DR: We introduce Concept-based Memory Reasoner (CMR), a novel CBM that uses a neural selection mechanism over learnable logic rules and symbolic evaluation to enable human-understandable and provably-verifiable task predictions.

Abstract: The lack of transparency in the decision-making processes of deep learning systems presents a significant challenge in modern artificial intelligence (AI), as it impairs users’ ability to rely on and verify these systems. To address this challenge, Concept Bottleneck Models (CBMs) have made significant progress by incorporating human-interpretable concepts into deep learning architectures. This approach allows predictions to be traced back to specific concept patterns that users can understand and potentially intervene on. However, existing CBMs’ task predictors are not fully interpretable, preventing a thorough analysis and any form of formal verification of their decision-making process prior to deployment, thereby raising significant reliability concerns. To bridge this gap, we introduce Concept-based Memory Reasoner (CMR), a novel CBM designed to provide a human-understandable and provably-verifiable task prediction process. Our approach is to model each task prediction as a neural selection mechanism over a memory of learnable logic rules, followed by a symbolic evaluation of the selected rule. The presence of an explicit memory and the symbolic evaluation allow domain experts to inspect and formally verify the validity of certain global properties of interest for the task prediction process. Experimental results demonstrate that CMR achieves better accuracy-interpretability trade-offs to state-of-the-art CBMs, discovers logic rules consistent with ground truths, allows for rule interventions, and allows pre-deployment verification.

Interpretable Concept-Based Memory Reasoning
David Debot
Pietro Barbiero Universita' della Svizzera Italiana University of Cambridge
KU Leuven
david. debiiddeeet@@kleeeeer be
barbiero@tutanota.. com
Francesco Giannini
Gabriele Ciravegna
Scuola Normale Superiore
DAUIN, Politecnico di Torino gabriele. cwravegna@pollto..... it
francesco. cetteesco.giannnnn giannini@sns. it
Michelangelo Diligenti
Giuseppe Marra
University of Siena
KU Leuven giuseppe. mar be
mi michelangelo. hww.:aagee.o..iiiigeeeeeieiiao Suniiiiiti
Abstract
The lack of transparency in the decision maaoon processes of deep learning sys- tems presents significant challenge in modern artificial intelligence (AI), as it impairs users' ability rely and verify these systems. To address this challenge, Concept-Basse Models (CBMs) have made significant progress by incorporating human-inters interrraantee concepts into deep learning architectures. This approach allows predictions to be traced back to specific concept patterns that users can understand and potentially intervene However, existing CBMs' task predic- tors are not fully interpretable, preventing a thorough analysis and any form of formal verification of their decision-making process prior deployment thereby raising significant reliability concerns. To bridge this gap, we introduce Concept- based Memory Reasoner (CMR), novel CBM designed to provide human- understandable and provably-verifiable task prediction process. Our approach is model each task prediction as a neural selection mechanism over memory of learnable logic rules, followed by a symbolic evaluation of the selected rule. The presence of an explicit memory and the symbolic evaluation allow domain experts to inspect and formally verify the validity of certain global properties of interest for the task prediction process. Experimental results demonstrate that trade-offs to state-of-the-ae CBMs,
CMR achieves better eecuracy-interpretabble discovers logic rules consistent with ground truths, allows for rule interventions, and allows pre-deployment verification.
Introduction
The opaque decision process of deep learning (DL) systems represents one of the most fundamental
problems in modern artificial intelligence (AI). For this reason, eXplainable AI (XAI) [133] is currently one of the most active research areas AI. Among XAI techniques, Concept-Band Based Models (CBMs) [48 represented a significant innovation that made DL models explainable-by- by-design by introducing layer of interppeaateeeee concepts within DL architectures. CBMss consist of at least two functions: a concept encoder, which maps low-level raw features (c.g. image's 's pixels) to high-level interpretable concepts (e.g. "red" and "round"), and a task predictor, which uses the learned concepts to solve downstream task (e.g. "apple")) This way, each task prediction can be traced back
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

to specific pattern of concepts, thus allowing CBMs to provide explanations in terms of high-level interpretable concepts (e.g. concepts "red" and "round" were both active when the model classified an image as "apple") rather than low-level raw features (e.g. there were 100 red pixels when the model classified an image as "apple"). In other terms, a CBM's task predictor allows understanding what the model sees in given input rather than simply pointing to where it is looking (9]
However, state-of-the-ar CBMs' task predictors are either unable to solve complex tasks (e.g. linear layers), non-differentiable (e.g. decision trees), or black-box neural networks. CBMs employing black-box task predictors are still considered locally interpretable, as concept interventions allow humans to understand how concepts influence predictions for individual input examples. However, they lack global interpretability: the human cannot interpret the model's ' global behaviour, i.e. on any possible instance. This prevents proper understanding of the model's working, as well as any chance of formally verifying the task predictor decision-making process prior to deployment, thus raising significant concerns in practical applications. As result, a knowledge gap persists in the existing literature: the definition of a CBM with a task predictor whose behaviour can be inspected, verified, and potentially intervened upon before the deployment of the system.
To address this gap, we propose Concept-based Memory Reasoner (CMR), a new CBM where the behaviour and explanations can be inspected and verified before the model is deployed. CMR's s task predictor offers global interpretability as it utilizes a differentiable memory of learnable logic rules, making all potential decision rules transparent to humans. Additionally, CMR avoids the concept bottleneck that often limits the accuracy of interpretable models when compared to black-box approaches. Our key innovation lies in an attention mechanism that dynamically selects relevant rule from the memory, which CMR uses to accurately map concepts to downstream classes.
CMR = neural selection over a set of human-understandable ppecisio-makiing processes
Accuracy
Interpretability
We call this paradigm Neural Interpretable Reasoning (NIR), which involves neurally generating (i.c. selecting from memory) an interpretable model (i.c. logie rule) and symbolically executing Once learned, the memory of logic rules can be interpreted as a disjunctive theory, which can be used for explaining and automatic verification. This verification can take place before the model is deployed and, thus, for any possible input the model will face at deployment time. The concept-based nature of CMR allows the automatic verification of properties that are expressed in terms of high-level suman-understandable concepts (e.g. "never predict class 'apple" when the concept 'blue' is active") rather than raw features (e.g. "never predict class 'apple' when there are less than ten red pixels").
Our experimental results show that CMR: (i) improves over the sccurac-inteepretaaiitt performances of state-of-the- CBMs, (ii) discovers logic rules matching ground truths, (iii) enables rule inter- ventions beyond concept interventions, and (iv) allows verifying properties for their predictions and explanations before deployment. Our code is available at athhtps://gii //gt:/u///h/tb///. gitb....c.. com/ damiddddooooo/CM
2 Preliminary
Concept Bottleneck Models (CBNMs) 4 10 are functions composed of (i) a concept encoder Rd (e.g. an image) set of ne concepts
g: X + C mapping each entity x E X (e.g. "red", "round"", and (ii) a task predictor : C + Y mapping concepts to the class E Y (e.g. "apple"") representing a downstream task. For simplicity, in this paper, a single task class is discussed, as multiple tasks can be encoded by instantiating multiple task predictors. When sigmoid activations are used for concepts and task predictions, we can consider g and as parameterizing a Bernoulii distribution of truth assignments to propositional boolean concepts and tasks. For example, (red(x) 0.8 means that there is an 80% probability for the proposition "x red" to be true. During training, concept and class predictions (c. y) are aligned with ground-trutt -tuut labels (ć. g). This architecture and training allows CBNMs to provide explanations for class predictions indicating the presence or absence of concepts. Another main advantage of these models is that, at test time, human experts may also intervene on mispredicted concept labels to improve CBNMs' task performance and extract counterfactual explanations 4 [1]. However, the task prediction ʃ is still often a black-box model to guarantee high performances, thus not providing any insight into which concepts are used and how they are composed to reach the final prediction.
C
C
2

---

3 Model
In this section, we introduce Concept-based Memory Reasoner (CMR), the first concept-based model that is globally interpretable, provably verifiable and a wniversal binary classifier. CMR consists of three main components: a concept encoder, a rule selector and a task predictor. CMR's s task prediction process differs significantly from traditional CBMs. It operates transparently by (1) selecting logic rule from a set of jointly-eaamndd rules, and (2) symbolically evaluating the chosen rule on the concept predictions. This unique approach enables CMR not only provide explanations by tracing class predictions back to concept activations, but also to explain which concepts are utilized and how they interact make a task prediction. Moreover, the set of learned rules remains accessible throughout the learning process, allowing users to analyse the model's 's behaviour and automatically verify whether some desired properties are being fulfilled at any time. The logical interpretation of CMR's 's task predictor, combined with its provably verifiable behaviour, distinguishes it sharply from existing CBMs' task predictors.
3.1 Probabilistic graphical model
In Figure D show the probabilistic graphical model of CMR. There are four variables, three of which are standard in (discriminative) CBMs: the observed input x E X, the concepts encoding c E C and the task prediction the rule r E [P,N,I 1 rule is a conjunction the concept set, like e₁ -3- conjunction is uniquely identified when, for each concept C; we know whether, in the rule, the concept irrelevant (I), positive (P) negative (N). We call r; NN, 7} the role of the i- th concept in rule r. For example, given the three concepts C1, 02: the conjunction C1 A cc can be represented Tı P. r2 1, N, since the role of cı is positive (P), the role of is irrelevant (I) and the role of C3 negative (N).
f0, 1}. CMR adds additional variable:
y
Figure Probabilistic graphi-
cal model of CMR
This probabilistic graphical model encodes the joint conditional distribution p(u. r, c/z) and factorizes
ply, cla) p(ole,r)p(rlz)pp(H
and consists of the following components:
p(c|ær is the concept encoder. For concept bottleneck encoders it is simply the product of ne independent Bernoulli distributions p(q/æ), whose logits are parameterized by some neural network encoder 9: X R.
[T
p(rlæ) is the rule selector, described Section 3.1.1 Given an input x, p(rlar models the uncertainty over which conjunctive rule must be used.
p(yle,r) r) is the task predictor, which will be described in Section 3.1.2 Given rule r p(r(z) and an assignment of truth values ~ p(c[æ=) the concepts, the task predictor evaluates the rule on the concepts. In all the cases described in this paper, p(yle, r) is degenerate deterministic distribution.
3.1.1 Rule selector
We model the rule selector p(rlæ) mixture of nn rule distributions. The selector "selects" rule from set of rule distributions (i.e. the components of the mixture), and we call this set the rulebook. The rulebook is jointly learned with the rest of CMR and can be inspected and verified at every stage of the learning. Architecturally, the selection is akin to an attention mechanism over
differentiable memory [12].
To this end, let s E [1, C N be the indicator of the selected component of the mixture, then:
p(r(z) = Σ p(rls)p(s(z)
In case of encoders that provide additional embeddings other than concept logits, they are modelled as Dirac delta distributions, thus not modelling any uncertainty them.

---

(A)
conappts
(E)
R S
T
logical eval p
p(c(x)
red=1, square table
apple
R
S
T
P
p(s(x)
7
p(rls)
2
8
apple red A -square
decded rule
RULEBOOK
(D)
e
e1
62
"rule embedding"
(C)
lateet repr. of nule
Figure 2: Example prediction CMR with a rulebook of two rules and three concepts (i.e. red (R), square (S), table (T)). In this figure, sample (~> for clarity, but practice, compute every probability exactly. Every black box is implemented by neural network, while the white box is a pure symbolic logic evaluation. (A) The image is mapped to concept prediction. (B) The image is mapped by the component selector to a distribution over rules. (C) This distribution is used to select rule embedding from the encoded rulebook.
(D) The rule emhedding decoded into logic rule by assigning to each of the concepts its role in the rule, i.e. whether is positive (P), negative (N), or irrelevant (I). Finally, (E) the rule is evaluated the concept prediction to provide the task prediction on the task apple.
Here, p(s[z) the categorical distribution defining the mixing weights of the mixture. parame- terized by a neural network g(*), : X - R outputting one logit for each of the næ components.
Each ()) is distribution over all possible rules and is modelled as a product of nc categorical distributions, i.e. p(r(s) [[ p(ri(s). [(rsss)) We assign to each component s j a rule embedding ej E IR9, and each categorical r; is then parameterized by a neural network Φ(()) RR + R3, mapping the rule embedding to the logits of the categorical component. Intuitively, for each concept e, the corresponding categorical distribution p(ri(s = j) decodes the embedding ej to the three possible roles {P.N, }} of concept c; in rule This way, each embedding in the rulcbook the latent representation of a logic rule Lastly, we define the set of all rule embeddings, i.c.
E
(e) the encoded rulebook.
3.1.2
Task predictor
The CMR task predictor p(ylr, c) provides the final y prediction given concept predictions and the selected rule r. We model the task predictor as degenerate deterministic distribution, providing the entire probability mass to the unique value y which corresponds to the logical evaluation of the rule on the concept predictions c. In particular, let r; E N, 1} the role of the i-th concept in rule r. Then, the symbolic task prediction obtained by evaluating rule on concept predictions c is:
ng y + A
c,) ((r = N) + -)))
Here, the y prediction is equivalent conjunction of ng different conjuncts, one for each concept. If a concept is irrelevant according to the selected rule (i.e. r; T), the corresponding conjunct is ignored. r; = P. then the conjunct is True if the corresponding concept is True. Otherwise, i.e. = N, the conjunet is True if the corresponding concept is False.
graphical representation of the model shown in Figure 2
PActually, each embedking is the latent representation of distribution over all possible rules, factorized per concept. However. this distribution is typically quite crisp after training (see also Appendix BB) After training. we convert each distribution into the most likely rule by taking the argmax among the roles of the concepts (see Section 5). This way. each embedding in the encoded rulebook corresponds with a rule, and the decoded rulebook is set of rules.

---

4 Expressivity, interpretability and verification
In this section, we will discuss the proposed model along three different directions: expressivity, interpretability and verification.
4.1 Expressivity
An interesting property is that CMR is as expressive neural network binary classifier. Theorem 4.1. CMR is universal binary classifier [33 if na 2 3.
Proof. Recall that the rule selector implemented by some neural network (h(*) : + R=*.
Consider the following three rules, easily expressible in CMR as showed on the right of each rule:
nc y - True (i.c. Vi: :4 = 1), y y+ A c (ic.VV: P), y A cc r
PC
By selecting of these three rules, the rule selector can always make desired y prediction, regardless of the concepts c. particular, to predict y = 1, the selector can select the first rule. To predict y = 0 when at least one concept has probability less than 50% in the concept predictions (i.e. 3i p(c,/2)) (2) < 0.50), it can select the second rule. Lastly, to predict y = 0 when all concepts have probability of at least 50% in (i.e. vi p(c](z)) 0.50), it can select the last rule.
Consequently, CMR can in theory always achieve the same accuracy as a neural network without concept bottleneck, irrespective of which concepts employed in the model. This distinguishes CMR sharply from CBNMs.
4.2 Interpretability
CMR's task prediction is the composition of a (neural) rule selector and the symbolic evaluation of the selected rule. Therefore, we can always inspect the whole rulebook to know exactly the global behaviour of the task prediction. In particular, let be the selected rule, ej the embedding of the j-th rule, and r(6) E {PP, (P,,, 1 the role of the i-th concept in the j-th rule at decision time, i.c. r(()) argmax(of")) (eg)). Then, CMR's task prediction can be logically defined as the global rule obtained as the disjunction of all decoded rules, each filtered by whether the rule has been selected or not. That is:
n
nc = =DV P) = + (( ( -a))
5 V (s=
(4)
clear that if the model learns the three rules in the proof of Theorem 4.1 the selector simply becomes proxy for y. It is safe to say that the interpretability of the selector (and consequently of CMR) fully depends on the interpretability of the learned rules.
Prototype-based rules. In order to develop interpretable rules, we focus on standard theories in cognitive science [14] by looking those rules that are prototypical of the concept activations which they are applied to. Prototype-base bbseed models are often considered one of the main categories of interpretable models [9] and have been investigated in the context of concept-based models (155. However, CMR distinguishes itself from prototype-based models in two significant ways. First, in prototype- based CBMs, prototypes
built directly in the input space, such as images [16] on part of it (17), and are often used to automatically build concepts. However, this approach carries similar issues w.r.t. traditional input-based explanations, like saliency maps (i.c. prototype made by a strange pattern of pixels can still remain unclear to the user). In contrast, in CMR, prototypes (i.e. rules) are built on top of anderstaaaeee concepts and are therefore interpretable well. Second, differently from prototype- based models, CMR assigns a logical interpretation to prototypes as conjunctive rules. Unlike prototype networks that assign class labels only based on proximity to prototype. CMR determines class labels through the symbolic evaluation of prototype rules on concepts. Therefore, prototypes should be representative of concept activations bui also provide correct task prediction. This dual role of CMR prototypes adds constraint: only prototypes of positive instances (i.e. y = 1) can serve as effective classification rules. Thus, when rules are selected for negative instances (i.c. y 0), they do not need to be representative of the concept predictions.
5

---

Interventions. In contrast to existing CBMs, which only allow prediction time concept interventions, the global interpretability of CMR allows humans intervene also on the rules that will be exploited for task prediction. These interactions may occur during the training phase to directly shape the model that is being learned, and may come various forms. first approach involves the manual inclusion of rules into the rulebook, thereby integrating expert knowledge into the model [18]. The selector mechanism within the model learns to choose among the rules acquired through training and those manually incorporated. As a result, the rules that are being acquired through training can change after adding expert rules, A second approach consists of modifying the learned rules themselves, by altering the role r, of concept within rule. For instance, setting the logit of P, to 0 ensures that & cannot exhibit a positive role in that rule, and setting the logits of both P. and N, to ensures irrelevance. This type of intervention could be exploited to remove (or prevent) biases and ensure counterfactual faimess 19
4.3 Verification
One of the main properties of CMR is that, at decision time, it explicitly represents the task prediction as a set of conjunctive rules. Logically, the mixture semanties of the selector can be interpreted as a disjunction, leading to standard semantics in stochastic logic programming [20] [1]] As the only neural component is in the selection and the concept predictions, task predictions generated using CMR's ' global formula (cf. Section 4.2 can be automatically verified by using standard tools of formal verification (e.g. model checking). no matter which rule will be selected. Being able to verify properties prior to deployment of the model strongly sets CMR apart from existing models, where verification tasks can only be applied at prediction time. In particular, given any propositional logical formula over the propositional language {c1, (21,02, .... C v}. can be automatically verified to logically follow from Equation 4 This formula can be converted into a propositional one by (1) evalu- ating the role expressions (i.e. r becomes True or False), (2) replacing the selection expressions
with a new propositional variable per rule (i.e. (s = j) becomes 8y), and (3) adding mutual exclusivity constraints for the different 8j. For example, for Tic nn r = p,r I. = P,r N, we get:
(81 ① 82) A (y < (8) A (52 a -2))
(5)
with @ the xor connective. In logical terms, if the formula a is entailed by such a formula, it means that a must be true every time Equation is used for prediction.
5 Learning and inference
5.1 Learning problem
Learning in CMR follows the standard objective in CBM literature, where the likelihood of the concepts and task observations in the data is maximized. Formally, let be the set of parameters of the probability distributions, such that CMR's probabilistic graphical model is globally parameterized by Ω i.c. p(y. r, (2). Let D = {(t, è, })} be a concept-based dataset of i.i.d. triples (input, concepts, task). Then, learning is the optimization problem:
max
Σ (2.2,))
púg c)
(6)
Due to the factorization of the mixture model in the rule selection, CMR has tractable likelihood computation. In particular, the following result holds.
Theorem 5.1 LLog-iikeihhoo), The maximum likelihood simplifies to the following O(ne ng)
objective:
max
Σ
Σ 1=1
logplc; = =c((t) + log Σ
p(s 8| = Ø/c, 8)
(2,G,0)ED
with:
no
p = 1/c, = II + [[q 11]===]++[[i 1[c 0))
where 1[:] is an indicator function of the condition within brackets.

---

Proof. See Appendix
The maximum likelihood approach only focuses on the prediction accuracy of the model. However, discussed in Section 4.2 look for the set of learned rules to represent prototypes of concept predictions, as in prototype-base based learning [[5]. To drive the learning representative positive prototypes when observe positive value for the task, i... when 1, add regularization term to the objective. Intuitively, every time a rule is selected for a given input instance x with task label 1, we want the rule to be close possible the observed concept prediction. At the same time, since the number rules limited and the possible concept activations are combinatorial, the same rule is expected to be selected for different concept activations. When this happens, will favour rules that assign an irrelevant role to the inconsistent concepts in the activations. The regularized objective is:
nC logple = =c138) + log >
an p(s 8 ply g1è, 8) Preg 2|3)
Σ
Σ
(8)
(EAD)D
Regularization Term
and:
nc
Preg = c|s) II (0.5 p(r: 1[s) p(r: Pls) 1[a 1] p(r. N|s) 1(a 0])
(r
This term favours the selected rule reconstruct the observed é as much as possible. When such reconstruction is not feasible due to the limited capacity of the rulebook, the term will favour irrelevant roles for concepts. In this way, will develop rules that have relevant terms (i.e. r; (P,N)) only they are representative of all the instances in which the rule selected. Appendix BeciixxBoooo contains an investigation of the influence of this regularization the optima of the loss.
5.2 Inference
After training, we replace each role distribution p(r:|s = for each concept and rule embedding with the most likely role. This ensures that each embedding corresponds to single logic rule rather than distribution over all possible rules. Moreover, at decision-time the concepts are unobserved, leading to the following likelihood computation:
pΦy = 1|x) = Σ = = 8|z) I = + 1/r P)p(e()) L[r, = N)p(-a(z)) (9)
1=1
with r; argmaa fils 8).
6 Experiments
Our experiments aim to answer the following research questions:
Generalization: Does CMR attain similar task and concept accuracy as existing CBMs and black boxes? Does CMR generalize well when the concept set is incomplete
(2) Explainability and Intervenability: Can CMR recover ground truth rules? Can CMR learn meaningful rules when the concept is incomplete? Are concept interventions and rule interventions effective in CMR?
(3) Verifiability: Can CMR allow for post-training verification regarding its behaviour?
6.1
Experimental setting
This section describes essential information about experiments. We provide further details in Ap- pendix C
'Icomplete concept sets do not contain all the information present in the input that is useful for task prediction. Models with concept bottleneck cannot achieve black-box accuracy with them.
7

---

Table Task aceuracy on all datasets. The best and second best for CBMs are shown in bold (black and purple, respectively).
MNIST+
MNIST+
C-MNIST
CELEBA
CUB
CEBAB
CBM+linear 0.00
0.00
99.07=0.31
49 49.0211.20
50.60
45.15
CBM+MLP
97.41
T2.51+2.42
99.42:0.11
50.29:1060
55.8310.33
83.70.0.300
CBM+DT
96.73
77.6310.44
99 09.44
49. 0.060
51.83::0.48
83,1510.115
CBM+XG
96.73
76.54
99.44=0.08
50.39+0.24
62.97=0.96
43.80+0.01
CEM
92.44
92.9411.16
99.32-0.11
65.4440.25
57.03=0.80
83.30#1.50
DCR
90.70:1.21
92.24
98.99=0.0s
35.65: 1.53
50.00
67 67.30
Black box
83,26
83.26
99.19=0.11
65.33++0.60
64.07
88.67
CMR (ours) 97. 07.2540.24
94.65::
99.12-0.04
63.1711.13
60.07=1.70
85.1410.44
Data & task setup. We base our experiments on four different datasets commonly used to evaluate
CBMs: MNIST+ 222, where the task to predict the sum of two digits; C-MNIST, where we
adapted MNIST to the task of predicting whether coloured digit is even or odd; MNIST+*, whheee
we removed the concepts for the digits 0 and from the concept set; CelebA [23] large-scale face attributes dataset with more than 200K celebrity images, each with 40 concept annotations CUB (24). where the task is to predict bird species based on bird characteristics; and CEBaB [25], text-based task where reviews are classified as positive negative based different criteria (e.g. food, ambience, service, etc). These datasets range across different concept set quality, i.e. complete (MNIST+, C-MNIST, CUB) incomplete (CelebA, MNIST+*), and different complexities of the concept prediction task, i.c. easy (MNIST+, MNIST+", C-MNiST), medium (CEBaB) and hard (CelebA, CUB). All the datasets come with full concept annotations.
Evaluation. To measure classification performance on tasks and concepts, we compute subset accuracy and regular accuracy, respectively. For CUB, we instead compute the Area Under the Receiver Operating Characteristic Curve (26) for the tasks due to the large class imbalance. All metrics reported using the mean and the standard error of the mean over three different runs with different initializations.
Baselines. In our experiments, we compare CMR with existing CBM architectures. We consider Concept Bottleneck Models with different task predictors: linear, multi-layer (MLP), decision- (DT) and XGBoost (XG). Moreover, we add two ofate-of-the-arr CBMs: Concept Embedding Models (CEM) 27 and Deep Concept Reasoner (DCR) [T]. We employ hard concepts in CMR and our competitors, avoiding the problem of input distribution leakage that can affect task accuracy [28 29 (see Appendix [ additional details). Finally, we include a deep neural network without concepts to measure the effect of an interpretable architecture on generalization performance.
We provide additional experiment serving an ablation study on CMR's joint rule learning in Appendix C.3.3
6.2
Key findings & results
6.2.1 Generalization
CMR's high degree of interpretability does not harm accuracy, which is similar to or better than competitors'. In T compare CMR with its competitors regarding task accuracy. On all data sets, CMR achieves an accuracy close to black-box accuracy, either beating its concept-based competitors obtaining similar results. In Table of Appendix C we show that CMR's training does not harm concept accuracy, which is similar to its competitors. Moreover, we provide an experiment showing that CMR's accuracy is robust to the chosen number of rules in Appendix C.3.4
CMR obtains accuracy competitive with black boxes even on incomplete concept sets. We evaluate the performance of CMR settings with increasingly more incomplete concept sets. Firstly, as shown in Table [ in MNIST+* CMR still obtains task accuracy close to the complete setting, beating its competitors which suffer from concept bottleneck. Secondly, run experiment
We remove the concepts Wavy_ Hair, Black Hair and Male from the concept set and instead use them tasks.
8

---

CelebA where we gradually decrease the number of concepts in the concept set. Figure shows the achieved task accuracies for CMR and the competitors. CMR's accuracy remains high no matter the size of the concept set, while the performance of the competitors with a bottleneck (i.e. all except CEM) strongly degrades.
6.2.2 Explanations and intervention
CMR discovers ground truth rules. We quantitatively evaluate the correctness of the rules CMR learns on MNIST+ and C-MNIST. In the former, the ground truth rules have no irrelevant concepts; in the latter, they do. In all runs of these experiments, CMR finds all correct ground truth rules. In C-MNIST, CMR correctly learns that the concepts related to colour are irrelevant for classifying the digit even odd (see Table
CMR discovers meaningful rules in the absence of ground truth. While the other datasets do not provide ground truth rules, qualitative inspection shows that they are still meaningful. Table shows two examples for CEBaB, and additional rules can be found in Appendix C
Table 2:
Selection of learned rules. For brevity in C- MNIST, negated concepts are not shown and irrelevant concepts are shown between parentheses. We abbreviate good to g, bad to b and unknown to u.
Task Test Accuracy
0.6
0.5
Black hec
0.4
CBK-DT
Deven <- 0 A (red) A (green)
CRM-MUP
CBM-XG
C-MNIST
Veven 2 A (red) A (green)
0.3
Bodd < 3 A (red) A (green)
CMR
12
24
37
Uneg foodg A -amb, A -moise,
Number of Concepts
CEBaB
Upos + food, A -amb, A noise, A -noises A -noiseg
Figure Task accuracy on CelebA with varying numbers of employed concepts.
Rule interventions during training allow human experts improve the learned rules. We show this by choosing a rulebook size for MNIST+ that is too small to learn all ground truth rules. Consequently, CMR learns rules that differ from the ground truth rules. After manually adding rules to the pool in the middle of training, CMR (1) learns to select these new rules for training examples for which they make correct task prediction, and (2) improves its previously learned rules by eventually converging to the ground truth rules. This is the case for all runs. Table 3 gives some examples of how manually adding rules affects the learned rules. In Appendix C.3.3 we provide an additional experiment with rule interventions, where we add rules extracted from a rule learner. Additionally, as concept interventions are considered a core advantage of CBMs, we show in Appendix that CMR is equally responsive competitors, consistently improving its accuracy after concept interventions.
Table Selection of rule interventions and their effect on learned rules. For brevity, negated concepts are not shown, and irrelevant concepts shown between parentheses.
Learned rule before intervention
Added rule
Learned rule after intervention
ys + (Co,s) A (ci,s) A (co.4) A (c1.4)
ys + Co.3 A C1,5
+ Ca,4 A C1,4
1/ (Co,8) A (c1,1) A (Co.1) ) A (ct,8)
Yg + Co.8 C1,1
y/ C0 C1,8
1/ + (Co.o) A (c1.9) A (Co.2) A (c1,7)
Uo <- Co.o C1,9
yo Co,2 A C1,7
6.3 Verification
CMR allows verification of desired global properties. In this task, we automatically verify semantic consistency properties for MNIST+ and CelebA whether CMR's task prediction satisfies some properties of interest. For verification, we exploited naive model checker that verifies whether the property holds for all concept assignments where the theory holds. When this not feasible, state-of-the-art model formal verification tools can be exploited, both the task prediction and the property are simply two propositional formulas. For MNIST+, we can verify that, for each task y, CMR never uses more than positive concept (i.c. digit) per image. This can be done by

---

verifying one formula per concept of digit k: Vy, + Ck. = Ckk. This is also easily verifiable by simply inspecting the rules in Appendix C Moreover, in CelebA, we can easily verify that Bald- = - @Vavy_Hair with the learned rulebook for ni = 12 (see i0 Appendix ( as BBll is a conjunct each rule that does not trivially evaluate to False
Related works
In recent years, XAI techniques have been criticized for their vulnerability to data modifications 30 BI, insensitivity to reparamettizations, [32], and lacking meaningful interpretations for non- expert users [33] To address these issues, Concept-based methods [34 35
3 T0 have emerged, offering explanations in terms of cumanninterpretable features, a.k.a. concepts. Concept Bottleneck Models 4 go step further by directly integrating these concepts as explicit intermediate network representations. Concept Embeddings Models (CEMs) [7] 8.811 11 close the accuracy gap with black- box models through vectorial concept representations. However, they still harm the interpretability, as it is unclear what information is contained in the embeddings. In contrast, CMR closes the accuracy gap by using a neural rule selector coupled with learned symbolic logic rules. As result, CMR's task prediction is transparent, allowing experts to see how concepts are being used for task prediction, and allowing intervention and automatic verification of desired properties. To the best of our knowledge. there is only one other attempt at analysing CBMs' task prediction in terms of logical formulae, namely DCR [1. For a given example, DCR predicts and subsequently evaluates a (fuzzy) logic rule. As rules are predicted on a per-example basis, the global behaviour of DCR cannot be inspected, rendering interaction (e.g. adding expert rules) and verification impossible. contrast, CMR learns (probabilistic) logic rules in memory, allowing for inspection, interaction and verification.
The use of logic rules by CMR for interpretability purposes aligns it closely with the field of
neurosymbolic Al (36 37). Here, logic rules [8 39 18 or logic programs [40] [2 [1] are used
in combination with neural networks through the use of neural predicates (22. Concepts in CMR are akin to a propositional version of neural predicates. However, in CMR, the set of rules is learned instead of given by the human and direct concept supervision is used for human alignment. While neurosymbolic rule learning methods have been developed, many are constrained by specific assumptions about the nature of the task, limiting their usability particular datasets or environments (e.g. requiring multitask scenarios [4] or specific datasets like MNIST 42).. Additionally, some approaches, unlike ours, explore the rule space in a discrete manner [43], which computationally expensive. Furthermore, they do not provide expressivity results, while we show that CMR is universal binary classifier.
Finally, the relationships with prototype- based models have already been discussed in Section 4.2
8 Conclusion
We propose CMR, a novel Concept-Based Model that offers human- uunerstaadee and provably- verifiable task prediction process. CMR integrates a neural selection mechanism over memory of learnable logic rules, followed by a symbolic evaluation of the selected rule. This approach enables global interpretability and verification of task prediction properties. Our results show that (1) CMR achieves blar-black-box accuracy, (2) discovers meaningful rules, and (3) facilitates strong interaction with human experts through rule interventions. The development of CMR can have significant societal impact by enhancing transparency, verifiability, and human-Al Al interaction, thereby fostering trust and reliability in critical -eciionn-making processes.
Limitations and future works. CMRs are still fundamental models and several limitations need to be explored further in future works. In particular, CMRs focus on positive-only explanations, while negative-reasoning explanations have not been explored yet. Moreover, the same selection mechanism can be tested in non-logic, globally interpretable settings (like linear models). Finally, the verification capabilities of CMR will tested on more realistic, safety critical domains, where the model can be verified against safety specifications.
10

---

Acknowledgments and Disclosure of Funding
DD a fellow of the Research Cuundation-Flandder also received funding from the KU Leuven Research Fund (STG/22/021, CELSA/24/008) and from the Flemish Government under the OOddezzoekprrorramma Artificiële Intelligentie (AI) Vlaanderen" programme. FG has been supported by the Partnership Extended PE00000013 "FAIR Future Artificial Intelligence Research" Spoke 1 "Hummn--entered Al". PB acknowledges support from SNSF project TRUST-ME (No. 2051211 (0I1__11999 214991). MD was supported by TAILOR and by HumanE- AI-Net projects funded by EU Horizon 2020 research and innovation programme under GA No 952215 and No 952026, respectively. This study has received funding from the European Union's EU Framework Program for Research and Innovation Horizon under the Grant Agreement No 101073307 (MSCA-DN LeMuR).
MVV--Vlandeern, 1185125N). This research has
References
[1] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM computing surveys (CSUR), 51(5): (155:11-42) 42, 2018.
[2] Amina Adadi and Mohammed Berrada. Peeking inside the black box: a survey on explainable artificial intelligence (xai). IEEE access, 1.52138-52160, 2018.
[3] Alejandro Barredo Arrieta, Natalia Diaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Ben- jamins, al. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information fusion, 58:82-115, 2020.
[4] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning, pages 5338 5348. PMLR, 2020.
[5] David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. Advances in neural information processing systems, 31, 2018.
[6] Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772-782, 2020.
[7] Mateo Espinosa Zarlenga, Pietro Barbiero, Zohreh Shams, Dmitry Kazhdan, Umang Bhatt, Adrian Weller, and Mateja Jamnik, Towards robust metrics for concept representation evaluation. arXiv preprini ar ar3iv: 10367,
[8] Eunji Kim, Dahuin Jung, Sangha Park, Siwon Kim, and Sungroh Yoon. Probabilistic concept bottleneck models. arXiv preprint urXiv:2306.01574 2023.
[9] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206-215: 2019.
[10] Eleonora Poeta, Gabriele Ciravegna, Eliana Pastor, Tania Cerquitelli, and Elena Baralis. Concept- based explainable artificial intelligence: survey. arXiv preprint arXiv:2312. 2023.
[11] Pietro Barbiero, Gabriele Ciravegna, Francesco Giannini, Mateo Espinosa Zarlenga, Lucie Char- lotte Magister, Alberto Tonda, Pietro Lio', Frederie Precioso, Mateja Jamnik, and Giuseppe Marra. Interpretable neural- soural-symbolic concept reasoning. In ICML, 2023.
[12] Jason Weston, Sumit Chopra, and Antoine Bordes. 110. 2014.
Memory networks. arXiv preprint
[13] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 255):359-366, 1989.
[14] Fleanor Rosch. Principles of categorization. In Cognition and categorization, pages 27-48. Routledge, 1978.

---

[15] Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistic Surveys, 16: 16:1-85, 2022.
[16] Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case- based reasoning through prototypes: A neural network that explains its predictions. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
[17] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019.
[18] Michelangelo Diligenti, Marco Gori, and Claudio Sacca. Semantic-based regularization for learning and inference. Artificial Intelligence, 244:143-165, 2017.
[19] Matt Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. Advances in neural information processing systems, 30, 2017.
[20] James Cussens. Parameter estimation in stochastic logic programs. Machine Learning, 44: 245- 245-271, 2001.
[21] Thomas Winters, Giuseppe Marra, Robin Manhaeve, and Luc De Raedt. Deepstochlog: Neural stochastic logic programming. Proceedings of the AAAI Conference on Artificial Intelligence, 10100-1000000 Jun. 2022. doi: 10. org/ brg/index.oo php/
URL URL hattpa://ojs aaai
[22] Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. DecpProbLog: Neural Probabilistic Logic Programming. In NeurlPS, pages 3753-3763, 2018.
[23] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, 2015.
[24] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Be- longie, and Pictro Perona. Caltech-ucsd birds 200. Technical Report CNS-TR-201, Cal- tech, 2010. URL /ttpo///aoooo////////////oooooon http:///wwppppooo/////////////ee
http://wwx: //www. wwsiin..ii...m wwtpoaaww..ccccccceeeeeeeeeaaaaa
2hmmmmm11
[25] Eldar D Abraham, Karel Oosterlinck, Amir Feder. Yair Gat, Atticus Geiger, Christopher Potts, Roi Reichart, and Zhengxuan Wu. Cebab: Estimating the causal effects of woal-world concepts on nlp model behavior. Advances in Neural Information Processing Systems, 35: 17582-17596, 2022.
[26] David J. Hand and Robert J. Till. A simple generalisation of the area under the roc curve for multiple class classification problems. Mach. Learn., 45(2):171-186, oct 2001. ISSN 0885-6125, doi: 10. 10011///11001002081199 19831. htt.itttt///oo...rg//oooo 10t:://:...00/:1111:::::::::: 10110//21/11000011111111
[27] Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, Pietro Lio, and Mateja Jamnik. Concept embedding models: Beyond the accuracy- explainability trade-off. Advances in Neural Information Processing Systems, 35, 2022.
(28) Emanuele Marconato, Andrea Passerini, and Stefano Teso. Glancenets: Interpretable, leak leak-proof concept-based models. Advances in Neural Information Processing Systems, 35:21212-212277, 2022.
(29) Marton Havasi, Sonali Parbhoo, and Finale Doshi-Velez. Addressing leakage in concept bottleneck models. Advances in Neural Information Processing Systems, 35:23386-23397 2022.
[30] Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. Explainable AI: Interpreting, explaining and visualizing deep learning, pages 267-280, 2019.
12

---

[31] Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. In Proceedings of the AAAI conference on artificial intelligence. volume 33, pages 3681-3688, 2019.
[32] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances neural information processing systems, 31, 2018.
[33] Forough Poursabzi-Sangdett Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Wort- man Vaughan, and Hanna Wallach. Manipulating and measuring model interpretability. In Proceedings of the 2021 CHI conference on human factors computing systems, pages 1-52, 2021.
[34] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages 2668-2677, PMLR, 2018.
[35] Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept- based explanations. Advances in neural information processing systems, 32, 2019.
[36] Artur d' Avila Garcez, Sebastian Bader, Howard Bowman, Luis C Lamb, Leo de Penning, BV Illuminoo, Hoifung Poon, and COPPE Gerson Zaverucha. Neural-symbolic learning and reasoning: survey and interpretation. Neuro-Symbolic Artificial Intelligence: The State the Art, 342(1):327, 2022.
[37] Giuseppe Marra, Sebastijan Dumančić, Robin Manhaeve, and Luc De Raedt. From statistical relational neurosymbolic artificial intelligence: A survey, Artificial Intelligence, page 104062, 2024.
[38] Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Broeck. semantic loss function for deep learning with symbolic knowledge. In International conference on machine learning, pages 5502-5511. PMLR, 2018.
[39] Samy Badreddine, Artur d' Avila Garcez, Luciano Serafini, and Michael Spranger. Logic tensor networks. Artificial Intelligence, 303: 303:103649, 2022.
[40] Gustav Sourek, Vojtech Aschenbrenner, Filip Zelezny, Steven Schockaert, and Ondrej Kuzelka. Lifted relational neural networks: Efficient learning of latent relational structures. Journal of Artificial Intelligence Research, 62:69-100, 2018.
[41] Hao Tang and Kevin Ellis. From perception to programs: regularize, overparameterize, and amortize. In International Conference on Machine Learning, pages 33616-33631. PMLR, 2023.
[42] Alessandro Daniele, Tommaso Campari, Sagar Malhotra, and Luciano Serafini. Deep symbolic learning: Discovering symbols and rules from perceptions. arXiv preprint (xXiv:2288.11661 2022.
[43] Richard Evans and Edward Grefenstette. Learning explanatory rules from noisy data. Journal of Artificial Intelligence Research, 61:1-64, 2018.
[44] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):227--2324 1998. doi: 10. (0.1109/5.72691.
[45] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[46] Jacob Devlin, Ming-Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810. 10.044855 2018.
13

---

[47] Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing. In Proceedings of the 2020 Conference in Natural Language Processing: System Demonstrations, pages 3845,, Online, October 2020. Association for Computational Linguistics. URL https: /wwww wwwwablwwwbb..... org/ anthology/ 2020 emm.p-demos....
Empirical Methods
[48]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach Vitito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019.
[49]
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Andreas Müller, Joel Nothman, Gilles Louppe, Peter Pretten- hofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthicu Brucher, Matthieu Perrot, and Édouard Duchesnay. Scikit-leam: Machine learning in python, 2018
[50] D. Hunter. Matplotlib: 2d graphics enviromment. Computing Science Engineering, (3):90-95, 2007. doi: 10. 00.100//CSE.2000.555.
14

---

A
Maximum likelihood derivation
We show that the maximum likelihood problem Equation 6
simplifies to:
log
log =c((²) + log >
pls p(y 5/., - 8)
$=1
with:
s) II + = p(N.s) = 0))
Proof. Let n: =n: nc. First, we express the likelihood as the marginalization of the distribution over the unobserved variables
c(y, = p(c(x) Ep(sla) Pa. Tas C1o Cr.
Due to the independence of the individual components of the rule distribution:
p(r1,..., Fnls)
II p(r|ss)
2=1
The logical evaluation of rule be expressed in terms of indicator functions the different variables involved.
p(ylr Twi C1, Co, Cm) II
1/r 1] + 1[r P] 1] + 1/r, n| 0]
i=1
Let us define f(r,, G) = 1|r, = I] + 1[; P|L[c = 1] + A[r = n|l[c; = 0].
Then, the likelihood becomes:
ply, c|x) = p(c|z) Σ p(s|x) Σ EII [(rr))))(rrrc)))
In 1=1
= =p(ck)) Σ p(sla) Ep(rils) [ptrib)) slr (--) Σ Eptrals) ((ena)))((m
=p(ckz) Σ p(s|a) II p(rals) f(r.,c)
2m1
T
= =(cc()) Σ p(s(a) I
((N())) ((N.))
= =(c[=) Σ p(s|a) II ((P
1cc +] 1cc = 0))
= =p(c/)) Σ p(slc))(((c.)
Using p(c/x) = I p(cl))) and applying the logarithm, we find:
logr(y. clar =
Σ
log p(r((()))) + log p(s))) plule))
15

---

N
0.-
1.0
0.0
1.0
0.0
1.0
0.8
0.33
0.33
0.33
00
0.
00
14
14
04
10
0L0
100
0.0
10-
000
(a) p(yle)
(b) p(ul [c)
(c) p(vlc) * p(ul
N
N
N
1.0
0.25
0.33
06
06
0.15
0.65
0.65
0.66
02
02
1.0
0D
n0
1.0
0.00
(d) Prey (r)
(c) Preyy (-r)
(f) Prey (r) Prey (-r)
N
N
N
0.0
0.33
0.66
10
0.0
0.35
0,66
100
0.0
0.33
0.06
1.0
0.5
0.8
0.20
0.33
0.33
0.35
06
8665
0.66
02
02
0.06
(g) p(ylc) (r)
(h) p(y) -c) Preg (-r)
ii) plycc) preg( Preg T)
p(vl -c)
Figure 4: Likelihoods be maximized when y = w.r.t the role r of single concept in selected rule, for different situations. As P N + = 1, irrelevance the coordinate (0, 0). Likelihoods that cannot be achieved (i.c. when P + N >> 1) are put to 0. In the first column, the concept label is 1. In the second column, it is O. In the third column, two examples with opposite labels select the rule.
B Implementation and optimization details
Selector re-initialization To promote exploration, we re-initialize the parameters of p(s[æ) multiple times during training, making it easier to escape local optima. The re-initialization frequency is hyperparameter that differs between experiments, see Appendix
Effect of the regularization term Figure 4 shows the probabilities be maximized when the label y = 1, for a selected rule and a single concept, with respect to different roles r for that concept.
Figures 4a 4b and 4c the probabilities to be maximized without the regularization. Figures 4d] 4e and 4f show the regularization probabilities (remember, we only have these if 1). Figures and 41 show the probabilities when both present. As mentioned in the main text, when e 1S True for all examples that select the rule, we want that concept's 's role to be P. However, when optimizing without regularization, it is clear that e.g. / also an optimum (Figure 4ap. Because the regularization only has an optimum in P (Figure (dD. adding the regularization results in the correct optimum (Figure 4g). A similar reasoning applies when False for all examples that select the rule; consider Figures 4b 4e and 4h It should be noted that the regularization alone is insufficient to get the correct optima; indeed, when examples with different e select the rule, the regularization term has many optima (Figure 4f). of which only I is desired (as explained in the main text). As the original loss only has an optimum
4g
in this case (Figure 4c), the only optimum that remains is the correct one (Figure 4i). In summary: the loss function we use has the desired optima (last row in Figure 4 dropping the regularization (first row in Figure 4 or only keeping the regularization (middle row in Figure both have cases where the optima are undesired.
16

---

Passing predicted concepts While probabilistic semantics require passing the ground truth to the task predictor during training (as these are observed variables), CBMs typically instead pass the concept predictions (opposite of teacher forcing) make the model more robust to its own errors. For this reason, we also employ this technique in CMR in the experiments with CEBaB, CUB and CelebA For competitors, we always employ this technique. However, we pass hard concept predictions in both CMR and the competitors, as to avoid leakage (see Appendix C).
Weight in the loss We introduce a weight 9 in the regularized objective of Equation g (see below). This adjustment helps to balance the regularization and the task loss. For instance, if the regularization dominates, CMR might learn rules that can only be used to predict y 1 in practice. Additionally, larger might assist settings where achieving high accuracy is difficult for some concepts; such cases, concepts with lower accuracy can be more easily considered irrelevant by choosing larger 3, as they are unable to reliably contribute to correct task prediction. The updated objective as follows:
log p(: = =2((E)
+ log > p(s = 8| ply = g/e, 8) Preg = E 8=1
max Q
(2
Regularization Term
(10)
C Experiments
C.1 Datasets
MNIST+ This dataset consists of pairs of images, where each image an MNIST image of a digit and the task is the sum of the two digits. For these tasks, all concepts are relevant. There is total of 30,000 training examples and 5,000 test examples.
MNIST+ We create this dataset as MNIST+ except that the concepts for digits 0 and are removed from the concept set. This makes the concept set incomplete.
taking the MNIST training (60,000 examples) and test set (10,000 examples), randomly colouring each digit either red or green, and adding these two colours concepts. There are two tasks: The first is whether the digit is even, and the second is whether is odd. For these tasks, the concepts related to colour are irrelevant.
C-MNIST We derive this dataset from MNIST 44
This is a large-scale face attributes dataset with more than 200K celebrity images [3 Each image has 40 concept annotations. As tasks, we take the concepts W avy H Hoir, Black Hair, and Male, removing them from the concept set.
CelebA
CUB In this dataset, the task is to predict bird species from images, where the concepts are 112 bird characteristics such as tail colour, wing pattern, etc [24). CUB originally consists of 200 tasks, of which take the first 10. Some concepts strongly imbalanced (some True in only ++0.5 of examples, others in 40%, etc.) and there is a large task imbalance (each task is True in only +0.5% of examples).
CEBaB
This is text-based dataset that consists of reviews, where the task is to classify them as positive negative (25). There are 12 concepts (food, ambience, service, noise, each either unknown, bad or good) and 2 tasks (positive or negative review).
C.2 Training
Reproducibility
We seed all experiments using seeds 1, 2 and 3.
Soft vs hard concepts When departing from pure probabilistic semantics, CBMs allow not only the use of concepts as binary variables, but they allow for concepts to be passed to the task predictor together with their prediction scores, which is called employing soft concepts (vs hard concepts). While some CBMs use soft concepts this results in higher task accuracy, the downside of this
17

---

is that the use of soft concepts also comes with the introduction of input distribution leakage: The concept probabilities can encode much more information than what is related to the concepts, severely harming the interpretability of the model [28 29 For this reason, in our experiments, all models use hard concepts, which realized by thresholding the soft concept predictions at 50%.
Model input For MNIST+, MNIST+* and C-MNIST, train directly on the images. For CelebA and CUB, instead of training on the images, we train the models on pretrained ResNet18 embeddings [45. Specifically, using the torchvision library, we first resize the images to width and height 224 (using bi-linear interpolation), then normalize them per channel with means (0.485, 0.456, 0.406) and daaiatnd-deviations (0.229, 0.224, 0.225) (for CelebA only). Finally, remove the last (clas- sification) layer from the pretrained ResNet18 model, use the resulting model on each image, and flatten the output, resulting in an embedding. For CEBaB, we use pretrained BERT model [46] to transform the input into embeddings. Specifically, using the transformers library [47]. we create a BERT model for sequence classification from the pretrained model bbert-base-nncased with 13 labels (I per concept and representing both tasks). Then, we fine-tune this model for 10 epochs with batch size 2, 500 warm-up steps. weight decay 0.01 and 8 gradient accumulation steps. After training, we use this model to transform each example into an embedding by outputting the last hidden states for that example.
General training information In all experiments, we use the AdamW optimizer. All neural competitors are optimized to maximize the log-likelihood of the data with a weight on the likelihood of the task (1 if not explicitly mentioned below). After training. for each neural model in CelebA, CEBaB, MNIST+ and MNIST+*, we restored the weights that resulted in the lowest validation loss. In CUB, C-MNIST and the MNIST+ rule intervention experiment, we do not use a validation set, instead restoring the weights that resulted in the lowest training loss. In CelebA, we use a validation split of 8:2, a learning rate of 0.001, a batch size of 1000, and we train for 100 epochs. In CEBaB, we use a validation split of 8:2, a learning rate of 0.001, a batch size of 128, and we train for 100 epochs. In CUB, we use learning rate of 0.001, batch size of 1280, and train for 300 epochs. In MNIST+ and MNIST+*, we use a validation split of 9:1. We use learning rate of 0.0001, a batch size of 512, and we train for 300 epochs. In C-MNIST, we also use a learning rate of 0.0001, a batch size of 512, and we train for 300 epochs.
General architecture details In CMR, we use two hidden layers with ReLU activation to transform the input into different embedding. We use 3 hidden layers with ReLU activation and an output layer with Sigmoid activation to transform that embedding into concept predictions. The component p(s]x) takes as input that embedding and is implemented by single hidden layer with ReLU activation and an output layer outputting Nitaka * nR logits, which are reshaped to (Nrakke, RR) and to which a Softmax is applied over the rule dimension. The rulebook is implemented as an embedding module of shape (Mtasks n. rule emb size) that is reshaped to (Neasks: R2, rule emb size). The rule decoder is implemented by single hidden layer with ReI.U activation and an output layer outputting 3 x ncconcpps logits, which are reshaped to (Nconcepts: 3). after which a Softmax is applicd to the last dimension; the result corresponds to p(rls). At test time, we make p(rls) deterministic by setting the probability for the most likely role for each concept to I and the others to 0 (effectively collapsing each rule distribution to the most likely rule for that distribution). Then, exactly one rule corresponds with each s.
The deep neural network is a feed-forward neural network consisting of some hidden layers with ReLU activation and an output layer with Sigmoid activation.
Both CBM+linear and CBM+MLPP have a concept predictor that is a feed-forward neural net- work using ReLU activation for the 3 hidden layers and Sigmoid activation for the output layer. For CBM+linear, the task predictor is a single linear layer per task with Sigmoid activation. For CBM+MLP, this is a feed-forward neural network using ReLU activation for the 3 hidden layers, and a Sigmoid activation for the output layer.
For CEM, we use 4 layers with ReLU activation followed by Concept Embedding Module where the concept embeddings have a size of 30. The task predictor is a feed-forward neural network using ReLU activation for the hidden layers and sigmoid activation for the output layer.
For DCR, we also use 4 layers with ReLU activation to transform the input into an embedding. Then, one layer with ReLU activation and one with Sigmoid activation are used to transform the embedding
18

---

into concept predictions. The embedding and the concepts are fed to the task predictor, which is a Concept Reasoning Layer using the product-t norm and temperature of 10.
For CBM+DT and CBM+XGbbosst we train a CBM+MLP and use its concept predictions as training input to the trees.
In the MNIST+ and MNIST+ experiments, as mentioned earlier, we use as input the two images instead of embeddings. Therefore, in all models, we use CNN (trained jointly with the rest of the model) to transform the images into an embedding. This CNN consists of a Conv2d layer with output channels and kernel size 5, a MaxPool2d layer with kernel size and stride 2, ReLU activation, a Conv2d layer with 16 output channels and kernel size 5, another MaxPool2d layer with kernel size and stride 2, and another ReLU activation. The result is flattened, and linear layer is used to output an embedding per image (with size the same as the number of units in the hidden layers of the rest of the models). This results in 2 embeddings (one per image), which layers, each with ReLU activation (except the last one) and again the same number of units (except the first layer, which has number of units). Additionally, the CNN can output concept predictions by applying 3 linear layers with ReLU activation and a linear layer with a Softmax to each image embedding. This results in 10 concept predictions per image, which are concatenated. The final output of the CNN is the tuple of concept predictions and embedding.
combined using 10 linear
In the C-MNIST experiments, the input is a single image. We use similar CNN architecture for MNIST+ but with two differences. Firstly, the activation before the concept prediction is a Softmax on only the first 10 concepts (related to the digits), while the activation on the last 2 concepts is Sigmoid. Secondly, after the flattening operation, we use 3 linear layers with ReLU activation and 1 linear layer without activation to transform the embedding into different one.
Hyperparameters per experiment We use number of hidden units and embedding size inter- changeably. In CelebA, we always use 500 units in each hidden layer, except for CBM+linear and CMR where we use 100 units instead. For the deep neural network, we use 10 hidden layers. CMR uses rule embedding size of 100, most 5 rules per task, a 3 of 30, and we reset the selector every 35 epochs. The trained CBM+MLP for CBM+DT and CBM+XGboos has a weight on the task of 0.01.
In CEBaB, we use 300 units in the hidden layers, except for CMR where we use 100 units instead. For the deep neural network, we use 10 hidden layers. We additionally put weight the task loss of 0.01. For CMR, we use a rule embedding size of 100, at most 15 rules per task, a B of 4, and we reset the selector every 10 epochs. For CBM+DT and CBM+XGboost, the trained CBM+MLP has weight on the task of 0.01.
In CUB. we use 100 units in the hidden layers, the deep neural network has 2 hidden layers, and we use a weight on the task loss of 0.01 for concept-base additionally down-weigh the loss for negative instances with weight of 0.005 to deal with the class imbalance. We use rule embedding size of 500, at most 3 rules per task, and we reset the selector every 25 epochs. For CBM+XGBoost, we add a weight of 200 for the positive instances to deal with the class imbalance. For CBM+DT and CBM+XGboost, we do not train a CBM+MLP, instead using CMR's 's concept predictor.
competitors. For CMR, we use a B3 of 3, and
In MNIST+ and MNIST+", we use the CNN as described earlier. We always use 500 units in the hidden layers, except for CBM++inear and CMR where we use 100 units. For the deep neural network, we use 10 hidden layers. For CMR, we use a rule embedding size of 1000, at most 20 rules per task, we reset the selector every 40 epochs, and 3 is 0.1. In MNIST+ specifically, instead of passing the CNN's output embedding to the rule selector p(s|z). we pass the CNN's concept predictions, showing that this alternative can also be used effectively when using complete concept set. For CBM+DT and CBM+XGboost, we train a CBM+MLP with weight on the task of 0.01 and use its concept predictions as input to the trees. For the rule intervention experiment on MNIST+ (where we give CMR a rulebook size that is too small to learn all ground truth rules). we allow it to learn at most 9 rules per task. Here, the input to the selector is the CNN's output embedding. In MNIST+*, we pass the CNN's output embedding to the rule selector.
In C-MNIST, we use the second CNN described above. We always use 500 units in the hidden layers, except for CBM+linear where we use 100 units. For the deep neural network, we use 10 hidden layers. For DCR and CEM, we put weight the task loss of 0.01. For CMR, the rule book has at
19

---

most 6 rules per task, with rule embeddings of size 1000. We use 3 of 1, reset the selector every 40 epochs, and additionally put a weight on the concept reconstruction relative the concept counts. The CBM+MLP we train for CBM+DT and CBM+XGboost has a weight of 0.01 on the task.
Hyperparameter search For CBM+DT, we tune the maximum depth parameter, trying all values between 1 and nc. and report the results with the highest validation accuracy (training accuracy in absence of a validation set). Parameters for the neural models were chosen that result in the highest validation accuracy (training accuracy in absence of a validation set) (for CMR the / parameter, rulebook size and rule embedding size were also chosen based on the learned rules). For B. we searched within the grid [0.1, 1,3, 4, 10, 30), for the embedding size within the grid [10, 100, 300, 500, 1000], and for the rule embedding size within [100, 500, 1000]. For the deep neural network's number of hidden layers, we searched within the grid [2, 10, 20J. For unmentioned parameters in the competitors, we used the default values.
Remaining setup of the rule intervention experiment We first train for 300 epochs. Then, we check in an automatic way for rules that differ from the ground truth. A representative example is the rule yss +- (co.1) A (c1.2) 2 A (Co.3) A (c1,0 D which can be used by the selector for correctly and + 0 = 3. After this, we add each missing ground truth rule, except
predicting that 1 + one, and let CMR continue training for 100 epochs. Consequently, CMR improves its originally learned rules that differed from ground truth to the ground truth ones and learns to correctly select between the learned and manually added rules. For instance, if CMR originally learned rule Ys < (Co.1) A (c1,2) A (Co,3) A (c1,0), after manually adding Ya f cn,1 A C122 and fine-tuning, the rule improves Ys Co,3 C1. and CMR will no longer select this rule for the examples 3 instead selecting the manually added rule.
C.3
Additional results
C.3.1 Concept interventions
To measure the effectiveness of concept interventions 4 we report task accuracy before and after replacing the concept predictions with their ground truth. In Table 4 we observe that CMR is responsive to concept interventions: After the interventions, CMR achieves perfect task accuracy, outperforming CEM and DCR. We leave a more extensive investigation of concept interventions for CMR to future work.
Table 4: Task accuracy before and after concept interventions for MNIST+.
BEFORE
AFTER
CBM+MLP
97.51
100.0:0.00
CEM
:2.44
94.68+00.11
DCR
90.70
94.11+0083
CMR (ours)
97.25+0.21
100.0+0.00
C.3.2
Concept accuracies
Table 5 gives the concept accuracy of CMR and the competitors on all datasets. CMR's concept accuracy similar its competitors across all datasets.
C.3.3 Rule interventions with rule learner
This experiment serves as an ablation study on CMR's end-to-end rule learning component. We investigate whether pre-learnngg rules using external rule learner and manually integrating them into CMR's s memory impacts its accuracy. We employ a decision tree as the rule learner, although
'In this notation, for brevity, we only show concepts with a role that is positive (of which none are present in this rule example) or irrelevant (between parentheses); concepts with negative roles are not shown and are the remaining ones.
20

---

Table 5: Concept accuracies for all datasets.
MNIST+
MNIST+
C-MNIST
CELEBA
CUB
CEBAB
CBM+linear
99.76=0.02
99. 09.73+0.00
99.72+0.00
87.63=0.02
86.24=0.05
92.69+0007
CBM+MLP
99.75
99.71+0006
99.80
87.68
86.32:0008
92.67 10.07
CBM+DT
99.69
99.68++0.0
99.81+0.01
87.70=0.04
85.95=0008
92.83++0.0
CBM+XGB
99.69=0.04
99.68:20001
99.81+0.01
87.70=0.02
85.95=
92.83++002
CEM
0..26=10.04
99. 19.26-10.13
99.7740.01
87.57
85.8210.23
92.46 12.46:10.13
DCR
99.21 1.009
99.7410.02
99.7610.02
85.3510.26
85.6710.15
92.66110.04
CMR (ours)
99.7410.02
99.73.1.001
99.8210.01
86.80
85.95
92.7010.08
other rule learners could be used as well. We use the CEBaB dataset where we only take the first task and the first concepts.
we compare CMR's s accuracy against 3 alternatives: (1) using CBM+DT; (2) injecting the rules learned by the CBM+DT for predicting a positive class into CMR while preventing CMR from learning any rules itself; and (3) injecting these same rules but allowing CMR to learn an additional 15 rules.
In Figure 5
CMR's rule learning component enhances accuracy. When CMR is allowed to learn rules (both CMR (n=15) and DT CMR (n=15)), its accuracy improves significantly compared to when only the decision tree's ' rules are used (DT - CMR (n=0)).
Injecting rules into CMR does not reduce accuracy when CMR is allowed to learn additional rules. CMR achieves similar levels of accuracy in both cases where it only learns rules (CMR (n=15) and when supplemented with the decision tree's 's rules (DT - CMR (n=15)).
CMR's rule selector enhances accuracy beyond the pre-obtained rules. Even when CMR is restricted from learning any rules (DT CMR (n=0)) it still performs better than the original CBM+DT (DT) because of the rule selector.
For learning the decision tree (and the corresponding CBM used for predicting the concepts before- hand), used seed
C.3.4 Accuracy robustness to the number of rules
In this experiment, we investigate the robustness of CMR's ' accuracy with respect to the number of rules hyperparameter Rn.). We use the CEBaB dataset where only consider the first task, and train CMR for different values of n- Figure 6 shows that similar levels of accuracy are obtained regardless of the chosen ng value.
21

---

0.90
Task Test Accuracy
0.85
0.80
0.75
CMR (a-15)
DT
DT CMR (a-0)
DT CMR (6-15)
Model
Figure 5: Rule interventions on CEBaB where we predict only the first task and employ 6 concepts. We compare CMR's accuracy with CBM using a decision tree (DT) and CMR after adding the decision tree's rules to CMR's memory (DT > CMR) without any additional learnable rules (n 0) and when allowing 15 additional learnable rules (n 15). The mean and standard deviation is shown over 3 seeds. This means that (1) CMR 's end-to-end rule learning allows it to obtain higher accuracy than when purely integrating pre-obtained rules from other rule learners (removing CMR's rule learning component), (2) just integrating pre-obtained rules in CMR (while still allowing more rules to be learned) does not decrease its accuracy, and (3) using CMR with only pre-obtained rules still surpasses the performance of the rules in isolation due to the selector.
1.00
Mean Accuncy
Standard Deviation
0.95
Task Test Accursey
0.90
0.85
0.80
10
15
20
25
Number of Rules
Figure 6: Robustness of CMR's accuracy w.r.t. the number of rules on CEBaB where we predict the first task. The mean and standard deviation is shown over seeds. High accuracy can be obtained regardless.
C.3.5 Decoded rulebooks
In this section, we provide examples of the decoded rulebooks after training with seed for our experiments. We do provide decoded rulebooks for MNIST+, as these rules always correspond to the ground truth and are the same over all seeds, i.c. for every task i, CMR learns a rule per possible pair digits that correctly sums that number i. An example of such a rule is ys f- C1 C2,1 D where D is conjunction of ne 2 conjuncts. where each conjunct is the negation of different concept. In the tables this section, we provide the decoded rulebooks for MNIST+". C-MNIST, CelebA (for all concept subsets, but not the complete rulebook for nc 37), CUB (not complete either) and CEBaB. Importantly, for brevity, we use a different notation for rules for C-MNIST, MNIST+* and CUB: We only show concepts in the rule with as role either positive (as is) or irrelevance (between parentheses); we do not show concepts with negative roles. For the other rulebooks, the standard notation.
22

---

C-MNIST CMR has learned that the concepts red and green are irrelevant for predicting even or odd, learning the (same) ground truth rules for all seeds. We show these rules in 6
MNIST+ For brevity, we drop the notation that shows whether the concept is related to the first or second digit. Firstly, consider the tasks Yo and y/1 For these tasks, we are essentially missing all the concepts we need for the ground truth rules; we can only predict True by learning the rule that the negation of concepts (in the used rule notation, the empty rule), which CMR does. Then, to be able to predict False, it learns some arbitrary rules. Secondly, consider the tasks y/2 y1o- As we are missing the concepts for digits and I, we cannot learn all ground truth rules for these tasks. For example, while CMR can (and does) still Iearn the rule y1 2 A 2, it is impossible to learn rule like 34 < A 3. Instead, CMR learns the rule ya + 3. which can be selected in case it needs to predict that + 3 4. Lastly, consider the tasks y11 to y1s- CMR finds all ground truth rules, as they can be found. We show the learned rules in Figure 7
CEBaB For predicting negative, CMR has learned as one of its rules the empty rule (rule 1), signifying that the other rules (prototypes) it has learned are unable to predict True for all training examples. Similarly, it has learned a rule that always evaluates to False (rules 5, 6 and 7, food has to be both good and bad). The remaining rules are prototypes. For instance, rule 2 signifies that for some examples if food is bad and service is not good, even though ambiance might not be bad and noise is unknown, the review is negative. Rule 3 signifies that even when food, ambiance and noise not bad, if service is not good, the review can be negative. We show the learned rules in Table 7
We show some of the learned rules in Table and some examples that satisfy these rules in Figure
CUB
First consider the rules for ne = 1. For each task, it learns the empty rule, which always be used to predict True, and two rules that together can always be used to predict False. As mentioned in the main text, these rules cannot be considered meaningful, but since only have one concept, it makes sense that CMR learns these rules. Consider now the rules for ncc = 12. Rules 1, and are rules that always evaluate to False, as Bald and Blond Hat air cannot be active the same time. The remaining rules form prototypes. For ne = 37, we also provide a part of the rulebook.
CelebA We show the learned rules in Tables 9 10 and 11
Table 6: Rulebook for C-MNIST.
even + 0AA (r) A (g)
even 2 (r) (g)
even t 4AA (r) A (g)
even < 6 (
even < 8 (
odd 1 ( (g)
odd < 3 A (r) A (g)
odd 5 5 (g)
odd < 7 (
odd 9 (
23

---

90 + 3A5
No 6A7A8A9
7A4
2
V7 <- 3A
910 + 5 A
No +
910 6 A
+ V1 c- 9
MY C A
- 7 3
VT
910 8 2
913 + 7
+
6
913 < 8 A 5
M2 + 2
2 A
911 2 6V
913 <-
U/ - 3 5
+- 3 8
914 < 5 A 9
V/ 5
911 7
914 <
5 6
914 X 7 A
Ma 4-
VS GA2
911 + 6 5AD
414 5 &
+ 2 A
911 7 AAL 4
314 < 9 G
M4 +
VS F
911 +- 8 E
911 A
915 + 7 A
A
Ms + 3 A
99 + 3
912 +- 3 A 4 4
915 &
1 5
915 < 9 6
Va 5
912 +- 5 7
NG +
+
99 AA3
912 6 A
9IG 7A
- A2
+- 7 7AS 5 912 8
+ 8 A
Ms +- 3 A
916
A
MG 2
917 <- 8 A
910 2
91a 4
317
910 - 3A7
+- 5 8
Mo +
910 4A6
913 6 7
918 < 9
(a)
(b)
(c)
(d)
Figure 7: Rulebook for MNIST+
Table 7: Rulebook for CEBAB.
negative
negative +- food unknown A foodd bad - good
-ambiance bod beeuiissrriosgoooo howp:/io__.ooo/tooeeeeeemmu noise_unknoun
noise bad
negative +- food bad A -ambiance, bad A -ssrviie__epo gooo noise_ustee wateee__nnowmmmmmmm -moiac had A -noise_good
had Regative +- food unknown f food bodoobddddooooooooooooaooddddddd badA-ffoo__gooff (rfoodd_oodA-mmmiiaieee -awmbanee_poooossseeeeeeeee wwt.goooooooooonoooooooeee
moise_bed A -noise_good
aambiance,unknou anknown A ambiance, bad A ambiance good service anknown A servioe, bad service goriieeggood.. -moisse
(5) negative + food_un unknown A food_bas bad A food_good A ammiance
--moise_unkooon unknown -noise_bie bad noise_ _good
wood_unnoww A food_bad bad A food_good A =ambiance_umbnoun unknown A ambionce_ A mambiance_good A service unknown service bad service gorideegooddA_ -noi-e_oeeeuue unknown
negative +
-nosseennoonn noise_ bad noide_good negative + food_unknone unknown A food_bad A food_good -ambiance, unknown A -ambiance bed A momiincce service antrecee_nnkowwnnn A service_bee
-ambiance_unknoun
strricce_goode -noooe__uknounn A -noise_ noise_good
positive +
positive food bad nambience bad noise_ sarkee__nknownnnnc -noise_bad -noiee_good
(10) positive +- food unknown unkoouu A food bad -food_good A -service_good noise_umh fasse_unkoone unknown A -noise_baa bad A
mooee_goood
(11) positive -foo unknown A bad A -food_ nambiance_bad noise_unknoun A -noisc_bed A
moise_good
(12) positive food wnknown food bad food_good -amhiance unknown Sambiance had
nambionce_ good A service_ wnknown A service bad A service_ good A noise undeeow A -moise_ bad A noide_good
Table 8: Rulebook for CUB (not complete).
eye_black wel.eeenggboooot the www.aaaaaaoooaaaaaaaaa A white A back solid A solid A (bill booked scabard) A (wingggrey) A
Laysan_Albate Albatross
underparts
breast solid
preast_white throat
(uppeeeeurrrrrrrrrr porpprrrrrs..rrg//. (rppaaggrreereeeppporre (upperparts.aaac) Mlack) ((up)))(pprrparrt (aapprparssswhtte) (bock_)rowx) gre)) (back [forehead_ Mack)) (forehead_ whitee) (under_tasl casddrrttts__aacee baakk) wunder_tall_uhee Corimary.white
(upper_tail_ (aii__hitte)
(nape_white) (sise_medlium_]]] (0_.mmiiimmm_____.___... (9]16_ A (taiLsolid) A (primery.grcy) A pprimarg.blaet (ball (aak))Mack) A (bill_buff) (srown_Mack)) A (crown_whste) A (wing_solid)
(2) Albay_As +
bill_kooked seabird A wing_arey A upperparts A breast_solid A upper upper_tail grie A black A bell_length__aeee about_ the_ttteee_e_eeeeeeeeeeeeeeeeeee as head A forehead black A under_tail tail grey A under tasl dasl_biacke black
throst A eye, black vir__medium_(__11..
back_solid tas__oolid belly_solid leg_grey bill_bieck crokn_Mack wing_aolid
(3) Brewer Blackbird
wing.black belly_ solid black purpppoe)))) A (wing_whitee
Apperparts_Mack A soeid A eye_black A under_tail bail_ttallblck A (underparts black) (back blcc) (upper_tail_bien length shorter than head) black) A (wing_rounded, rounded wings) (size__mmall smaalmma____ (a___________) A..._nnn)AAAaai washape.perchinglleeee fforhhedd_bare (nope black) A (bakly
(head plain) (breast Mtck)A (Dall length about the same as head)
solid) saai_s (primary, black) (leg. Mack) (crown (crown_Macc) A (wing_rolid)
24

---

(a) Laysan Albatross wuudepparss white A breast_ solid A breast white A throal, white A eye black bill length, about the some head belly welly_while while back solid belly solid (bill hooked seabird) A (wingggrey) grey) A (wing wiitee A (upperparts perls_ A (upperparts black) (upperparts_wile (...)
(b) Sooty_ Albbtrrssssoss < bill hooked hooked_seahird seabird wing A upperparts_po _eaeee A breast, breast_solid upper upper_tail ttill tail_grey A throat chroat_black A eye black A bill bill_length weelllengabbuutttt cacchhhabot__thtt facuttffh__ampae sas___am_asssss head A forchead black A under tail grey A under tail black A cise__medium_. (9 [9_16 (6]))) A back solid A tail solid belly solid leg grey A bill black crown black wing solid
(c) Brewer Blackbird + wing block A upperparts black A breast_solide A eye_black A under_tar tail Abell__seelysoddrrr soliddAbbl_blaackA((ii blackA(billlaall ( parpose) (wing_wwite ( (underports__blan (back_ bacckblack) A (upper_tate tail_blackk) (bill heilllengh__at
(head_ploin) Bennghhabbutt___. about the_ttteeeeeee sa__same__tt as as_hhed) (...)
(breast_black)
tthroat_black,
Figure 8: Selection of training examples satisfying learned rules for CUB. For brevity, drop some of the irrelevant concepts and replace them with (...).
Table Rulebook for CelebA (nc 1) (seed 1).
Black Hair +-
Black Hair 5_0 Clock_ Clock_Shadow Shadow
Black_Hair Hair + -5_o_Clock Clock_Shadoo
Male
Clock_Shhdoo Male + o_ Clock Shadow
Male -b_a__loan Ctock__Shaaon Shadow
Wavy Hair +
W Wavy_fe Hair 5_ Clock_Shocllll W Wavy_HH Hair +- o_ Clock
ClocklookSSkheee Shadow
25

---

Table 10: Rulebook for CelebA (nc = 12).
(I Black Hair +- S_o____oo____ Clock-lohaoaakk A Arched Egebrows A Bags_Under Uuder__dderrr Bold Bangs Big Láps A
Nose A. Blond_ N. Blurry Brown_Hoo Hoir Busky,
B Eyebrows Chabby
Black_ Hair
Sa.o_Clock_Shacee A -Bags Under _yes A -Bald A Lips -Big_Nose
Bushy Eychraws A -Chubby
Black Hair AAchedd_ Eaarcchod_EEybross A -Blond_ Hair
-Blurry
(1)
Mule + 5_o_Clock_ Chocdoohhhhhwww Arched_ Eyebrows A Under Eyes A Bald A Bongs Lips A Nose Blond_Hart A Blurry Brown_Haft Eyebrows Chubby
Male + 5_0_
A .Bags_Under U BBldA .Blond -BlurrgA Brown_Hairt
Chubby Mule - -Bal A -Blond_Hoe
0
W HairrHairr + S_o_Clock___ Slook_SShddvu A Arched_ Eycbroww Under Eges Bold Bangs Big_
Nose A Blond Hair A Blarry Brown_ Hair B Eyebrows A Chubby
Wany_Har Hair -5_0_ Clock Clock_Shadoww A -Bags__uuaae Under_ __ees_eesse A -Bald A -Big_Non -BuusyyBByyeeee Eychrows
Chubby
W Wayy_ Wavy_Hair Hair Arehedd Byeboou -Bald -Blond Hair A -Blurry A -Chubby
Table 11: Rulebook for CelebA (nc = 37).
(1) Black_Hair Hair S__oCClock Cllcc_Shheoow Shadow A Arched, Egebrous Bags Under EYes BaldA-Big Lip. &Cabbbyy Hair
-Double_Chor Chin A -EEyeglsses A -Groy Hair A Heavy Makeup A -Mouth Slightll Open A -No_ -No_Beard A
Face A Pale_Skin -Pointy_Nose WW Weoring_eeet Necktie
Recedking Hasriine -Roey__ Cheeks eWearing_ Necklaae
(2) Black Hair <- 5_o_Clock -_oo_GlockSShddn Shodow A Arched Eyebrows Bags_Under Under__deeeree
-Bold A -Big_ Lips A Lig_Nose A Blond_ Hair AaiirA-Bushae -Bushy_ Eyebrows A Chubby A-oublle -Dooobe_Chhinn Chin Egeglasses A Goatee Baack--GaaHHaaaa Hair A F
Heavy M_yyMMMr MakeupA
Etooutt_Sigghll.... Opeall__ppp..... A.enMMuun Mustached (Narrow_ Eyest, No_Beard Oaaaeoooeoaaaeeeaaaaa Face SkinA Pointy Nose A Recading Hairline Rosy Cheeks Straight Hair Wearing, Hat Wearing, Jápstick Necktie A Attractive A -Young oung
Weoring Weoring_Necklact Necklace A -W earrinn_eecttiiee
(3) Black_Har Hair + -Arched Eyebrowe
HHeayy_Makew Narrow Eyes A -Pale_Skin A -Pointy_Ne Nose A -Rosy Cheeks A A- Wearing.Fo Wearing.Earrings A -W -Wearing. Necklace
-Bangs -Blond Hair -Blurry
-Chuuby -Double Chin -EEyelasses A Gootee -Gray Hair -Musstcch -Narrow Eyes A -Pal-__sSii SSin A -Pointy_Nooe Nose A Becoeddin_Hoirrinn A A -W carrin__eechie
Black Hair +- -Bald A -Big Lips A -BBlrry
Rosyy.hhecks
Rosy, Checks A -Siideurns A -
(5) Male + S_o__lock__hhai
A Arched Eyebrows A. Bags_Under Under_Eyes A-esA-Balle A Lip_ A Bog_Nose A Arched_Eyebrous
Blond Hair A Blurry Bushy Eyebrows A Chubby -Doulle Chin Eyeglasses Goater A -Grrag Hair A Heasy_ Makeup High_Chhekknnn Cheekbonea A Mouth Mouth__Sightll, Open A Mustache A -Norrow Eyes A -No Beard
-No_d A Pointy Nose Receding Hairlined .Rosy_Cheekss -.Striggt_HHirrhh SeadirrreegiggHtae HatA W Wearing Lipstick W Wearing Neckioee --Weerring Necktie Attraction A -Young
Oval FaceA Paleeaaals__iii
(G) Male + 5_o_Clock_Sho ShackwwShwwww A Arched, Eyebrows A -Bags_Unaee Sadgs__ndee__evee A -Big Lips A Chubby A Double Chin -Eyeglasses -Goatee -Gray_ Hair High Cheekbones No_Beard -Posnty_Nose Receding Hairline A Ross -Rosy_Cheekke
Cheeks A "Sideburns A W Baakinngeecklceeee A - -Wearing_MM Necktie
(7) Male
-5_n_Clock SSaddoo A Arched Eyehrows A -Bags Under Under_Eyees Eyes A -Bald Bangs A Big Lips A Big_Nose A Brown Hair A. Bushy BashyyEyyerouss A -Chubby A -Double_CHHi _Chin A Eyeglasses A Goatee A Heasy_Makeup Makeup A Narrow, Narrow_Eeees A -No_Beard A Oval_ Face A Pale_Skin, A. Receding Hairline A -Wearing, Lipstick -WW-Wreree -Wearing_h) NNeck_eceeeeee -Wearing__eeeae _Necktte A -Y -Young (8) Male + -Bold A Bloond Hair A -lurry A -Gray_ A -Mustache A -aNorrow_BBaaa Eges SPale_SkinA -Rosy_ossseeeeeee Checks A -W earing_Necklace
Yearing_Earring
BaicedingHHairiiie
(9) Wavy_Hais < 5_ Clockk_Shaaooo A Arched, Eyebrows A -Bags .Bags Under__ndeeppe A Bangs A Lips A Arched_Eyebrons -Chubby A -Double_Chii
Brown Hair Maneky_Fyeeronn
Heany_Makeup Eyeglasses Goatee Heany Makeup A Sideburns A EeearinggEarring A
Recedang_Hairline No Beard A -Pointy_Nos A Receding, Hairline A
W Wearinn Weering_Neckinne Neckiace eariing_ Necktic
oung
(10) W Wavy_Hais
+ 5_0_Clock Clock_ Shadow A Arched, Eyeeroos A .Big Lips A Blond Hair A Blurry A BBoww Hair A A-Bushy -Baushh_Fyebrous A Chubby A -Double Chin A Eyeglasses -Gray Hair A Heavy Makeap
Beard A -Pointy Nose A Sesedding_Hsirinn A Straight_ Huir A. Caaring_eccttie.. Wearing Lipatick A -Wearingg -Woaring_Necklee Necklaee A -Wearing. Necktic A Attractive -Young
Mastuche A - -Narrow__pee Eyes A -No_Beard
(LI) Wavy_Has Hair
-Bags_ Under_ Eyes A -Bald -Big_Nose -Blurr BBushy_Evue Eyebrows A -Chubby A -Double_ Chinn A -Eyeglaeses A -Goatee A -Gray_Hod A -Muetuche A -Sidebarns SStraight_ttt
-b_o. Cwwckklooohhoooa
-ReeedinngHairllalnn Norrow Eyes_Eesss A -Pale_Skin
A - Bearing_Neckieen
W Werring_ff CearinngNeckkte Necktic
(12) Wany Hair c- -Rald A -Chubby -Manang--Mushhhhhh -Norrow Eyes -Pale Skin Receding
Hoirline RecedinngHooiiiiee
-Rogy_Cheeel A
Necklace
26

---

D Code, licenses and resources
For our experiments, implemented the models in Python 3.11.5 using open source libraries. This includes PyTorch v2.1.1 (BSD license) 48 PyTorch-Lightmin v2. 1.2 (Apache license 2.0), scikit-learn v1.3.0 (BSD license) 49 and xgboost v2.0.3 (Apache license 2.0). We used CUDA v12.4. Plots were made using Matplotlib v3.8.0 (BSD license) [50]. Our code is publicly available at https://gtt thub com wwwwdaww.iii.ceeccoo www.ddddddddoood CMR under the Apache License, Version 2.0.
All datasets we used are freely available on the web with licenses:
MNIST CC BY-SA 3.0 DEED,
CEBaB CC BY 4.0 DEED,
CUB - MIT License +
CelebA The CelebA dataset is available for non-commercial research purposes only
We will not further distribute them.
The experiments for MNIST+. MNIST+*, C-MNIST, CelebA and CEBaB were run on a machine with NVIDIA GeForce GTX 1080 Ti, Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz with 128 GB RAM. The experiment for CUB and the fine-tuning of the BERT model used for the CEBaB embeddings
machine with 17-10750H CPU, 2.60GHz 12, GeForce RTX 2060 GPU with 16 GB RAM. Table 12 shows the estimated total computation time for a single run per experiment.
Table 12: Estimated total computation time for a single run each experiment.
Experiment
Time (hours)
CelebA
3.7
CUB
1.8
CEBaB
0.1
MNIST+
4.1
MNIST+ rule int
0.2
MNIST+
4.1
C-MNIST
4.4
https://mlaa.ie.ccuu cuhk. edu, wtt.kkkww..kkkkk.ooojoccooooooaa dataset
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The structure of our model (neural nets interpretable memory), discussion (expressivity, interpretability and verification) and experiments (research questions) closely follow the claimed contributions in both abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
[Yes]
Justification:
This is done in our conclusion (Section g
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
28

---

Answer: [Yes]
Justification: Theorem 4.1 is proven in Section 4.1 and Theorem 5.1 is proven in Appendix
A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Limited details are given Section 6 with the remaining details provided in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
29

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code is provided at htttp:://github com/daviddeeottCCM The data is publicly available on the internet, and we do not redistribute it. This is mentioned in the main text.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
spllccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
These details are given in Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provide the standard-deviations. multiple seeds in all our tables.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
30

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer:
[Yes]
We reported the details of the machines exploited in the experiments in Appendix D
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:////w.w..aepcppppepe////eea
Answer: [Yes]
Justification:
Our research does not raise any ethical issue.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our conclusion clearly states the broader impact of our work with reference to interpretable and trustworthy AI.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
31

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We mentioned and respected all the licences of software and data in Appendix
D
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
32

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Documentation for the code is provided in the form of README file in the corresponding repository.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
33

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "298yuhJi9l", "reviewer": "Reviewer_ERBX", "summary": "In this paper, the authors propose Concept-based Memory Reasoner (CMR), consisting of (a) a concept encoder, (b) a rule selector, and (b) a task-predictor. CMR is tested on a few different datasets-- easy (MNIST+, MNIST+∗, C-MNIST), medium (CEBAB), and hard (CelebA, CUB) for task and concept prediction accuracy, discovery of rules, verifiability of test behaviour, and the possibility of concept and rule-based interventions.", "questions": "- How does the use of a 2-layer MLP with ReLU to get the input embedding and 3 hidden layers with ReLU activation to get the concept prediction let the model remain globally interpretable?", "limitations": "The authors discuss the limitations and the positive societal impact of their work, though it would be good to see some negative societal impact listed as well.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- CMR achieves similar or higher task accuracy on the range of datasets considered than existing concept bottleneck models considered on both complete and incomplete concept sets.", "weaknesses": "- What are the rules and where are they coming from in the rulebook? Also, how do we know that the rules being selected for an input are valid for the concepts present in it? I think the motivation behind selecting the rulebook and how they connect with the concepts can be explained better.\n- What is an example of \"rule 1\" and how does it differ from the decoded rule in Figure 2?\n- In the research questions posed in section 4, the authors mention they evaluate concept accuracy-- where are the ground truth concepts for this obtained from? Can you mention it somewhere?\n- What do you mean by the CMR being \"globally interpretable\"-- how is the rule selector interpretable?\n- Why are rules only seen as being conjunctive?\n- In figure 3, why are CMR, CEM, and the black box model not much affected in terms of task accuracy with a varying number of concepts, whereas the other methods are? Also, how does the number of concepts on the x axis relate to the number of missing concepts? How many total concepts are there in CelebA?\n- Why does CBM+linear achieve an accuracy of 0 on MNIST+ and MNIST+*?\n\nWriting:\n\nThe writing of the paper can be heavily improved in terms of claims/remarks/adjectives peppered throughout the paper such as \"CMRs are still foundational models\" (how? how do they relate with foundation models?), \"the first deep learning model\nthat is concept-based, globally interpretable and provably verifiable\" etc. \n\nAlso, it is not really clear to me what exactly the rulebook and the concepts are and where they come from. If these things can be clarified, and the writing be made clearer, I am willing to reassess the paper and increase my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to sincerely thank the reviewer for taking the time to read and review our paper.\n\n&nbsp; \n\n**Question: \"It is not really clear to me what exactly [...] the concepts are and where they come from. [For] concept accuracy, where are the ground truth concepts for this obtained from, and can you mention it somewhere?\"**\n\nWe refer to General Question 1 for the answer to this question. Additionally, some examples of concepts provided by the datasets can be found in the rules we provide (e.g. in Table 2). Some examples are \"good food\" in the context of restaurant reviews, \"black wing\" in the context of birds, and \"bald\" in the context of faces. Yes, we will mention this explicitly in the paper by adding the following sentence at line 255: \"All these datasets come with full concept annotations.\"\n\n&nbsp; \n\n**Question: \"What are the rules, and where are they coming from in the rulebook? What is an example of \"rule 1\" and how does it differ from the decoded rule in Figure 2?\"**\n\nIn Section 3.1.1 (lines 114-123), we explain that the model contains a rulebook, which is a set of learnable embeddings. Each embedding is decoded into a logic rule using a neural network. Therefore, **each embedding is a latent representation of a logic rule**. We will make this more clear by adding at line 123 the following sentence: \"This way, each embedding in the rulebook is a latent representation of a logic rule.\" **Each rule is a conjunction of (possibly negated) concepts** (e.g. \"red AND NOT soft -> apple\"), which we mention in lines 91-92, and can therefore be evaluated using concepts to predict a specific class. In Figure 2, \"rule 1\" is such an embedding, and the decoded rule is its logical representation that can be evaluated using concepts. \n\n&nbsp; \n\n**Question: \"Also, how do we know that the rules being selected for an input are valid for the concepts present in it?\"**\n\n**The learning process guides the development of concepts, rules and the selection**. For each input, the selected rule is logically evaluated over the predicted concepts to provide the task prediction. Thus, the rules and the selection are automatically learned in such a way that the evaluation of the selected rule maximizes task accuracy, i.e. minimizes cross entropy on task ground truth labels (in addition to being prototypes of the data). In other words, rules are developed to ensure “validity” for the current task prediction. The training objective is given in Theorem 5.1 and Equation 4.\n\n&nbsp; \n\n**Question: \"Why are rules only seen as being conjunctive?\"**\n\nA single rule is a conjunction of possibly negated propositions (literals). In CMR, it represents the body of a horn clause, i.e. conjunctive_rule -> task, which is a possible alternative definition for the task. For example, in “red AND round” -> apple”, “red AND round” defines what an “apple” might be. **As multiple rules are allowed in the rulebook (multiple definitions for the task), the resulting language is very expressive** (the probabilistic semantics of the selection of one among multiple horn clauses brings CMR close to the semantics of stochastic logic programs [1]).\n\n&nbsp; \n\n**Question: \"What do you mean by CMR being 'globally interpretable'? How is the rule selector interpretable? How does the use of [...] to get the concept prediction let the model remain globally interpretable?\"**\n\nThese questions correspond to General Question 3, and we refer to the answers given there.\n\n&nbsp; \n\n**Question: \"In Figure 3, why are CMR, CEM, and the black-box model not much affected in terms of task accuracy with a varying number of concepts, whereas the other methods are?\"**\n\n**For CMR and CEM, this is a key advantage over the remaining methods**, and we refer to what we stated in response to General Question 1. The black-box model does not use concepts for making predictions, as it is just a deep neural network; hence, its accuracy remains unaffected by the number of employed concepts and serves as a reference point for comparison with the concept-based models, which we mention in lines 266-267.\n\n&nbsp; \n\n**Question: \"[In Figure 3,] how does the number of concepts on the x-axis relate to the number of missing concepts? How many total concepts are there in CelebA?\"**\n\nThe number of concepts completely to the right (37) is the total number of concepts available for the CelebA dataset, and the number of concepts on the x-axis is the number of concepts employed in each model. We mention this in lines 248-249, and we will make this clearer by changing the figure's caption to \"Task accuracy on CelebA with varying numbers of _employed_ concepts.\"\n\n&nbsp; \n\n**Question: \"Why does CBM+linear achieve an accuracy of 0 on MNIST+ and MNIST+*?\"**\n\nThe intuition behind this result is to be found in the choice of the subset accuracy metric, as task prediction is modelled as a multilabel classification problem and this introduced a strong class imbalance in MNIST+(*). As some of the tasks are clearly non-linear, the linear prediction has a very low chance to make all 19 tasks correct for any example. \n\n&nbsp; \n\n**Question: What do you mean with \"CMRs are still foundational models\"?**\n\nThank you for asking this. With \"foundational\", we actually meant \"fundamental\", as CMR represents the first step towards a new class of interpretable CBMs that are globally interpretable and verifiable. We will replace the term as it better reflects what we meant.\n\n&nbsp; \n\n**Question: \"The authors discuss the limitations and the positive societal impact of their work, though it would be good to see some negative societal impact listed as well.\"**\n\nThank you for making this remark. This remark corresponds to General Question 2, and we will make the changes mentioned in its answer.\n\n&nbsp; \n\n[1] Cussens, J. Parameter Estimation in Stochastic Logic Programs. Machine Learning, 2001.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S8ezfuhLc3", "reviewer": "Reviewer_jRYi", "summary": "The authors presented a novel framework to explain image classification models, specifically with an explainable-by-design deep model. This model is built to provide interpretability through discovering logic rules that matches ground truths. Intervention towards modifying the rules that changes the interplay among the concepts can be really helpful to incorporate the user preferences. The proposed methods does not much on accuracy-interpretability performances compared to state-of-the-art concept bottleneck models.", "questions": "1) Did the authors check the time consumption for different datasets and architectures? While the rules are more useful for the experiments shown in the paper, it should not be hugely computationally expensive compared to the other concept based models.", "limitations": "The authors explained limitations, but missed explaining potential negative societal impact of this work. I would be happy to check author response on above comments and improve my score based on that.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The proposed method discovers logic rules consisting of ground truth concepts that matches the true logic rules as working mechanism of a predictor model.\n2) The concepts as well as the rules are allowed to intervene that incorporates expert opinion about the dataset specially when the model does not behave properly.\n3) The properties behind model predictions and explanation can be verified before model deployment that shows the efficacy of the model in a specific scenario or deployment environment.\n4) The experimental results prove the efficacy of the derived networks in terms of predictive performance measure, apart from generating nice explanations.", "weaknesses": "1) Line 49-52 explains some example rules for image classification task, which seems very simple. But, how are they going to work in practice for more complex scenarios? For example, an apple with a blue background can not decide the image not containing an apple just based on the concept 'blue' is active.\n\n2) This method does not seem nicely scalable to datasets with complex scenarios (such as above) containing huge number of concepts and critical rule interactions. I would request the authors to comment on that.\n\n3) How do you decide the number of optimal rules in the rulebook? Is it trying different numbers and choosing the number with the best accuracy performance? Or, is there any expert opinion involved?\n\n4) How the correctness of the rules should be verified for more critical datasets? Should there be always a human to check the rules at the end?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first want to sincerely thank the reviewer for taking the time to read and review our paper.\n\n&nbsp; \n\n**Question: Lines 49-52 explain some example rules for an image classification task, which seem very simple. How are they going to work in practice for more complex scenarios? For example, one cannot decide that an apple with a blue background does not contain an apple just based on the concept 'blue' being active.**\n\nThe provided example is intentionally simple to illustrate the basic mechanism, as in practice often more nuanced and specific concepts are used. For example, in the CUB dataset for bird classification, there is a concept denoting whether there is a \"solid tail\". Still, **while it is true that most CBMs' task accuracy heavily depends on the chosen concepts, CMR does not suffer from this problem.** This is due to the rule selector, which chooses which rule to apply for a given input; this allows even simple rules to make accurate predictions for complex scenarios. We also refer to what we stated in response to General Question 1.\n\n&nbsp; \n\n**Question: \"How do you decide the number of rules in the rulebook?\" Is it trying different numbers and choosing the number with the best accuracy performance? Or is there any expert opinion involved?\"**\n\nAs a consequence of Theorem 4.1, once the number of rules is 3 or larger, CMR can in principle achieve black-box accuracy (i.e. like a deep neural network) no matter the concepts. In response to this and one of your following questions, **we have added an experiment showcasing the robustness of CMR's accuracy to the number of rules**, for which we refer to the caption of Figure 1 in the attachment to the general response.\n**Rather than accuracy, setting the number of rules influences the specificity and granularity of the learned rules.** The more rules, the more fine-grained they will be. This also impacts on the intervention capabilities of the human: fine-grained rules may allow for very targeted modifications of the model's behaviour with rule interventions. While it is not required for the working of the model (i.e. for obtaining good accuracy and meaningful rules), human interaction can be needed for finding the preferred granularity.\n\n&nbsp; \n\n**Question: \"How should the correctness of the rules be verified for more critical datasets? Should there be always a human to check the rules at the end?\"**\n\n**The human should not necessarily go over all the rules, but could just automatically verify whether the model satisfies some criteria for their specific use case** (e.g. whether a constraint is satisfied). This is because rules serve a double scope: (1) they form an interpretable prediction system that a human can indeed inspect; (2) they form a formal logic system for prediction, that an automatic verifier (e.g. model checker) can use.\n\n&nbsp; \n\n**Question: How does the method scale to datasets with complex scenarios containing a huge number of concepts and critical rule interactions? Is CMR not too computationally expensive compared to other CBMs?**\n\n**As indicated in Theorem 5.1, making a task prediction using CMR is computationally linear in the number of concepts (like other CBMs) and rules.** However, we acknowledge that if the number of concepts and/or rules becomes excessively large, it is possible that CMR's prediction would be deemed too slow. For the dependency on the number of concepts, we refer to the answer of General Question 1, where we discuss that, in contrast to most CBMs, the number of concepts in CMR can be reduced without harming accuracy. For the dependency on the number of rules, we have shown with Theorem 4.1 and with the additional experiment of Figure 1 in the attachment to the general response that CMR can achieve the same accuracy with very few rules as with many rules. Therefore, reducing the number of rules can speed up CMR, with the trade-off being a potential decrease in interpretability, while accuracy remains robust. **So, while there is a linear increase in complexity, this is a reasonable price to pay for the added interpretability, especially since this complexity can be tuned as desired.**\n\n&nbsp; \n\n**Question: \"The authors explained limitations, but missed explaining potential negative societal impact of this work.\"**\n\nThank you for mentioning this. This remark corresponds to General Question 2, and we will make the changes mentioned in its answer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KywfvxEJjZ", "reviewer": "Reviewer_TwPt", "summary": "Concept learning is a very important and current area of research. The paper is motivated from this perspective of neurosymbolic concept learning. However, there are a number of flaws. The paper refers to rules and \"reasoning\" informally; it does not define syntax and semantics neither does it define the reasoning process. This makes it very difficult to check the claims of verifiability and even to understand what exactly is meant by the notation used in the paper e.g. in the proof of Theorem 1 which is trivial. Claims are made of soundness, interpretability and even intervention which are not substantiated. The experimental results do not help clarify any of the issues. The rules provided are difficult to interpret and unrelated to the motivation of the paper, e.g. instead of concepts some rules refer to different noises.", "questions": "How do you handle the explosion of exceptions with the use of negation, e.g. Red and Not Square implies Apple. Why only Not Square? Why not also Not Triangular, Not Oval and the list of all the other properties that do not change? This is known as the frame problem.\n\nSince the syntax of the logic is not specified, we don't know the difference between e.g. the left arrow and the right double arrow. Also, how is equality used in the language?", "limitations": "There are far too many claims to do with verification, interpretability, reasoning that are not substantiated, i.e. not formalized properly or backed by experimental results.", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "Concept learning and neurosymbolic AI are relevant current themes.", "weaknesses": "The notation is not formalized and the experimental analysis is limited in scope.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first want to thank the reviewer for taking the time to read and review our paper.\n\n&nbsp; \n\n**Question: \"The rules provided are difficult to interpret and unrelated to the motivation of the paper, e.g. instead of concepts some rules refer to different noises.\"**\n\n**In all rules, we only refer to concepts.** We believe you refer to Table 2, where some of the concepts are \"noise_g\" and \"noise_b\". These are concepts, and in the caption of the table we state that \"g\" and \"b\" are abbreviations for \"good\" and \"bad\". In the CEBAB dataset, the task is to classify restaurant reviews as positive or negative, and these concepts denote whether the noise in the restaurant is good or bad. We mention this setting of CEBAB in Section 6.1 (lines 251-252). \n\n&nbsp; \n\n**Question: \"Since the syntax of the logic is not specified, we don't know the difference between e.g. the left arrow and the right double arrow. Also, how is equality used in the language?\"**\n\nAll the connectives are standard logic connectives. The \\leftarrow is used as assignment (or a computation direction), as stated in natural language in the sentence introducing the formula (line 129). We can definitely add a sentence to avoid confusion. The purely propositional logic expression for the task prediction is stated in Eq. 1, where only standard operators are used. The equality symbol is used as an indicator function for the role r_ij being positive (P), negative (N) or irrelevant (I). Such terms can be considered ground terms, avoiding exiting the propositional setting. We can change e.g. r_ij = P into r_ij^P to avoid using equality.\n\n&nbsp; \n\n**Question: \"How do you handle the explosion of exceptions with the use of negation, e.g. Red and Not Square implies Apple. Why only Not Square? Why not also Not Triangular, Not Oval and the list of all the other properties that do not change? This is known as the frame problem.\"**\n\n**The frame problem is addressed by the loss we use in the paper** (Eq. 4). Our loss aims at learning rules that are as close as possible to the seen concept activations during training, given the limited number of rules of the model. We explain our reasoning behind this in Section 4.2 (\"Interpretability\"): we consider rules to be meaningful if they are prototypes of the data, which follows standard theories in cognitive science (line 160). Some other CBMs do suffer from the frame problem (e.g. DCR [1]) and the added regularization loss solves their issues. \n\n&nbsp; \n\n**Question: \"There are far too many claims to do with verification, interpretability and reasoning that are not substantiated, i.e. not formalized properly or backed by experimental results.\"**\n\nWe refer to the response to General Question 4.\n\n&nbsp; \n\n[1] Barbiero et al. Interpretable neural-symbolic concept reasoning. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r3ZYUidffZ", "reviewer": "Reviewer_VcGf", "summary": "This paper presents an extension of Concept Bottleneck Models by modeling task predictions as neural selections over a learnable memory of rules, that is jointly learned during the training phase together with all the other model parameters. The resulting model is fully differentiable and can be exploited not only for its interpretability but also for the formal verification of specific properties. Experiments are conducted on a set of well-known benchmarks for CBMs, aiming to assess generalization, explainability and verifiability of the model.", "questions": "* Why doesn't the graphical model in Figure 1 contain also the edge from c to r? The hypothesis here seems that c is independent from r given x and y?\n\n* Wouldn't it be possibile to use a (probabilistic) symbolic system to learn the rulebook for the memory, since the concepts are fully supervised during training? What is the advantage of the proposed approach? I understand that the system is fully differentiable, but maybe at the cost of higher complexity?", "limitations": "-- The proposed approach could be compared against some neural-symbolic system or some probabilistic symbolic system (ProbLog, DeepProbLog, Probabilistic Soft Logic, maybe even Aleph) to be used in place of the rulebook learner. That would show better the advantage of the use of the memory component and the joint learning.\n\nTypos and minor corrections:\n- \"i.e.\" -> \"i.e.,\"\n- \"boolean\" -> \"Boolean\"", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Novel and interesting extension of Concept Bottleneck Models \n+ Combination of neural and symbolic approaches for rule learning and concept prediction\n+ Wide experimentation on several benchmarks\n+ Clarity of presentation", "weaknesses": "- No comparison with other neural-symbolic systems (see questions and limitations below)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first want to sincerely thank the reviewer for taking the time to read and review our paper. We are pleased that the reviewer considered CMR an interesting and novel extension of CBMs, and considered the presentation clear.\n\n&nbsp; \n\n**Question: The proposed approach could be compared against some neural-symbolic system or some probabilistic symbolic system (ProbLog, DeepProbLog, Probabilistic Soft Logic, [...]**\n\nThank you for this interesting remark. Since CMR is a probabilistic model rather than a system, **we can directly implement CMR in a system like DeepProbLog**, as DeepProbLog provides logical and probabilistic semantics, and can incorporate neural networks through the use of neural predicates. Moreover, as the bodies of our rules are essentially bodies of horn clauses, the probabilistic semantics of the selection of one among these horn clauses brings CMR close to the semantics of stochastic logic programs [1]), and therefore, to the neurosymbolic system DeepStochLog [2]. We chose to avoid using these explanations in the main text to keep the presentation and notation as simple as possible.\n\nFor a direct empirical comparison with more \"standard\" neurosymbolic models defined in the mentioned systems, it's important to first note the **fundamental differences in datasets typically used for neurosymbolic models versus CBMs**. In the former, the knowledge (logic rules) on how to predict the task given the concepts is typically provided, while there are no concept labels. In the latter, the knowledge is missing, but concept labels are provided. For this reason, we make our comparison with other CBMs.\n\n**Question (continued): […] maybe even Aleph**\n\nIn our experiments, we made the **conscious decision to compare with a CBM that uses a decision tree** as symbolic rule learner for task prediction, as opposed to a CBM that uses other rule learners such as ILP systems like Popper or Aleph, since ILP shines in the relational setting (as opposed to the tabular / propositional one of CBMs) [3].\n\n&nbsp; \n\n**Question: \"Wouldn't it be possible to use a (probabilistic) symbolic system to learn the rulebook for the memory, since the concepts are fully supervised during training? What is the advantage of the proposed approach? I understand that the system is fully differentiable, but maybe at the cost of higher complexity?\"**\n\nThank you for this very interesting question! The integration with symbolic learners is definitely a very interesting and different direction of neurosymbolic learning that we are planning to investigate. Actually, **we have a preliminary indication of this potential with one of our experiments** (lines 293-299), where we showcase that manually adding rules to the memory can improve the model that is being learned regarding the interpretability of the learned rules. We mention this possibility of manually adding rules to the rulebook in lines 181-192, and these rules can come from human experts or indeed from symbolic learners.\n\nThe **advantage of the end-to-end rule learning is that the joint optimization will make CMR learn rules that are accurate in combination with the selector**. Importantly, these rules need not necessarily be accurate on their own, but they need to be accurate when used with the selector. This can be seen by considering the proof for Theorem 4.1: while these rules do not form a universal binary classifier on their own, they do achieve this when used in conjunction with the selector.\n\nAs a response to this question, we have made **an additional experiment where we extract the rules learned by a decision tree and use them in multiple ways in CMR, to show the advantage of the rule learning component**, as explained in the caption of Figure 2 in the attachment to the general rebuttal. We show the effect on accuracy of (1) completely swapping out the rule learning component with the pre-obtained rules from the decision tree, and (2) adding the pre-obtained rules while also keeping the rule learning component. We used a decision tree as rule learner.\n\n&nbsp; \n\n**Question: \"Why doesn't the graphical model in Figure 1 also contain the edge from c to r? The hypothesis here seems that c is independent of r given x and y?\"**\n\nIn the graphical model, r represents the rule that is selected for evaluation. The task y depends on both r and c since y is the outcome of evaluating r using c. We indeed model c as conditionally independently of r given the input x. While modelling a dependency from c to r is certainly a reasonable design choice, we decided to not include this edge. This decision is based on the fact that the information present in the concepts is extracted from x, and the edge from x to r suffices to capture these possible dependencies.\n\n&nbsp; \n\nWe also want to thank you for mentioning typos in the text; we will correct them.\n\n&nbsp; \n\n[1] Cussens, J. Parameter Estimation in Stochastic Logic Programs. Machine Learning. 2001.\n\n[2] Winters et al. DeepStochLog: Neural Stochastic Logic Programming. 2022.\n\n[3] Muggleton, De Raedt. Inductive Logic Programming: Theory and Methods. 1994.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We first thank the reviewers for their insightful feedback. It has certainly improved the quality of our manuscript, and we hope we have been able to address all the raised concerns in this rebuttal. We reply to questions shared by multiple reviewers in this comment, and reply to specific questions in comments under their respective review.  

&nbsp;

# Additional experiments 

In response to some of the questions, we have performed two additional experiments (see the attached PDF for figures) with our model CMR.

- (@Rev-jRYi) **The first experiment shows the robustness of CMR's accuracy with respect to the number of rules it is allowed to learn.**

- (@Rev-VcGf) **The second experiment expands upon CMR's rule interventions, and also serves as an ablation study on the rule learning component.**

&nbsp;

# Answers to common questions

We paraphrase the shared questions and provide answers:

&nbsp;

**General Question 1 (@Rev-ERBX, @Rev-jRYi): "Where do the concepts and their supervision come from? Can the choice of concepts harm the accuracy of CMR and competitors?"**

- **The standard protocol in CBM literature is that concepts and their supervision are part of the problem (i.e. the dataset).** As the goal of such models is to be interpretable by an end user, concepts need to be designed as a communication language between the model and the users. Supervision needs to be provided as a form of alignment between user and model, as they need to assign the same meaning to the concepts.
- **Most CBMs have the significant limitation that their accuracy depends on the employed concepts, reducing performance w.r.t. deep neural networks** (black-box models). Therefore, decreasing the number of employed concepts is a trade-off, improving the computational efficiency, conciseness of the explanations, and effort from human annotators, but possibly harming accuracy and interpretability.
- **CMR can achieve black-box accuracy regardless of the employed concepts** (as a consequence of Theorem 4.1, and as shown empirically, e.g. in Figure 3) **_and_ CMR is globally interpretable** (as all decision rules are fully accessible, mentioned in lines 150-151). Thus, CMR removes the accuracy from the aforementioned trade-off: changing the number of concepts affects in general only its interpretability. While some other CBMs can achieve black-box accuracy regardless of the concepts, they lack global interpretability (e.g. CEM [1], lines 320-321).
- Therefore, CMR improves on the interpretability-accuracy trade-off w.r.t. other CBMs: (1) **among CBMs that can obtain black-box accuracy regardless of the concepts, CMR is the most interpretable, being the only globally interpretable one**, and (2) **among CBMs that are globally interpretable** (e.g. CBMs using logistic regression and CBMs using decision trees), **only CMR can obtain black-box accuracy regardless of the concepts.**

&nbsp;

**General Question 2 (@Rev-ERBX, @Rev-jRYi): "[...] Potential negative societal impacts would be welcome as well."**

As the human has direct access to the rules of CMR, the human can help resolve unfairness and bias by removing or changing such rules. However, the opposite is also possible: the human can add unfairness and bias this way. To provide a more nuanced view on the potential societal impacts, we will include this potential negative societal impact in the conclusion at line 346.

&nbsp;

**General Question 3 (@Rev-ERBX): "What do you mean by CMR being 'globally interpretable'?**"

Our interpretation of global interpretability follows the standard interpretation for CBMs. There are two terms involved: _interpretable_ and _global_. (1) CBMs are “intrinsically interpretable” models [4, 5] as they make explicit which human-interpretable concepts they use to make predictions. (2) A model is globally interpretable if the user "can interpret entire classes or sets of examples [...] and not just explain individual data inputs" [2, 3]. **CMR uses concepts to make its prediction, thus being _interpretable_, and, differently from most other CBMs, it allows inspecting the rules as descriptions of entire classes of examples and not on a “per-example basis”** (line 327)**, thus being _globally_ interpretable.**

&nbsp;

**General Question 4 (@Rev-TwPT): "Are the claims regarding interpretability, verifiability, intervenability and accuracy substantiated formally and/or empirically?"**

For the claim that CMR is a globally interpretable model, we refer to General Question 3. We evaluate the interpretability of the learned rules quantitively on 2 datasets (lines 284-288, Table 2) and qualitatively for all datasets (lines 289-291 and 707-746, Tables 2 and 5-11, Figures 5-6).

The claims made of having a verifiable task predictor follow from the fact that the task predictor has an equivalent logic formulation (Eq. 1). We explain how model properties can be verified (lines 194-204), and we have a verification experiment (lines 303-312).

We claim that with rule interventions during training, the human can directly shape the model that is being learned, which we explain in Section 4.2 (lines 181-192) and show with an experiment (Section 6.2.2, lines 293-299, Table 3). In Section 6.2.2 (lines 300-301), we also claim that CMR is responsive to concept interventions, shown with an experiment in Appendix C.3.1. 

We have given a proof (line 142) that CMR is a universal binary approximator (Theorem 4.1). We show with our experiments (lines 270-282, Table 1, Figure 3) that this allows CMR to obtain similar accuracy to black-box models, regardless of the concepts.

&nbsp; 

[1] Zarlenga et al. Concept embedding models. 2022.

[2] Kim et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors. 2018.

[3] Doshi-Velez & Kim. Towards a rigorous science of interpretable machine learning. 2017.

[4] Molnar, Christoph. Interpretable machine learning. 2020.

[5] Koh et al. Concept bottleneck models. 2020.

Meta-review: While most reviewers agree that the contribution (a concept-bottleneck model that includes a rule learning top layer) is valid and the empirical results support most of the claims made by the authors, the overall post-rebuttal rating is still somewhat borderline, with one reviewer being strongly against acceptance.

----

In order to get a better sense of the quality of the contribution, I have read the paper myself.  The paper is well written an presented.  The experimental setup is also okay, with one exception.  In fact, while I generally like the proposed architecture, I also think the paper has a serious issue that I feel obliged to voice here, broken down into three points:

- line 74: "we introduce Concept-based Memory Reasoner (CMR), the first deep learning model that is concept-based, globally interpretable and provably verifiable".  This statement is incorrect - many neuro-symbolc models comply with these desiderata; the authors actually reference a few of them in the paper - and should be either deleted or drastically toned down.

- In the Related Work, the authors correctly identify the strong ties between CMR and NeSy architectures, but ignore NeSy models that learn both the neural predicates and the rules.  A couple of examples are:

  Daniele et al. "Deep symbolic learning: Discovering symbols and rules from perceptions", 2022.
  
  Tang et al. "From perception to programs: regularize, overparameterize, and amortize", 2023.
  
There may be more recent works that I am unaware of.

- CBR is not compared against NeSy approaches that support rule learning.  It is unclear how DCR relates to them, performance wise (the DCR paper does not seem to acknowledge them at all, at least from my reading).   I think this substantially complicates assessing the novelty and significance of the contribution.  This aligns with reviewer VcGf opinion (although they were quite lenient about this issue).

----

UPDATE: I've thought about this paper for a while.  I think there is merit in the proposed approach.  The experimental setup is also quite convincing.  But, all in all, the paper should better position CBR against existing works from the NeSy AI camp. (The main differences being that NeSy approaches usually do not leverage supervision on the neural predicates; this would give CBR and edge.)

While I have been leaning towards rejection for a while, I think the paper contains enough new, useful and solid material to warrant acceptance, provided the authors address the reviewer's feedback (and my own) in the camera ready version.
Decision: Accept (poster)