Factor Graph-based Interpretable Neural Networks
Authors: Yicong Li
Kuanjiu Zhou
Shuo Yu
Qiang Zhang
Renqiang Luo
Xiaodong Li
Feng Xia
Venue: ICLR 2025 Poster (2025)
Keywords: interpretable neural network
factor graph
perturbation
explanation rectification
graph learning
Primary area: interpretability and explainable AI

Abstract: Comprehensible neural network explanations are foundations for a better understanding of decisions, especially when the input data are infused with malicious perturbations. Existing solutions generally mitigate the impact of perturbations through adversarial training, yet they fail to generate comprehensible explanations under unknown perturbations. To address this challenge, we propose AGAIN, a factor graph-based interpretable neural network, which is capable of generating comprehensible explanations under unknown perturbations. Instead of retraining like previous solutions, the proposed AGAIN directly integrates logical rules by which logical errors in explanations are identified and rectified during inference. Specifically, we construct the factor graph to express logical rules between explanations and categories. By treating logical rules as exogenous knowledge, AGAIN can identify incomprehensible explanations that violate real-world logic. Furthermore, we propose an interactive intervention switch strategy rectifying explanations based on the logical guidance from the factor graph without learning perturbations, which overcomes the inherent limitation of adversarial training-based methods in defending only against known perturbations. Additionally, we theoretically demonstrate the effectiveness of employing factor graph by proving that the comprehensibility of explanations is strongly correlated with factor graph. Extensive experiments are conducted on three datasets and experimental results illustrate the superior performance of AGAIN compared to state-of-the-art baselines.

Published as a conference paper at ICLR 2025
FACTOR GRAPH-BASED INTERPRETABLE NEURAL NETWORKS
Yicong Li', Kuanjiu Zhou', Shuo Yu Qiang Zhang': RRnniang Luo", Xiaodong Li Feng Xia Dalian University of Technology, Jiiin University, 'RMIT University
ABSTRACT
Comprehensible neural network explanations are foundations for better under- standing of decisions, especially when the input data are infused with malicious perturbations. Existing solutions generally mitigate the impact perturbations through adversarial training. yet they fail to generate comprehensible explanations under unknown perturbations. To address this challenge, we propose AGAIN, a factor graph-based based interpretable neural network, which is capable of generating comprehensible explanations under unknown perturbations. Instead of retraining like previous solutions, the proposed AGAIN directly integrates logical rules by which logical errors in explanations are identified and rectified during inference. Specifically, construct the factor graph to express logical rules between expla- nations and categories. By treating logical rules as exogenous knowledge, AGAIN can identify incomprehensible explanations that violate real-world logic. Further- more, we propose an interactive intervention switch strategy rectifying explana- tions based on the logical guidance from the factor graph without learning per- turbations, which overcomes the inherent limitation of adversarial training vraining-based methods in defending only against known perturbations. Additionally, we theoret- ically demonstrate the effectiveness of employing factor graph by proving that the comprehensibility explanations strongly correlated with factor graph. Exten- sive experiments are conducted three datasets and experimental results illus- trate the superior performance of AGAIN compared to state-of state-of-the-ar -tte-arrt baselines
INTRODUCTION
Comprchensibility neural network explanations depends on their consistency with human insights and real-world logic. Comprehensible explanations promote better understanding of decisions and establish trust in the deployment of neural networks in high-stake scenarios, such healthcare and finance (Virgolin & Fracaros 2023 Fokkema et al., 2023 Luo et al., 2024a|b) However, as shown Figure interpretable neural networks are vulnerable to malicious perturbations which are infused into inputs, misguiding the model to generate incomprehensible explanations (Tan & Tian. 2023 Baniecki Biecek] 2024). Such explanations may cause users to make wrong judgments, resulting in security concerns in high-stake domains. For example, a doctor prescribing medication based on a medically illogical explanation (i.c., incomprehensible explanation) of the pathological prediction may lead misdiagnosis. Therefore, it is crucial ensure that interpretable neural networks generate comprehensible explanations under perturbations.
Several existing efforts have been devoted to investigating comprehensible explanations (Kamath jet al., 2024] Sarkar et al. [221 Chen et al. 2019). Many of them craft adversarial samples by adding perturbations to the dataset beforehand and retrain the model with extra regularization terms. Regularization terms constrain model to generate explanations that are similar to the explanation labels of the adversarial samples. Empirical results show that retrained models are able to learn the adversarial sample data distribution and reduce the probability of being misled by the predetermined perturbation.
However, the above solutions assume perturbations are known to the model, which leads to their failure to generate comprehensible explanations under unknown perturbations (Novakovsky et al.)
"Corresponding authors: shuo. youuo.yu@ii ieee.org, zhangg@dlur.edd..o Source codes available at httppp/s//giiitbb..o///////puupp

---

Published as a conference paper at ICLR 2025
The reasons as follows: 1) it is impossible to craft adversarial samples for all unknown perturbation types (Gürel et al. 2021; 2) even the adversarial samples are available, retraining is effective for only limited number of perturbation types simultancously (Tan & Tian 2023). Thus, despite recent progress on comprehensible interpretability, is still challenging to provide compre- hensible explanations under unknown perturbations. Considering this, seek to solve this problem from a different perspective- instead of optimizing the training strategy, we innovate the inference process. Our goal is design an interpretable neural network capable of rectifying incomprehensi- ble explanations under unknown perturbations during inference. We draw inspiration from knowl- edge integration with factor graphs. Unknown perturbations cause the explanatory semanties to violate the exogenous knowledge in the factor graph (Tu et al. [223] Xia et al.) 202T). Factor graph reasoning enables us to identify and rectify explanatory logical errors without learning perturbations.
2023).
We propose AGAIN (fActor GrAph- based Interpretable neural Network), which generates comprehensible concept-level explanations based on the factor graph under unknown perturbations (Tiddi & Schlobach] 2022). AGAIN consists of three modules, including factor graph construction, explanatory logic emrors identification, and explanation rectification. In the first module, semantic concepts, label catergories, and logical rules between them are encoded as two kinds of nodes (i.e., variable and factor) in the factor graph, while their correlations are encoded as the edges. Based on the constructed factor graph, the logic relations among concepts and categories are explicitly represented. In the second module, AGAIN generates the concept-level explanations and predictive categories and then imports them into the factor graph to identify erroneous concept activations through logical reasoning. In the third module, we propose an interactive intervention switch strategy for concept activations correct logical errors in explanations. The explanations that are further regenerated align with external knowledge, The regenerated explanations are used to predict categories. Extensive experiments are conducted on three datasets including CUB, MIMIC-II EWS, and Prathetic-MNIST Experimental results demonstrate concept-level explanations generated by the proposed AGAIN under unknown perturbations have better comprehensibility compared to baselines such as ICBM, PCBM, free CBM, and ProbCBM.
Winge
Tosl
X
Input
Explanation
Phane
Brrd
Interpretable
Dog
Prediction
User
Perturhations
Neural Network
Figure 1: Interpretable neural networks suffer from pertur- bations that generate incomprehensible explanations. For in- stance, the model predicts the input as "Dog" but explains with "Wings" and "Plume".
Our contributions can be summarized as follows: 1) against unknown perturbations: we present an innovative interpretable neural network based on factor graph. It integrates real-world logical knowledge to generate comprehensible explanations under unknown perturbations: 2) forward feed- back: we design logic error identification and rectification methods based the factor graph. Our method is able to rectify logic violating explanations during inference without learning perturba- tions, unlike previous methods; 3) theoretical foundation of factor graph: prove that the com- prehensibility of explanations is positively correlated with the involvement of factor graph; 4) supe- rior performance: we conduct extensive experiments three datasets to demonstrate that AGAIN can generate more comprehensible explanations than existing methods under unknown perturba- tions.
2 RELATED WORK
Comprehensible Explanation under Perturbation. Studies of comprehensible explanations un- der perturbations can be divided into two categories: attacks comprehensibility and defenses of comprehensibility. Studies of attacks on comprehensibility aim to design perturbations that misguide the model to generate incomprehensible explanations. Some methods modify salient mappings with perturbations that make the explanation incomprehensible to users (Ghorbani et al. 2019 Dom- browski et al. (022). Furthermore, there are several efforts that propose additional types of pertur- bations (Rahmati et al., 2020 Carmichael & Scheirer 2023 Huai et al., 2022). They demonstrate that many types of perturbations undermine the comprehensibility of explanations. In contrast,
2

---

Published as a conference paper at ICLR 2025
studies on defenses of comprehensibility aim design defensive strategies to suppress the effects of perturbations on interpretations. These studies focus on adversarial training of interpretable neu- ral networks so that the model generates comprchensible explanations despite perturbations. These studies are implemented in two ways. In the first approach, some methods annotate the adversarial samples with explanatory labels, and constrain the model to generate explanations similar to the labels (Boopathy et al., 2020; Lakkaraju et al. 2020 Chalasani et al.) (2020). While promising, excessive adversarial training can easily lead overfitting, In the second approach, some efforts further utilize different regularization terms based on adversarial training to mitigate overfitting, and allowing reasonable local shifts in explanations (Kamath al., [2244] Sarkar al.) [021] Chen al,
2019). In addition, concept-based interpretable methods, which explain model decisions by gener- ating set of high-level semantic concepts, have gained great attention recently Koh et al. (2020b);
Havasi et al. )2022); Wang et al. (2022). Moreover, it has been demonstrated that concept-based explanations can be erroncous and lose comprchensibility under perturbations |inha et al. (2023).
Mcanwhile, they verify that retraining is effective in enhancing the comprehensibility of concept-
based explanations. However, all the above methods assume that the perturbation is known to the model. Thus, how improve the comprehensibility of the explanation under unknown perturbations remains open.
Knowledge Integration with Factor Graph. There have been extensive studies knowledge integration with factor graphs (Tian et al. 2024 Güirll et al. 2021 Yang et al. 2022). 'These studies typically utilize factor graph reasoning assemble predictions from multiple ML. models. When one model predicts incorrectly, the factor graph can combine the exogenous knowledge to correct the error based on the predictions of other models. Empirical evidence suggests that integration of exogenous knowledge in factor graphs contributes to the predictive accuracy of ML models. In this paper, instead of improving predictive accuracy, we explore the possibility of using exogenous knowledge to guide interpretable neural networks for generating comprehensible explanations.
3 NOTATIONS AND PRELIMINARIES
Interpretable Neural Network. Interpretable neural networks are defined as neural networks that automatically generate expla- nations for decisions (Esterhuizen et al. 2022, Rieger et al.] (2020] Peng et al.) 2024). For more comprehensible explanations,
Factor Variable
uti-
f
lize concept bottleneck model to generate concept-level explana- tions, which utilize various semantic concepts to explain the pre- dictions Koh et al. (2020b); Huang et al.] (2024). Specifically, let denote input sample, the concept bottleneck model predicts the category y and outputs boolean vector с E {0, 1] of M concepts. Let e E denote a concept. Let c 1 indicate that concept is present in z and influences the model decision. e is the concept-level explanation of the model prediction.
V
Figure 2: An example of the factor graph. It consists of factors and 8 variables.
Factor Graph. Factor graph serves as probabilistic graphical model to depict relationships among events (Yu et al., 2023 Bravo-Hermsddorff et al., 2023). As shown in Figure 2 within factor graph, two node types exist: 1) variables, which delineate events; 2) factors, which articulate the relationships between events. Formally, a factor graph G = (V,F) contains the set of variables V and the set of factors F. We denote the set of edges as For any V, E V and fi E F, we let (v,, fi) E denote an edge of G. Let (ff) {u; [1, fi) E} denote the set of neighbors of factor ft in G. We let variables correspond to concept and category labels. We let factors correspond logical rules. This enables encode logical rules between concepts and categories.
Known and Unknown Perturbation. Formally, let å denote perturbations uniformly. The de- signer of the model crafts adversarial samples against one perturbation & obtain retrained model that minimizes |/A (a; 0) h (xr + 5x: f)]l 0 is the model parameter. For model h, 6g denotes one known perturbation, and any 8,, E {ōld dn denotes one unknown perturbation.
Adversarial A Attacks against Concept-level Explanations. Unlike standard adversarial attacks, adversarial attacks against explanations do not compromise task predictions. For concept-level in-
3

---

Published as a conference paper at ICLR 2025
False
Factor
Concept variable
v
v
v
True
Rectified
Category variable
(a) Factor graph construction
Potential function
h
Factor graph
concept intervention
h
h
y
Concept
Bottleneck
Prediction
(b) Explanatory logic errors identification
(c) Explanation rectification
Figure 3: Overall structure of AGAIN.
terpretable models, such attacks can be categorized into three types: erasure attacks, introduction attacks, and confounding attacks (Sinha et al., 2023). 1) Erasure attacks: the goal of the crasure at- tacks is to subtly remove concepts from a concept-level explanation; 2) introduction attacks: the goal of introduction attacks to allow the existence of irrelevant concepts; 3) confounding attacks: the goal of confounding attacks is simultaneously remove relevant concepts and introduce irrelevant concepts, These attacks are technically simpler to implement (see the Appendix for implementa- tion details of these attacks).
THE DESIGN OF AGAIN
AGAIN consists of three modules: 1) first, we encode logic rules of the real-world factor graph (Section 4.1; 2) then, we generate the initial concept-level explanation through the concept bottle- neck. The lactor graph reasoning is utilized identify whether the explanation of concept bottleneck violates the external logic, and thus to detect whether the perturbation exists (Section 4.2 3) finally, an interactive intervention strategy is designed to rectify the explanation and input the category The overall architecture of AGAIN is shown in Figure
predictor (Section 4.3)
4.1 FACTOR GRAPH CONSTRUCTION
To construct factor graph, we first define the logic rule set R {r:)]=m which contains two types: 1) concept-concept rule: all predicates consist of concepts. Such rules are used to constrain potential relationships among various concepts. For instance, there
rule of coexistence or exclusion
between concepts G and cy. which be formalized in logical notation C Cj C cj- 2) Category-conceyy rule: all predicates are defined by con- cepts and categories. Such rules used constrain po- tential correlations between concepts and categories. For instance, the coexistence or exclusion rule that exists be- tween concept and category label can be formalized as: G < y c; $ V/-
Then, we encode the above logic rules into factor graph G. As shown Figure 4 we illustrate the construction of the factor graph. Specifically, there are two types of variables = De VM, where V and V denote concept and cate- gory variable set, respectively, and are linked by F. In this way, each factor f. E corresponds to the i-th logic rule ri Each factor is defined as a potential function that per- forms logical operations based on different rules, which can be categorized into cocxistence and exclusion operations. Moreover, we define a potential function th: for each factor f.. which outputs a boolean value for each A (f:). If I (f+) makes true, 8/s (N (ft) 1, otherwise the (N (fi))
M
Factoe Graph
Factor
Potential function
Concept variable
Category variable
Figure Factor graph construction.
4

---

Published as a conference paper at ICLR 2025
For convenience, denote 1h (fi) th, We define the weight wy [0,1] to represent the confidence level of f. in two methods, i.., prior setting and likelihood estimation (The details can be referred to the Appendix (C.5). Higher w, indicates that the logic rules of fü are more important for reasoning, and vice versa.
4.2 EXPLANATORY LOGIC ERRORS IDENTIFICATION
AGAIN generates an initial concept-level explanation and identifies logical errors in the initial expla- nation. Specifically, employ concept bottleneck structure, popular concept-level interpretable module, to capture the semantic information from instances, which can learn the mapping between semantic information and concepts (Koh al. 2020a). The concept bottleneck contains concept predictor he RD' + R and category predictor by RM - R. The instance is first mapped the concept space by he and obtains the corresponding concept activation vector, i.e., è (z))),, where è E [0, MM and then the conceptual activation vector is fed into hy to yield the final predicted category, i.e., ú (C). where y E {0, is defined the initial explanation.
Next, 9 takes and ÿ as inputs to assign y and V", respectively. If concept è > 0.5, 2 E è, we set variable va 1, va ve, otherwise va O. For the category variables, we vg 1, vg VV, and V 0g = (0) K-1
Subsequently, we evaluate the likelihood of the variable assignment under rule constraints through logical reasoning. Firstly, after each variable (concept and category) in 9 is assigned a value, boolean values output from potential functions of all factors. These boolean values indicate whether the assignments of concepts and categories satisfy the logical rules represented by potential functions. Therefore, the weighted sum of all potential functions quantífies the extent to which concept assign- ments satisfy the logic rules in G.
Then, we seek to obtain the likelihood of the current concept assignments occurring, conditional on the known categories and logic rules. We quantify this likelihood by computing a conditional prob- ability using the weighted sum of potential functions. We consider all possible concept assignments and compute the expectation of current concept assignments. This expected value is considered as the conditional probability, which is then used to detect whether concept activations are perturbed. For illustrative purposes, we provide an example. Suppose there are concepts A and B. The current concept assignment is {1.0) denoting A (active) and = (inactive). We iterate through all four possible assignments: {1, 0), {0, 1}. {1, (), We compute the weighted sum of the po- tential functions for each of the four cases and compute the expectation of the potential function for {1, 0). This expectation is the conditional probability that concept assignment {1. 0) occurs condi- tionally on the known categories and logic rules. Formally, we denote this conditional probability as P (Ve )
P P(V (V) = exp Σ wath
exp Σ V
Σ wi VIE
where & represents all cases of concept assignments, and DE represents case in d. This implies that the denominator of Eq. (1) is the normalized constant term. We use IP (Vo 1 to evaluate the comprehensibility of explanation è. Higher IP (Ve (2²) indicates that è is more comprehensible, and vice versa. theory, we consider that a comprchensible è should satisfy each r; E R, ensuring that P PV* ()) attains upper bound denoted as IP (vo )
1)
vPP (V (PVVIVV)
max exp Σ w wbbh)) ViN
a
where a denotes the denominator of Eq. (1. However, in practice, even concept explanations gener- ated in benign environment (without perturbations) rarely satisfy all the rules. Overly strict logical constraints may instead cause G to lose its ability to recognize perturbations. Therefore, we allow a comprehensible explanation violate some low-weight rules. Naturally, we establish a relaxed identification condition for distinguishing explanations corrupted by perturbations from comprehen- sible explanations:
TP (V= 1 0 (P )
(3)
5

---

Published as a conference paper at ICLR 2025
where 0 E [0, 1] is hyperparameter controlling the relaxation. Ô approximate 1 implies stricter
constraint imposed by 9 on the explanation. If Ve, V satisfies Eq. (3). then è is comprchensible; otherwise, it is recognized theoretically that 9 contributes to comprehensible explanations. For a detailed theoretical analysis, please refer Appendix [
having logical error under perturbation. Further, demonstrate
4.3 EXPLANATION RECTIFICATION
Once explanations with logical errors are identified, AGAIN rectifies the explanation and put it as an input to the category predictor for the final prediction. For this objective, we propose an interactive intervention switch strategy aimed at enhancing the conditional probability of the G. The proposed strategy intervenes on the values of V and interactively observing the potential function difference. In this paper, we assume that ÿ unaffected under perturbations, thus we do not intervene in V The pseudocode of the interactive intervention switch is listed in Appendix
Our intervention strategy can be divided into three steps. First, traverse all factors with 3ن 1. For factor Ja E F, we modify the boolean value of its concept variables, considering the modification single intervention operation. Given that may be connected to multiple concept variables, there exist numerous intervention cases. For instance, consider fi containing concept variables ey and vj. There are three possible intervention cases: intervene only U; intervene only Vj and intervene both U; and Uy. We define the full set of possible intervention cases for fü as Tí. For each case tę E Tr we compute the potential function difference 8;, which represents the change in the potential function after executing t;. Note that t; does not only change fi, but also changes the I-hop neighbor factors of N (Je). Thus, we define 8; as follows:
81
Σ JELFI
Wj (1/ -1)))
(4)
where =; =f e U vy denotes the w value after t, intervention. Subse- quently, after traversing through all possible interventions in Tr we identify the intervention with the highest 8, as a candidate intervention. We generate the candidate intervention for each factor with 4/; = 1. We aggregate all the candidate interventions into final intervention, denoted as t.. We execute t, on Vo. From the set of intervened V and t., we generate a binary concept intervention vector z E 13 M and a binary mask vector m,, 10,1 denotes the concept activation status, where -1 indicates activated, and 1 indicates inactivated. m, denotes whether the concept intervened or not, where indicates intervened, and () indicates not intervened.
Finally, we employ z and m,, to rectify the initial explanation We utilize m,, to aggregate è and z for rectified concept activation vector Cre:
(5) where m is obtained by flipping the bits of m @ denotes dot product operation. The purpose of employing the intervention mask is to facilitate čre to retain activations in é that are not intervened.
čre = z c m. + é m',
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Datasets and Baselines. We evaluated AGAIN on two real-world datasets, CUB, MIMIC-II
EWS, and one synthetic dataset, Synthetic- MNIST. We choose two categories of methods. (1)
Concept-level methods: CBM (Koh et al. (020a). Hard AR (Havasi et al.] (022). ICBM (Chauhan
al. 2023). PCBM YYuksekgonul et al. 2023), ProbCBM (Kim et al., 2023). Label-free
CBM (Oikarinen et al. 2023). ProtoCBM (Huang et al. 2024). and ECBMs (Xu et al. 2024).
(2) Knowledge integration methods: DeepProblog Manhaeve et al. (2018), MBM Patel et al. 2022).
C-HMCNN Giunchiglia & Lukasiewicz 2220)) LEN Ciravegna et al. (2023). DKAA Melacci et al.
2021), and MORDAA Yin et al. (2021). In addition, we compare AGAIN with the retrained ver- sions of these baselines that employ ofte-oof-the-rr adversarial training strategy. More details on the datasets and baselines are provided in Appendix C.T and C.2 The experimental results on the
synthetic dataset presented in Appendix D.4

---

Published as a conference paper at ICLR 2025
Evaluation Metrics and Implementation Details. To evaluate the performance of AGAIN, use five metrics: predictive accuracy (P-ACC), explanatory accuracy (E-ACC), logical satisfaction metric (LSM), identification rate (IR), and success rate (SR). Higher scores indicate better perfor- mance for all metrics. Detailed descriptions of each metric are given in Appendix C.3 Additionally, the implementation details of AGAIN are provided in Appendix
C.4 C.4
5.2 EXPERIMENTAL RESULTS ON REAL-WORLD DATASETS
Identifying Perturbations. We apply adversarial perturbations acquired during black-box train- ing to randomly perturb multiple instances in the test set. Known and unknown perturbations denoted by 5g and 8u. respectively, with representing the perturbation magnitude. We evaluate the abillity of AGAIN to recognize logical errors of explanations by reporting SR and IR values under different perturbation magnitudes in Table The results demonstrate that AGAIN achieves remark- able IR and SR values under both 0g and ou- Specifically, AGAIN attains nearly 100% IR across all perturbation magnitudes. With SR results averaging up to 98%, we also validate that factor graph can effectively identify explanations from benign instances and permit them to directly predict categories without logical reasoning. Furthermore, it is also worth noting that as the perturbation magnitude increases, the IR value also gets larger. This observation is attributed to the larger pertur- bation magnitude causing a more pronounced logical violation in the generated explanations. The g more readily identifies these violations.
Table 1: IR and SR on two worl-world datasets.
6
6,
Dataset
Metries
Clear
c=4
c=8
c=16
c=32
c=d
c=8
c=16
c=32
97.3(1.3)
98.9(0.4)
98.9(0.8)
99.3(1.0)
97.3(0.5)
97.2(0.8)
97.5(1.1)
98.3(0.9)
CUB
SR
98.7(0.4)
98.0(1.2)
98.7(0.4)
97.7(0.6)
97.2(0.5)
97.4(0.7)
97.2(11)
96.8(0.9)
91.7(11)
IR SR
97.4(0.2)
98.3(0.3)
99.2(0.2)
99.80.1)
98.3/0.4)
99.510.1)
100.000.0)
100.000.0)
MIMIC-II I1I EWW
100.0(0.0)
98.91(0.4)
99.3 99.3(0.1)
98. 98.7(0.4)
98.2(1.1)
99.3(0.3)
(7.3))
98.6(0.4)
99.4(0.1)
Comprehensibility of Explanations. To investigate the comprehensibility of the cxplanations generated by AGAIN, we perform extensive experiments on both datasets for evaluating the LSM of the explanations, and the comparison are reported in Table 2] The baselines subjected to the retraining are identified by the "-AT" suffix. The results reveal that the comprehensibility of the explanations generated by AGAIN outperforms all concept-level methods, including the "-AT" ver- sions of these baselines, under different perturbation magnitudes. Particularly, previous interpretable models fail to generate logically complete explanations with LSMs lower than 48 under unknown perturbations of magnitude 32. but explanations from AGAIN can reach as high as 92.30. More- over, we demonstrate that AGAIN is hardly affected by the perturbation magnitude compared the baseline methods. This effect is attributed to the corrective capability provided by g for any level of logic violation. For kowledge integration methods, since DeepProblog, MBM, and C-HMCNN are unable to generate concepts, we splice their knowledge integration modules onto the CBM. The re- sults show that the LSM of AGAIN is optimal. In contrast, deepProblog can only constrain category predictions, not concept predictions, which results in low LSM under perturbation. The knowledge introduced by methods MBM and C-HMCNN can constrain concepts, but they only use logical rules between concepts and concepts, making their performance inferior AGAIN. Meanwhile, since DKAA and MORDAA have Multi-lahel predictors, we directly use Multi-label predictors to predict concepts. LEN can only constrain category predictions. DKAA and MORDAA detect ad- versarial perturbations in the samples using external knowledge, but they cannot correct the wrong concepts triggered by these perturbations.
Validity of the Factor Graph. As the theoretical analysis in Appendix demonstrates, 9 im- proves the comprehensibility of explanations. We experimentally validate this claim and further demonstrate that increasing the number of factors in G enhances the predictive accuracy of con- cepts. Specifically, we employ subgraph G' extracted from the original G for reasoning and analyze the impact on prediction accuracy by increasing the ratio of G' to g. In Figure 5 we depict the changes in P-ACC and E-ACC across four perturbation magnitudes on both datasets. It is evident that both P-CCC and E-ACC exhibit substantial improvement as the number of factors in g' in- creases. This observation indicates that G contributes in generating explanations with similarity to

---

Published as a conference paper at ICLR 2025
Table 2: Comparisons of LSM for AGAIN with other concept-level interpretable baselines.
6%
6
Dataset
Method
clear
6=4
<=8
=16
c=32
e=4
e=8
e=16
e=32
CBM
96.3(2.)
89.2(3.5)
77.422.)
33.7(1.))
39.415.4 4]
89.3(311 1)
77.465.5)
53. (3.8)
39.4(33)
Hard AR
85.6(0.9)
77.3(1.2)
66.4(3.2
50.8
47.610.8)
70.)
66.2(2.6)
10.4(1
1.8
ICBM
95.3(1.3)
86.4(..))
77.300.4)
16.461.6
39.5(1.5)
86.7
77.6(2.4)
1.99
39.8(3.2)
PCBM
95.7(1.6)
85
76.31.55
58.7(2.1)
40.3(1.2)
87.5(1.5)
78.1(1.4)
1.
41,622
ProbCBM
96.8/0.4)
k5.5(0.2)
77.611.1)
59.7(1.5
39.7(1.3)
87.7(12)
77.40..)
1..4
40.341.2
Label-ftee CBM
36.40133
84.301.7)
76.91144)
58.5(2.1)
40.810.3)
87.3(1.6)
77.8(1.2)
57.30144
40.0(1.8)
ProtoCBM
97.3(0.2)
92.3(1.3)
84.5(1.6)
64.5(3.2)
54.3(1.3)
84.4(19) 92.4(1.1)
64.502.4)
54.1(3.6)
BCBMs
96.4(1.3)
92.1(1.5)
87.5(3.7)
70.4(2.8)
64.7(2.11
92.2(0..)
86.6(3.7)
70.4(2.9)
64.4(6.1)
CBM-AT
93.4(1.4
92.911.3)
85.6(0.7)
75.6(1.7)
59.6(1.6]
87.6(0.6)
77.5(1.0)
53.3(1.9)
39.7(1.5)
Hard AR AT
82.5(0.3)
18.5
76.6(1))
60.5(1.) 1.39
76.600.6
6.42(1.4)
1)
47.5(2.1)
CUB
ICBM-AT
91.5(1.3)
91.6424)
86
70.611
80.41.2)
77.62.1)
56.4188)
39.40.61
PCBM-AT
93.6(0.6)
92.5(1.6)
84.4
76.5(1.3)
70.9(1
80.4(1.5
35.3(1.2)
15.6
4
PrebCBM-AT
93.5(0.9)
90.3(1.4
83.3L13)
78.2
70.311.8
80.4
71.601.2
57.50
4..6(.44
Label-free CBM-AT
93.4(1.)
91448))
86.2(1.4)
80
78.5(1
87.((4)
57.7 73.4
41.8/99)
BMotaCBM-A AT
94.40077)
92.7(1.1) 1722(19)
6x.712.7) 20.3(4.2)
91.7(1.)
82.5(1.5)
60.2(244
52.16..1
BCB
93.6(1.0)
91.9(2)5)
88.1(2.1)
83.1(388)
78.43.4)
90.7(2.4)
83.7(2.5)
68.7(2.7)
66.7(3.7)
LEN
96.40.8) 89.1(3.4)
77.
26.7(1.)
40.4(1.) 3]
89.1(3.4)
56.7(1.2) 77.8(1
40.4(1.))
DKAA
96.2(1.1)
91.2(11))
(..6(1.6)
76.9(1.))
73.7(5.3) 31
91.2(15)
(5.6(1.))
7.39(1.))
73.7(5.3)
MORDAA
96.5(0.1)
91.7(1.5)
16.1
10.6(2.1)
76.813.1
91.7015)
80.6(21) 86.1
76.8(31)
DespPrabing
96.4(0.2)
89.2(3.5)
77.4(2.8)
53.7(1.3)
39.445.4
89.2(3.5
42.55.5
63.1(3.8
39.44544
MBM
96.200.2)
93.5(3.1)
90.32.4)
88.7(3.2)
85.716.7
93.5
90.72
88.70.2)
85.7(6.7)
C-HMCNN
96.500.4
93.6(7.2)
89.7(1.2)
87.622.5)
85.0(3.)
93.6(7.2)
89.7(1.2)
87.6225)
85.0(3.2)
AGAIN
96.30.55
92.4(1.2)
93.1(2.3)
93.8(1.9)
91.5(1.71
94.5(1.6)
93.M1.7)
93.8(1.4)
92.1(2.1)
CBM
95.7(0.2)
90.4(17)
75.7(1.)
50.4(1.)
39 39.8(144)
90.7(0.9)
75.7(1.3)
50.901.4)
30.7(1.5)
Hard
96.7(0.3)
7..8(1
69.601.3)
53.8(1.7)
45.3(1.6)
77 4C
65. 3(1.))
53.8(3.2)
47.9(2.8)
ICBM
95.601.)
1..5(1.
56.8(2.1) 75.0(1.))
39.3(3.2)
86.5(1.7)
77.6(14)
56.7(2.1)
30.7(1.))
PCBM
96.100.2)
86.5(1.4)
73.001.2
53.8(2.5)
44.2(2.6)
11.2(1.)
78.701.4)
$7.8(2.2)
32.6/2.5)
ProbCBM
96.1(0.1)
8..6(14)
76.6(16)
36.9(1.))
39.7(3.1)
86
76.93.1)
53.45)
40.3/4.0
Label-free CRM
96.100.1
86.5(1.)
76.5(1.6)
65.5(2.1)
40.30
86.9(0.9)
17.64.1)
56.(6.3)
42.37.4)
PBotoCBM
96.70066
87.6(1.)
81,41000
70.812.4) 76.4(14)
87.4(12)
81.70.1)
76.3(2.1)
69.4224)
BCBMs
97.9(0.2)
88.4(1.2)
79.4(1.3)
70.8(2.6)
65.6(3.2)
88.6(2.7)
79.4
70.4(5.4
66.1(4
CRM-AT
94.2(0.4)
90.3(1.)
35.4()3)
78.8(1.))
60.9(1.9
55.7(2.5)
37.5
10.8(3.)
40.8(3.7)
MIMIC-III EWS
Hard AR-AT
ICBM-AT
94.2(0.7)
92.3(0.4)
90.3(21)
88.(1.4)
76996.6)
86
70.212.61
86.5(2.7)
71.1(3.5)
65.313.1)
77.)(1))
86.5
62.1011)
77.644.)
50.7011)
58. 78.7668)
44.2011)
39.49.3)
PCBM-AI AT
94.2(0.6)
90.3(1.7)
84.4(32)
76.9(3.4)
69.2(4.7)
81.7(33)
73.3(3.)
54.6(4.1)
42.34.9)
PrebCBM-AT
93.000.4
91.8(14)
88.41.5)
73.013.8) 78.8(3.6)
86.5(1.)
56.3474) 76.944.)
40.6
Label free CBM-AT
94.2(0.6)
19.5(1.7)
86.7(1.))
77.3338) 84.3(3.7
84.2(1.4)
78.9(2.5)
56.7(4.6
44,2(7.9)
PMotaCBM-AT
94.5(1.2)
19.7(11)
81.4(1))
70.812 (..4(1.4
80.((12)
76.7(4.1
66.12.1)
61.345.4
BCBMs.AT
93.9(0.6)
89.8(4.2)
82.3(2.1 I)
75.4(2.4)
70.6(2.8)
79.6(2.7)
74.216.2)
69. 69.7().77
65.9(6.5
LEN
96.4(0.6)
90.3
75.7(2.8)
50.3(3.8
40.2(7 11
90.3(2.6
75.7(2.8)
50.3(388
40.2(7.1)
DKAA
96.5(1.5)
96.1(1.6)
87.32.7) 71
79.8(2.1)
75.8(4.7
16.6
87.32.7)
79.82211
75.(47))
MORDAA
95.2(1.4
95.9(2.)
93.7
86.3
79 44.4
95.9(2.4
93.702.3
86.8
79446.6
DeepPrabing
95.5(1.3)
90.41.7
75.7
50.4(1.5
39.8 4)
90.4(.
75.7(1.3)
50.4
1..8
MBM
95.9(0.5)
92.7(4.)
88. 7024)
86. 16.3
84.415.7
92.764.)
88 78.7(
16.3(13)
84.45.7)
C-HMCNN
96.6(1.2)
94.0(2.)
92
85,56577
83.515.2)
94.022.6
92.5(1.)
85.5(5.7
83.5(5.2)
AGAIN
(6.1(0
96.1(0.7)
94.2(1.4)
96.1(1.2)
94,2(1.2)
94.0(2.7)
94.1(6.3)
94.2(2.4)
92.3447)
(E-ACC)
MIMIC-II EWS (C-ACC)
(O) (P-ACC)
MIMIC-III EWS (P-ACC)
96
09
88
85
63
gnd
85
-B-u-d
0-0-0
78
c=16
70
p-16
--1118
77
0.2
04
0.5
0.8
1.0
1.2
0.2
0A
0.5
0,8
1.0
3.2
0.2
04
0.6
1.0
1.2
0.2
0.4 0.6
08
10,
The ratio of'
ralio to
ralio
Figure 5: The impact of the factor graph size on P-ACC and E-ACC across 4 perturbation magni- tudes two real-world datasets.
the ground truth explanations for improving the predictive accuracy. Moreover, as the number of fac- tors in G' exceeds that of G (ratio > 1.0). E-ACC begin to converge. This also validates the setting for the number of factors in the original G reasonable. In addition, we report the P-ACC and E- ACC comparison results of AGAIN on the CUB dataset (see Table and Table 4). The comparison results of P-ACC and E-ACC on other datasets are provided Appendix D.5 The results indicate that AGAIN is optimal for E-ACC on all three datasets. Furthermore, since perturbations do not im- pact the final predictions, the P-ACC remains consistent across different levels of perturbation. The P-ACC of AGAIN are comparable to the other baselines because the factor graph does not improve the task predictive accuracy. We present a comparison of E-ACC and A-CCC between the CBM and AGAIN on the two real-world datasets in Figure G The results show that AGAIN achieves higher
8

---

Published as a conference paper at ICLR 2025
E-ACC values and comparable P-ACC compared to CBM under 32 perturbation. In the benign environment, E-ACC and P-ACC of AGAIN also remain comparable. The results suggest that factor graph logical reasoning does not affect prediction accuracy in the absence of perturbation.
Table 3: Comparisons of E-ACC between AGAIN and baselines on CUB.
6
bu
Method
clear
c=16
c=
CRMLAT
9.26(12)
91.41 41.4115
90.1
86 16.901. .))
85.4(2.1)
90.6(0.6)
89.5(1.2)
87.7(2.1)
80.4(1.4)
Hard AR-AT
97.3(1.4)
93.110.4)
9..41.6)
86.3(1.7)
84.7(2.5)
91.210
8..71.5
12.3)
79.62.11
ICBM-AT
96.8(1.1)
94.0r2.1)
89.7(2.4)
8(2.3)
84.8(3.2
89.312.7
85.122.11
80.5(3.1)
78.9(1.2) 21
PCBM-AT
97.8(1.1)
92.4(2.7)
89.41.2)
86,4(1.9)
83.5(2.4)
92.4(1.6]
89.8(3.1)
85.3(2.6)
80.41.5)
ProbCBM-AT
96.9K1.0)
92.5(1.0)
90.2(1.4)
84.6(2.3)
83.9(2.1)
90.6(3.2)
88.61166)
82.5(1.7)
79.4(1.3)
Label free CBM-AT
97.9K0.9)
93.1((11)
92.1(2.3)
86.7(1.6)
84.7(3.2)
90.4(2.1
86.7(2.1)
85.6(1.2)
80.5(2.8)
ProtoCBM-AT
98.1(0.7)
93.1(1.3)
90.41155
85.4(3.1)
83.9(2.4)
91.6(1.8)
86.8332)
83.1(1.3)
81.5(3.1)
BCBMs-AT
98.2/0.5)
92.8(2.4)
89.3(3.4)
86.6(2.2)
83.7(2.5)
91.3(1.3)
89.4(1.3)
84.8(2.5)
80.7(19))
LEN
97.700.2)
93.101.5)
89.4(1.4)
86.101.4
79.963-7)
53
89.4(1.4
86.101.5
79.9(3.7)
DKAA
98.2(1.1)
92.001 4)
90.6(133
86.4(3.2)
80.3(2.6)
92.01
90.621.3)
86.4(3.2)
80.3/2.61
MORDAA
98.1(1.)
93.110.7)
91.2(1.4)
86.3(1.8)
79.4(4.6)
93. 110.7)
91.2(11.4)
86.3(1.)
79.4/4.6)
DeepProblog
96,9(0.8)
91.0(2.3)
89.8(1.1)
85.2(1.3)
80.2(1.4)
91.012.3)
19.8(1.)
85.2(1.)) 3)
80.2(1.4)
MRM
97.9K1.0)
91 (1.)
90.3(1.4)
86.3(1.3
82.7(3.2)
91.611 22)
90.3(1.4)
86.3(1.3 3)
82,7(3.2)
C-HMCNN
98.1(022)
91.(114)
91.6(2.1)
85.515.7)
84.5(2.3)
94,0(2.6)
92.5(1.6)
85.5(5.7)
83.545.2)
AGAIN
97,5(0.1)
94.1(0.2)
94.1(0.2)
93.6(0.7) 93.8(0.4)
94.9(0.3)
94.9(0.7)
93.5(1.1)
93.9(0.7)
Table 4: Comparisons of P-ACC between AGAIN and baselines on CUB.
bn
Method
clear
c=
c=8
CRM-AT
62.5(1.2)
62.4(1.2)
62.4(1.1)
62.2(1.4) )) 60.5(1.1)
62.4(1
62.6(1.2)
59.4(0.8)
59.6(1.2
Hard AR-AT
62.411.3
62.2(0.6)
61.2(0.6)
61.6(0.4)
59.6(0.8)
61.6(1.2)
60.3(0.3)
59.4(0.6)
59.7(1.2
ICBM-AT
62.4K1.7)
62.2(0.6)
61.3(0.6)
61.8(100)
59.3(1.1)
62.4(1.1)
60.200.2)
59.5(0.4)
59.8(1.2)
РCBM-AT
62.5(0.4)
62.1(0.6)
61.0X0.6)
61.6(0.3)
59.610.8)
62.6(1.2)
61.000.1)
59.4(0.7)
59.5(1.1)
PrT
62.400.5)
62.2(0.4)
61.41.1)
61.6(0.4)
59.5(0.8)
61.6(1.1
60.6K0.3)
59.3(1.1)
59.600.8)
Label free CBM.AT
62.7(1.3)
62.7(0.6)
61.1(0.4)
61.30.22
60.7(11)
61.3(12)
60.6(0.3)
59.3(0.6)
59.5(0.8)
ProteCBM-i AT
62.4113)
62.1(0.7)
61.3(088
61660.3)
59.210.7)
61.4(10)
60.1(1.2)
59.400.9)
59.5(1.2)
BCBMs-AT
62.6(122)
62.4(1.6)
62.4(1.6)
61.70.4)
59.6(0.8)
61.((1.)
60.3(12)
59.3(1.1)
59.6(1.2)
LEN
62.4K0.4
62.2(1.3)
62.340.5)
59.401.)
59.4(1.4)
62.2(1.3)
62.3(0.5)
$9.4(1.6)
59.4K(44
DKAA
61.41.55
62.6(1.2)
62.4(0.5)
59.((12))
59.2(1.7)
62.6(1.2)
62.4K0.5)
59.4(1.2)
59.2(1.7)
62.2(0.1)
62.510.8)
59.5(1.1)
59.3(0.9)
61.8(13)
59.5(1.1)
DeepProblng
59.2(0.3)
59.3(14)
59.7(0.2)
59.3(1.4)
59.5(1.7)
59.700.2)
59.3(1 14)
59.5(1.7)
MBM
62.5(1.2)
62.5
61.6(11)
59.701.4
19.4(1.1)
62.5(1.2)
61.6(11)
59.7(14)
59.4(1.1)
C-HMCNN
62,5(1.2)
62.6(1.2)
62.4(1.1)
39.6(144
19.4(1.1)
62.6
62.A(1.1)
59.611.4 ()
19.4(1.1)
AGAIN
62.5(0.4)
62.2(0.6)
61.2(0.6)
61.6(0.4)
59.6/0.8)
61.6(1.2)
60.3(0.3)
59.4(0.6)
59.6(0.8)
Rectification of Interactive Intervention Switch. In Figure a and (7),, we illustrate several in-
Clear
Penurbtton
Clear
Penzittioo
stances of two real-world datasets along with rec-
100
100
AGAAN [rrs
[ AGANN I
EI AGAIN
tified explanation segments with a dimension of 5 and show the utilized rules. The interactive inter- vention switch effectively rectifies the logical error of the explanation based on the predefined rules,
[ MM
thereby enhancing the overall logical coherence of
the explanation.
39
Ablation Study. We conducted ablation studies to examine the effectiveness for each rule type in G. The larger number of rules in CUB compared to MIMIC-III EWS contributes to a more signifi- cant ablation effect, we executed ablation stud- ies on the CUB data. We investigated the perfor- mance of all factor graph variants by reporting the LSM results in Table 5 Ft and Jee denote the set of factors encoding category-concept rules and the set of factors encoding concept-concept rules, respectively.
B-ACC
P-A0C
T-ACC
P-400 B-A0C
B-4CD
CUC
Figure 6: The comparison of P-ACC and E- ACC on the two real-world datasets.
According Table 5 we can draw the following conclusions. First, the variant without (i.e., F == 0) yielded the lowest LSM values, affirming the essential role of G in the model. Second, G encoding both concept-concept and eetegory-concepp rules (i.e., F FF U FO) achieved the best performance. This factor graph (constructed in this paper) demonstrates an average improvement of 6,86 over other variants of G that encode only the concept-concept or category-concept rules.

---

Published as a conference paper at ICLR 2025
(o)
WIMIC-HI
Synthetic-MMN oriare
Oriin
Gerveded
Corgeyd
IMprycrdt W011
Mach S
veveellne Poerseeiee be
<>>
AM
Vili (1 Mishypeeende
Corger
Cooeyy??
Oriiil
Denooned
Origen
Goneeed
Orignd
Gevered
Mil
odtars
Bail
Evy upp
1811
Now w ① some vid
(1)
Comnan
Derill dupe
Wiscoah
only
Colrrat viten
Tuekos (
Vid
lance Tuales CD Sinell Pett
Cargey??
Soough
Dosighh
Figure 7: Rectified explanation results on three datasets. The bar represents the normalized activa- tion values of the concepts. Blue bars indicate activated concepts and red bars indicate inactivated concepts. The orange area shows the logical rules followed.
This indicates that both rules are essential for comprehensibility of explanations. Fi- nally, the variant containing only JE per- forms lower than the variant containing only FV. This suggests that the category-concept rules contain more direct logical knowledge about category prediction.
Table 5: Ablation study on the CUB dataset: impact of rule types.
Factor Set
16
32
F 0 J F = 5 u
89.2(1.1)
77.4(21)
53.7(2,4)
39.4(3.1)
F
92.5(0.3)
89.5(2.1)
85.5(2.2)
84.0(2.5)
(0.4)
86.8(2.4)
82.6(2.1)
80.1(2.3)
94,5(0.3)
93.3122.9)
93.8(2.2)
92.1(6.1)
6 CONCLUSION AND DISCUSSION
In this paper, we explore the comprehensibility of explanations under unknown perturhations and propose AGAIN, an factor graph-based interpretable neural network. Inspired by the knowledge integration of factor graphs, AGAIN obtains comprehensible explanations by encoding prior logical rules as the factor graph and utilizing factor graph reasoning to identify and rectify logical error in explanations. It addresses the inherent limitations of current adversarial training- based interpretable models by guiding explanation generation during inference. Furthermore, we provide a theoretical analysis to demonstrate that factor graphs significantly contribute to obtaining comprehensible ex- planations. We present an initial attempt to generate comprehensible explanations under unknown perturbations from the inference perspective. AGAIN provides an effective solution for the defense of interpretable neural networks against various perturbations and meanwhile saves the high cost of retraining. It takes significant step towards resolving the crisis of trust between humans and interpretable models. In addition, we note two limitations of AGAIN: 1) the validity of AGAIN relies on the correct prediction categories. Wrong categories imported into the factor graph cause explanations to be wrongly rectified; 2) when domain knowledge changes, the factor graph needs to be reconstructed, which implies AGAIN lacks generalizability. We leave these for future work.
ACKNOWLEDGMENTS
This paper is supported by the National Key R&D Program of China (Grant No. 2024YFA1012700). Thanks to Shan Jin and Ciyuan Peng for their help with data preprocessing.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Hubert Baniecki and Przemyslaw Biecek. Adversarial attacks and defenses in explainable artificial intelligence: survey. Information Fusion, 107:102303, 2024.
Akhilan Boopathy, Sijia Liu, Gaoyuan Zhang, Cynthia Liu, Pin- Yu Chen, Shiyu Chang, and Luca Daniel. Proper network interpretability helps adversarial robustness in classification. In ICMI., pp. 1014-1023, 2020.
Gecia Bravo-Hermsdorff, David Watson, Jialin Yu, Jakob Zeitler, and Ricardo Silva. Intervention generalization: A view from factor graph models. In NeurIPS, pp. 43662-43675, 2023.
Zachariah Carmichael and Walter Scheirer. Unfooling perturbation-basee post hoc explainers. In AAAI, pp. 6925-6934, 2023.
Prasad Chalasani, Jiefeng Chen, Amrita Roy Chowdhury, Xi Wu, and Somesh Jha. Concise expla- nations of neural networks using adversarial training. In ICML, pp. 1383-139/ 2020.
Kushal Chauhan, Rishabh Tiwari, Jan Freyberg, Pradeep Shenoy, and Krishnamurthy Dvijotham. Interactive concept bottleneck models. In AAAI, pp.
2023.
E Chen, Yang Cao, and Yifei Ge. A generalized shuffle framework for privacy amplification: Strengthening privacy guarantees and enhancing utility. In AAAI, pp. 11267-11275, 2024.
Jiefeng Chen, Xi Wu, Vaibhav Rastogi, Yingyu Liang, and Somesh Jha. Robust attribution regular- ization. In NeurIPS, pp. 14302-14312, 2019.
Gabriele Ciravegna, Pietro Barbiero, Francesco Giannini, Marco Gori, Pietro Lió, Marco Maggini, and Stefano Melacci. Logic explained networks. Artificial Intelligence, 3 14: 14:103822, 2023.
Ann-Kathrin Dombrowski, Christopher Anders, Klaus-Robert Müller, and Pan Kessel. Towards robust explanations for deep neural networks. Pattern Recognition, 121:108194, 2022.
Jacques A Esterhuizen, Bryan R Goldsmith, and Suljo Linic. Interpretable machine learning for knowledge generation in heterogeneous catalysis. Nature Catalysis, 5(3): 175-184, 2022.
Hidde Fokkema, Rianne de Heide, and Tim van Erven. Attribution- explanations that provide recourse cannot be robust. Journal of Machine Learning Reseurch, 24(360):1-37, 2023.
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. In AAAI, pp. 3681-3688, 2019.
Eleonora Giunchiglia and Thomas Lukasiewicz. Coherent hierarchical multi-label classification networks. In NeurlPS, volume 33, pp. 9662-9673, 2020.
Nezihe Merve Gürel, Xiangyu Qi. Luka Rimanic, Ce Zhang, and Bo Li. Knowledge enhanced machine learning pipeline against diverse adversarial attacks. In ICML, pp. 3976-39999999 3987, 2021.
Marton Havasi, Sonali Parbhoo, and Finale Deshi-Velez. Addressing leakage in concept bottleneck models. In NeurlPS, pp. 23386-23397, 2022.
Mengdi Huai, Jinduo Liu, Chenglin Miao, Liuyi Yao, and Aidong Zhang. Towards automating model explanations with certified robustness guarantees. In AAAI, pp. 6935-6943, 2022.
Qihan Huang, Jie Song. Jingwen Hu, Haofei Zhang, Yong Wang, and Mingli Song. On the concept trustworthiness in concept bottleneck models. In AAAI, pp. 11161-21168, 2024.
Sandesh Kamath, Sankalp Mittal, Amit Deshpande, and Vineeth N Balasubramanian. Rethinking robustness of model attributions. In AAAI, pp. 2688-2696. 2024.
Anuj Karpatne, Gowtham Atluri, James Faghmous, Michael Steinbach, Arindam Banerjee, Au- roop Ganguly, Shashi Shekhar, Nagiza Samatova, and Vipin Kumar. Theory-guider data science: A new paradigm for scientific discovery from data. JEEE Transactions on Knowledge And Data Engineering, (110023118-2333 2017."
11

---

Published as a conference paper at ICLR 2025
Eunji Kim, Dahuin Jung, Sangha Park, Siwon Kim, and Sungroh Yoon. Probabilistic concept bot- tleneck models. In ICML, pp. 16521-16540, 2023.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and 2020a.
Percy Liang. Concept bottleneck models. In ICML, pp.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In ICML, pp. 5338-5348, 2020b.
Himabindu Lakkaraju, Nino Arsov, and Osbert Bastani. Robust and stable black box explanations. In ICMI, pp. 5628-5638, 5638, 2020.
Xiao-Hui Li, Caleb Chen Cao, Yuhan Shi, Wei Bai, Han Gao, Luyu Qiu, Cong Wang, Yuanyuan Gao, Shenjia Zhang, Xun Xue, et al. A survey of data-driven and knowledge-awarr explainable ai. IEEE Transactions on Knowledge And Data Engineering. 34(1):29 49, 2020.
Renqiang Luo, Huafei Huang, Shuo Yu, Zhuoyang Han, Estrid He, Xiuzhen Zhang, and Feng Xia. FUGNN: Harmonizing fairness and utility graph neural networks. In KDD, 2024a.
Renqiang Luo, Huafei Huang, Shuo Yu, Xiuzhen Zhang, and Feng Xia. FairGT: A fairness-wwar graph transformer. In IJCAI, 2024b.
Robin Manhaeve, Sebastijan Dumaneic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. Deepproblog: Neural probabilistic logic programming. In NeurIPS, volume 31, pp. 1-24, 2018.
Stefano Melacci, Gabriele Ciravegna, Angelo Sotgiu, Ambra Demontis, Battista Biggio, Marco Gori, and Fabio Roli. Domain knowledge alleviates adversarial attacks in multi-label classifiers IEEE Transactions Pultern Analysis and Machine Intelligence, 44(12):994499559 2021.
Gherman Novakovsky, Nick Dexter, Maxwell W Libbrecht, Wyeth W Wasserman, and Sara Mostafavi. Obtaining genetics insights from deep learning via explainable artificial intelligence. Nature Reviews Genetics, 24(2):125-133 2023.
Tuomas Oikarinen, Subhro Das. Lam M. Nguyen, and Tsui-Wei Weng. Label-free concept bottle- neck models. In ICLR, pp. 1-32, 2023.
Dhruvesh Patel, Pavitra Dangati, Yoon Lee, Michael Boratko, and Andrew McCallum. Mod- eling label space interactions in multi-label classification using box embeddings. In ICLR, pp- 1-21, 2022.
Ciyuan Peng, Tao Tang, Qiuyang Yin, Xiaomei Bai, Suryani Lim, and Charu C. Aggarwal. Physics- informed explainable continual learning on graphs. IEEE Transactions on Neural Networks and Learning Systems, 35(9):11661111777)) 2024.
Ali Rahmati, Seyed-Mohsen Moosavi-Deziee Pascal Frossard, and Huaiyu Dai. Geoda: A geo- metric framework for black-box adversarial attacks. In CVPR, pp. 8446 8455, 2020.
Laura Rieger, Chandan Singh, William Murdoch, and Bin Yu. Interpretations are useful: penalizing explanations to align neural networks with prior knowledge. In ICML, pp. 81 16-8126, 2020.
Anindya Sarkar, Anirban Sarkar, and Vineeth N Balasubramanian, Enhanced regularizers for attri-
butional robustness. In AAAI, pp. 2532-2540, 2021.
Sanchit Sinha, Mengdi Huai, Jianhui Sun, and Aidong Zhang. Understanding and enhancing robust- ness of concept-based models. In AAAI, pp. 15127-15135. 2023.
Zeren Tan and Yang Tian. Robust explanation for free or at the cost of faithfulness. In ICML, pp. 33534-33562, 2023.
Jiapeng Tian, Hui Song. Gchao Sheng, and Xiuchen Jiang. Subsystem measurement-based condition
assessment for power transformers via joint inference of data and knowledge. IEEE Transactions on Instrumentation And Measurement, 73:1-12, 2024.
Iaria Tiddi and Stefan Schlobach. Knowledge graphs as tools for explainable machine learning: A survey. Artificial Intelligence, 302: 103627, 2022.
12

---

Published as a conference paper at ICLR 2025
Huiling Tu, Shuo Yu, Vidya Saikrishna, Feng Xia, and Karin Verspoor. Deep outdated fact detection in knowledge graphs. JCDMW, pp. 1442-1452, 2023.
Marco Virgolin and Saverio Fracaros. On the robustness of sparse counterfactual explanations to adverse perturbations. Artificial Intelligence, 316:103840, 2023.
Laura Von Rueden, Sebastian Mayer, Katharina Beckh, Bogdan Georgiev, Sven Giesselbach, Raoul Heese, Birgit Kirsch, Julius Pfrommer, Annika Pick, Rajkumar Ramamurthy, et al. Informed machine learning- taxonomy and survey of integrating prior knowledge into learning systems.
IEEE Transactions Knowledge And Data Engineering, 35(1):614-633, 2021.
Andong Wang, Wei-Ning Lee, and Xiaojuan Qi. Hint: Hierarchical neuron concept explainer. In CVPR, pp. 10264, 2022.
Feng Xia, Ke Sun, Shuo Yu, Abdul Aziz. Liangtian Wan, Shirui Pan, and Huan Liu. Graph learning: A survey, IEEE Transactions on Artificial Intelligence, 2(2):109-127, 2021.
Xinyue Xu, Qin, Lu Mi, Hao Wang, and Xiaomeng Li. Energy-based concept bottleneck models: Unifying prediction, concept intervention, and probabilistic interpretations. In ICLR, pp. 1-23, 2024.
Zhuolin Yang, Zhikuan Zhao, Boxin Wang. Jiawei Zhang, Linyi Li, Hengzhi Pei, Bojan Karlaš, Ji Liu, Heng Guo, Ce Zhang, et al. Improving certified robustness via statistical learning with logical reasoning. In NewrlPS, pp. 34859-34873, 2022.
Mingjun Yin, Shasha Li, Zikui Cai, Chengyu Song, M Salman Asif, Amit K Roy-Chowdhury, and Srikanth Krishnamurthy. Exploiting multi-object relationships for detecting adversarial attacks in complex scenes. In CVPR, pp. 7858-7867, 2021.
Shuo Yu, Ciyuan Peng, Chengchuan Xu, Chen Zhang, and Feng Xia. Web of conferences: A conference knowledge graph. In WSDM, pp. 1172-1175, 2023.
Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. In ICLR, pp. 1-20, 2023.
13

---

Published as a conference paper at ICLR 2025
A ALGORITHMS
The overall algorithm for the interactive intervention switch summarized in Algorithm 1. Algorithm 1: Interactive intervention switch
Input The factor graph 9, original concept activation vector a Output: Rectified concept activation vector are
for f; in F do
bh 0 then
Obtain the set T; of all possible intervention cases for fi;
for I, T: do
Compute s; through single intervention I; according to Eq.
Select 4 with the highest 8, as a candidate intervention;
Aggregate all candidate interventions into final intervention t.: Obtain intervention vector z and mask vector m,.: Aggregate and based on Eq. ()) yields the resulting crei
10 return Cre:
B THEORETICAL ANALYSIS OF AGAIN
In this section, we present theoretical proofs ensure that the G in AGAIN contributes to generating comprehensible explanations under unknown perturbations. Previous theoretical analyses on the validity of interpretable models have emphasized that comprehensible explanations tend to have high similarity to explanatory labels (Li et al. 2020: Karpatne et al., 2017 Von Rueden et al., 2021]. Therefore, we hope guarantee that AGAIN can generate comprehensible explanations by proving that G contributes to explanations under unknown perturbations in approximation to the explanatory labels. Specifically, our theoretical proof is divided into two parts. In the first part, we establish correlations between the conditional probabilities of the factors and the lower bounds on the predictive accuracy of concepts. the second part, we show that this lower bound increases according to the larger size of G.
B. THE CONCEPT PREDICTIVE ACCURACY LOWER BOUND FOR AGAIN
Under the adversarial distribution, given perturbation å and the concept explanation label the accuracy of the explanation is denoted as Ah;
A
II Ps (2 c) cEc
(6)
To simplify the writing, use lig () to denote the process of identifying (Section 4.2 and rectifying (Section 4.3) AGAIN. We extend this definition to assess the accuracy of explanations generated by AGAIN:
A AGAIN
(m) Ps (n(T) oEe
II
(c) -
where h((m) (.) denotes the output of the factor graph reasoning with respect to the m-th concept.
Lemma 1. Given factor graph G, the following equation is valid:
A AGAIN
II cEc
Ps (AN (va) > 0 (c),
(8)
where
AN (va)
Σ
(2wrehs -wz)-
(9)
(va)
14

---

Published as a conference paper at ICLR 2025
Proof. First, AAGAIN = Ileec Ps (ng" (m))) (é. ú) =c) is known. Then, based on the properties of
9, P6 (1g (m) (c. 0) = can be expressed equivalently as Ps (P (Fe, Iie = c) > P(e, # c) Ic).
According to Eq. 11. we extend IP (N (va) U2 - c) > (N (va) lea + c) to formulation that incor-
porates potential functions within G:
P (W (wz) Ue c) (W (us) luv + c)
Σ
wit; -
E
wi(1 v.)) > 0
(va)
(e()(va)
Σ
Wy +
Σ
wwsb >
(v))
(E()(ve)l
(E))((b))
Σ
2w, W1 0,
then, we obtain:
A
AGAIN
II Po
(ig 0) = -c)=-I
Ps (FFo, 20l = =c) (Fre wus + (c)lc)
<Ec
nEe
= III P% <Ec
Σ EiENN(OD)
2wae/s
wi > 0lc)
=I Po <Ec
II
(EreAriess)
Σ
2wath: w; > 0|c)
We observe that Ax (2a) determines the lower bound of Eq. (8) Further, we characterize the con-
straints on the concepts within G to impose bounds on the AN (rz), thereby restricting its left-tailed probability below 0. To this end, considering a concept variable caa we characterize the four cases of constraints imposed by on ta during reasoning with its neighbor factors fa N (va):
LS (v: (c) - U,
(10)
LP < T Ps (v: Uu = c) S UT
LN S T Ps (1h = 0/2a - c) S UN
(11)
LN S FN Ps (1/2 = 0/2a = c) S
LP < F = PA (v: 1/ea 1 - c) < UP.
Moreover, we employ factor graph characterizations to represent the lower bound of AN (va). To this end, lemma introduced below (Gürel et al., 2021). This lemma illustrates an inequality property between Av (va) and factor graph characterizations.
Lemma 2. Suppose each factor has the optimal weight. Then there exists
(2-2c)
E (A (va) le) 2 Z1 72 log 1-c'
(12)
where
LT. 21 T 1''log log
1 1 UT
U
1 1- LT,
(1 (1-T") T log
EN
log
+ (1 - F
log T- U
U.
1 L.P
LT
1 Uf
Ut
1 L7
Zg (1 (1-c) NNN log
(1 T-V) log
F log
+ (1- FP) log
Th
LN
L
U.
(13)
To facilitate the writing, we have opted for a simplified symbolic representation:
(1-2c)
I A, Zi + Zz - log 1
(14)
To further solve the bound of AGAIN we introduce another lemma (Chnn et al. 2024). This lemma outlines the definition and properties of Hoeffding's inequality.
15

---

Published as a conference paper at ICLR 2025
Lemma 3. Suppose the given æ1...., 2, independent random variables, and 21-1 E [aa, be]- The empirical mean these variables denoted as 2 = (I1 +... + xt//t. Then for any B > there exists the Hoeffding's inequality shown below:
2
P P( [ B) 2xx
(15)
Eier (Ti)
where (b; - a:) S Ta and B.)) denotes the expectation.
Herein, we provide the lower bound with respect to AAGAIN, This lower bound imposes minimum accuracy constraint on AGAIN.
Theorem 1. For the AGAIN with G, under the assumption that factors satisfying Z1 > 0 and Z/ > 0 are considered, the following inequality is established:
AAGAIN
III (L (exp(0)))
(16)
where
(1. (I.N(oo))))
θ -2 ΣiEN log (T(L-F)) 2' (I(()=FF)
(17)
Proof.
According to the symmetry of G, derive the following equation:
AAAIN
II 8 (0) (c. =c)
-II (1 - PPP (AN(ee) < olcc).
According to Eq. 12 we can get
Ps (AN (Va) <0(c) = Pp (AN(ua)
E (vr)))
+ [(vo)) < (c
I's ((AN(es)
- E Av((o) 1) < -E -I lc)
Ps ((AN(va)
- E (A lc]) - -[(AN(e)) le]
< Ps ((AN(e)) E[ E[AN((e) |c]) < - x((e) le).
Substituting I's (AN(ez) < <0cc < Ps ((A E (w le]) -Lntraa) le) into Hoeffd-
ing's s incquality and letting 3 = L (vve). in Lemma 3, we evidently derive the following inequality:
2 (T.. 2
P. (AN( <0(c) Po (Aw(o))-E(wwa - E(AN(e))))))))))))))))))()) |c] < - Ar(aai 1e) S exp
ΣIEN (T:)²
According to Lemma 3, we can use Eq. 9 and Eq. 12 to establish Hoeffding's inequality with respect AAGAI,, If we consider AN the independent random variable 3; in Lemma
assume
wi exp (va)
s.t. iE [W (ue)l,
according to Eq. (10) Eq. then there exists
and Eq. 12 can infer a, and b; for vi in two cases. th; 1,
UT
log UF I exp (1i) = w.e S log
LF
3/: 0, then there exists
UT
LT
log
S w, exp (w)) = w S log
L
UF
Since w; > 0, it is evident that wse > w, Therefore, for both cases above, there is uniform range interval that can inferred
1 U
U
log
S w; exp (1b.) S log I,F
16

---

according to Published as a conference paper at ICLR 2025
further, it can be inferred that w, exp (1%) S Ty. We know (b, a4) S and as ws exp (d:) 1 b,s0 24 It-1 E [ae, by]
holds.
e log
U
U 0² LE)
log
= =log
L'(1- T s.t. iE IW (u=)l,
LF
L
(- [2(r0x(0o00)
Ps (AN (vz) <0)c) exp (- ΣiEn (Ti)
we can be inferred that
II dEc
PA
(e) 3
(c, -c) = =II
Po (AN(v)) > 0 = II (1 PP (AN(He) < <0lc))
dEc
DEC
L. N((e)
II
1 E
exp
Σ (TT)²
99c
IEN
I. (r(E)i) 2
II cEc
1 EE
exp
log UT(I-LF) 1'(1-U"5
iNN
B.2 LOWER BOUND OF ACCURACY VERSUS NUMBER OF FACTORS IN THE FACTOR GRAPH
After analyzing the lower bound of concept accuracy, we aim to demonstrate a positive correlation between this lower bound and the number of factors in the factor graph. In simpler terms, increase in the number factors is directly proportional to an enhancement in predictive accuracy. This implies that contribute the predictive performance of the model. To facilitate the analysis, introduce a lemma below (Gürel et al. 2021). This lemma illustrates an unequal relationship of the accuracy lower bound under specific factor graph characterizations.
Lemma 4. Suppose that the upper and lower bounds of each factor graph characteristic are identi- cal, then there exists
2 (IAAx())))))
II dEc
1 1
E
exp
II oEC
exp (-2N e
-f)))
(18)
Σ REN
log
(Fi1-1A) (F((I-UT))
where
θ T=:-U -T- LT -F==U==1~
(19)
Furthermore, we present theorem that highlights a significant correlation between the lower bound of concept accuracy and the number of factors in
Theorem 2. For the given G in AGAIN, if satisfies the assumptions of Lemma 4 and each factor in G satisfies 6 > F the lower bound on the concept predictive accuracy increases strictly monotonically the number of factors N of the factor graph increases. Moreover, the lower bound of concept predictive accuracy with factor graph is strictly larger than the lower bound without factor graph.
Proof.
According to Eq.
18
O" > OF,, we can achieve
.A AGAIN
P
hg
(c. 9) = = =) > II (1 exp (02NN(0T-e9))
ccc
oEc
17

---

Published as a conference paper at ICLR 2025
POCAMII
POCASS
IOGLUTS
ICCHUNI
PCC-449
Figure 8: Correlation of association scores between concepts on the CUB dataset. The horizontal and vertical axes indicate the correlation scores of the two concepts. The top row shows the associ- ation scores for the groups of concept pairs with PCCs > 0.8 (red scatter). The bottom row shows the association scores for the 6 sets of concept pairs with PCCs < -0.5 (blue scatter).
Let OT - = and f(N) = 1 - exp ( IN (OT - -O")). Then there exists the first order derivative ('( = 2ne 2o > 0 of f(N). Therefore, f(NV is monotonically increasing, i.e., AAGAIN is strictly monotonically increasing function with respect to N.
Notably, for the case without the factor graph (i.e., N = 0), we have S(N) =
exp (- (0" @²)) = O. And for a factor graph G with an arbitrary number of nodes, f(N) f(0) Therefore, the concept predictive accuracy with the introduction of the fac- tor graph is constantly greater than the case without the factor graph.
B.3 DISCUSSION
Our theoretical proof validates that & has the potential to enhance prediction accuracy, provided that the rules embedded in the factor graphs restrict concept activation aligmment with prior logic, satisfying the qualifying sufficient condition outlined in Theorem 2 (i.e., T > F). Intuitively, as the size of N increases, the lower bound on AGAIN grows exponentially, leading to a substantial improvement in AAGAIN We observe that Theorem 2 holds depending on the condition set by Lemma 4 (i.e., U = I.T and U I.F). Indeed, U I,T and U = L.F are inherently met when the rules governing G adhere to prior logic. More specifically, if factor satisfies @ @r and its neighboring variables adopt the same value as ground-true -ruth labels, then its potential function value must be e. In this case, the conditional probability of the potential function is fixed.
C EXPERIMENTAL DETAILS
C.1 DATASETS AND DATA PREPROCESSING
All data processing and experiments are executed on a server with two Xeon-E5 processors, two RTX4000 GPUs and 64G memory. We construct logical nule sets the three datasets respectively. The detailed information of data processing on both datasets is summarized as follows:
MNNIHEtic-MNIST The SyNII dataset is a composite dataset derived from the origi- nal MNIST dataset. Each category of the MNIST handwritten digits treated as a concept, and four digits from different categories are concatenated to form a SynIhetic-MNIST sample. Conse- quently, each Synthetic-MNISST sample contains concept labels and a synthetic category label. The Synthetic-MNIST comprised 79,261 samples from 12 synthetic categories. The mapping of each synthetic category to concepts is shown in Table 9 According to Table 9 we construct the first-order logical rule set for manthetccMMIST The samples, whose category label is O, taken as examples. We construct cetleory-conceppy rules: co + yo- C2 + yo, C2 + you Cg + Yn! concept- concept rules: op C1, C2 Cg Ca (es, C4 C, C7 Cg, Cg C5. Note that the logical rules used on the
18

---

Published as a conference paper at ICLR 2025
Table 6: Statistics of rule and factor graphs constructed on three datasets.
Synthetic -NNIST
MIMIC
Dataset
CUB
-III EWS
Logical Rule
concept
33
11,300
144
Concept concept
13
3763
35
Concept Variable
10
112
22
Factor
Calegury Variable
12
200
Graph
Logical Factor
46
15,063
179
Synthetic-MNIST concept label Figure 9: Category labels and concept labels for Synthetic-NNIST.
Synthetic-MNIST category label
synthetic dataset can encompass the entire knowledge required for the downstream tasks. Therefore, randomly omit subset of these rules to simulate the incompleteness of explicit knowledge
The CUB (Caltech-UCSD Birds-200-201 1) dataset comprises 11,788 images of birds dis- tributed across 200 categories. Each image is annotated with 312 high-level semantic labels, such as wing color and beak shape, in addition cial semantic labels as concepts following denoising process. Furthermore, potential associations between concepts and categories are manually labeled by the bird experts and quantified as an asso- ciation score in the interval [0, 100j. Association scores approaching 100 or 0 signify significant degree coexistence or exclusion, respectively, between the concept and the category. Conver- gence towards 50 indicates the absence of any discernible association. We formulate the logical rule set based on the association scores. Association scores near 100 are regarded as category-concenn rules representing coexistence constraints, while association scores approaching 0 are identified as category-coneee rules indicating exclusion constraints. To construct concept-concept rules, we cre- ate score vector that includes association scores corresponding We calculate the Pearson correlation coefficient (PCC) between each score vector, an elevated PCC between score vectors suggests a greater similarity between the two concepts. As illustrated in Fig- ure 8 we visualized the association scores of 6 sets of exclusive concepts and 6 sets of coexisting concepts across 200 categories, respectively. We observe that for concept groups with PCCs >0.8, their association scores are positively linearly correlated. While concept groups with the PCC <-0.5 showed negative linear correlation. Therefore, we construct coexistence rules for groups of concepts with PCCs 0.8 and exclusion rules for groups of concepts with PCCs < -0.53
CUB.
single category label. We have retained 112 cru-
concept under each category.
MIMIC-III EWS. The MIMIC-III EWS dataset is medical dataset for early warning score (EWS) prediction task, comprising electronic health records from 17,289 patients. The EWS the patient's vital sign score, ranging from 0 15. Deviation from normal vital signs results in an increase in EWS. We utilize 15 input attributes of patients to predict 16 labeled categories (cach integer value of EWS as a category). Additionally, we predefine 22 concepts related to vital signs (such as body temperature, mean blood pressure, etc.) based on the Hard AR (Havasi et al.) 2022) recommendations for generating explanations. We directly establish category-concept rules based on the existing probability analysis results from Hard AR and formulate concept-concept rules using cosine similarity]
The statistics of the rules and factor graphs constructed on the aforementioned dataset are presented Table illustrating the number of each type of logical rules and each type nodes in G.
hute://yaannLeeumccom//a//a////s 2000 htm httt:///ww.issoo..lleeeeeh.h....
http://yaam..ecun.com
19

---

Published as a conference paper at ICLR 2025
C.2 CONCEPT-LEVEL INTERPRETABLE MODEL BASELINES
To evaluate the comprehensibility of the explanations, we compared AGAIN with 8 popular concept- level interpretable model baselines. The baseline models are listed as follows:
CBM (Koh et al. 2020a is a classical interpretable neural network implemented with the concept bottleneck structure.
Hard AR (Havasi et al.,
2022) expands the concept set with side-channels, and predicts
categories with binary concept vectors.
ICBM (Chauhan al.
2023 introduces interactive policy learning with cooperative pre- diction to filter important concepts.
PCBM Yuksekgonul et al.) 2023) introduces a concept-level self self-interprettation module that automatically captures concepts through multimodal models.
ProbCBM (Kim et al., 2023) uses probabilistic concept embedding to model uncertainty in concept predictions and explains predictions in terms of the likelihood that the concept exists.
Label-free CBM OOikarinen et al. 2023) introduces a multimodal model to generate a predefined concept set for the inputs and learns the mapping between input features and the concept set.
ProtoCBM (Huang et al. 2024) utilizes cross-layer alignment and cross-image alignment to learn the mapping of different parts of the feature map to concept predictions, thereby promoting the model to capture trustworthy concept prototypes.
ECBMs (uu al. 2024) utilizes conditional probabilities to quantify predictions, concept corrections, and conditional dependencies to capture higher-order nonlinear interactions between concepts for improving the reliability of concept activations.
C.3 EVALUATION METRIC
P-ACC. P-ACC measures the validity of concept-level explanations by reapplying rectified explanations to the category predictor and examining the outcomes of category predictions.
E-ACC. E-ACC measures the similarity between a concept-level explanation and the ground-truth explanatory concept set, which is computed as follows:
E-ACC EmEm [rram Cm
(20)
where êye,m represents the binary transformation result for the m-th concept activation, and Gm denotes the ground-truth label of the m-th concept.
LSM. Since the comprehensibility of explanations is based on prior human logics, the extent to which an explanation adheres to logical rules can be utilized as a criterion for evaluating its comprehensibility. In this experiment, we formulated LSM to quantify the extent to which an explanation adheres to the prior human logics rules. This metric is employed to evaluate the comprehensibility of the explanation. A higher LSM indicates higher comprehensibility of the explanation. In particular, we calculate the LSM by consid- ering the weighted sum of potential functions within the factor graph. For explanation a, the definition of LSM is as follows:
exp (EIEN wryls))
LSM = E
(21)
IheN Wse
The maximum value for LSM is 1, signifying that the explanation adheres to all the logical rules encoded in G.
IR and SR. In this experiment, IR and SR are devised to evaluate the capacity of the factor graph for recognizing perturbations. IR quantifies the rate at which perturbed instances are identified by the factor graph, computed E [PP (Ve (V") 0 vIP (V"(>") x + 8). while SR measures the rate at which benign instances are allowed to pass through the factor graph, computed as E [P ( ( 2 IP (ve 1 lar]
20

---

Published as a conference paper at ICLR 2025
Remarkably, only CBM. ProtoCBM, ECBMs, and ProbCBM baselines suitable to Synthetic- MNIST dataset, as the concepts in this dataset is synthetic and cannot be automatically captured by other baselines.
C.4 IMPLEMENTATION DETAILS
AGAIN is implemented in PyTorch 1.1.0 based on Python 3.7.13. We construct & by instantiating the as a Markov logic network in Pracmin 1.2.4. Within the AGAIN, a fully connected layer utilized as the category predictor. Inception V3 (Koh et al., 2020a), popular convolutional neural network structure, employed as the concept predictor trained on real-world datasets. We use convolutional neural network with two convolutional layers and normalization operations as the concept predictor trained on Synthetic-MNISST. We train the concept predictor (real-world datasets) for 500 epochs, the concept predictor (Synthetic-MNISST) for 30 epochs, and the category predictor for 15 epochs. We leverage the sgd optimizer with a learning rate of 0.01 to optimize the model. We mitigate the overfitting, weight decay of 0.00004 was configured. In the experiment, 0 is set to 0.9. All experiments were repeated 4 times and the average of the results is reported.
C.5 ESTIMATION OF WEIGHTS
Due to the differences in the datasets, we use two methods to set rule weights the factor graphs: prior setting and likelihood estimation. For the CUB dataset, we use the prior-based weighting method, as it provides predefined confidence for the mapping relations between concepts and cate- gories. Therefore, treat the weights as hyperparameters and directly convert the confidence into corresponding weights.
For the MIMIC-III EWS and Synthetic-MNEL datasets, weights as these datasets do not contain confidence information. Specifically, we directly apply standard maximum likelihood estimation Yang et al. (2022) to leam the weights of each factor. We assume that the samples in the training set should satisfy all logical rules when there is no perturbation. Therefore, after assigning the concept activations of the samples to the factor graph, the weights of the factors should maximize the conditional probability I2 (Ve [VV). In summary, we minimize the negative log-likelihood function:
likelihood estimation to learn the
{-> log (P (V (P VV www
w arg min
(22)
where N denotes the number of samples in the training set, and w denotes the weight vector formed by concatenating the weights of all factors in the factor graph.
D MORE EXPERIMENTAL RESULTS
D.I ABLATION ANALYSIS OF FACTOR GRAPHS
To validate the necessity of using factor graphs, we conducted set of ablation studies comparing the impact on LSM with and without factor graphs across three datasets. The altered model without factor graphs is denoted as "w/o Factor Graph," which only uses predefined set of logic rules, and the model with factor graphs is denoted as "w/ Factor Graph," which encodes the logic rules using factor graph. The experimental results, shown in Table 7 indicate that the model using factor graphs consistently yields better LSMs across all three datasets compared the model that uses only the logic rule set. This improvement is attributed to the uncertainty reasoning capability of factor graphs, which can estimate the confidence levels of different rules and assign appropriate weights accordingly. Notably, on the Synthetic-MNIST dataset, the LSMs achieved with just the logic rule set are comparable to those with factor graphs. This is because most logic rules in the Sontheti-MNNIS
dataset are synthesized and deterministic by nature. Purely deterministic logic rules are often less applicable in real-worke scenarios. Therefore, factor graphs offer an irreplaceable advantage in detecting erroneous explanations in real-world scenarios.
21

---

Published as a conference paper at ICLR 2025
Table 7: Comparison of LSM with and without the factor graph.
6g.
6g
Daraset
Methods
c=8
c=16
c=32
c=8
c=16
cm32
w/o Factor Graph
89,065.2)
87, 37.6(6.7)
87.1(7.1)
(66.2)
89.0(5.2)
87.666.2
(7.1(7.1)
86.166.2)
CUB
w/ Facter Graph
92.4(1.2)
93.1(2.3)
93.8(1.9)
91.3(1.7)
94.5(1.))
93.3(1.7)
93.8(1.4)
92.1(2.1)
MIMIC
w/o Factor Graph w/ Factor Graph
19.9(4.1)
89.8(7.2)
88.9(7.6)
87.4(6.9)
89.9(4.1)
89.8(7.2)
88.9(7.6)
87.4(6.9)
-111 EWS
96.1(0.7)
94.2(11.4
(1.2(1.2)
94.2(1.2)
94.0(2.7)
94.1(6.3)
94.2(2.4)
92.3(4.7)
Synthetic
w/o Factor Graph
98. ((0.8)
97)
96 9(4
94.9(388)
98. 1(0.8)
91.3(1 S)
96 06.9(4.9 9
94.9(3.8)
MNNST
w/ Factor Graph
98.2(
97.9(1.3)
97.8(1.4)
95.6(1.2)
98.2(1.4)
97.9(1.3)
97.8(1.6)
97.8(2.0)
D.2 COMPUTATIONAL EFFICIENCY ANALYSIS
We assess the potential impact of increasing factor graph size on computational efficiency. Specif- ically, we evaluate four factor graph sizes, containing 30, 60, 90, and 112 concepts, on the CUB dataset, measuring their running time and LSM performance, respectively. To the best of our knowl- edge, CUB is currently the dataset with the highest number of concepts (with 1 112 concept labels). In addition, not necessary to discuss other datasets that have more concepts than CUB. Too many concepts can lead to lengthy explanations, reducing comprchensibility. The evaluation is performed under the unknown perturbation of c=32.
We report the running time required to detect and correct erroneous explanations and the LSM of the corrected explanations for all 4 scales of factor graphs (see Figure IO). According to the results, the running time of the factor graph is inevitably higher than that of the other baselines due to the extra steps of detection and correction required for the factor graph. However, this extra overhead can be contained the millisecond level. Furthermore, the running time is indeed proportional to the size of the factor graph, but there is no exponential explosion, suggesting that AGAIN has good scalability.
80
Mog (mo)
L.SM
60
20
%
112
Numbee concept variables the factor graph
ISM(AGAANN)
SMC RM)
1.HM AR) LMMPPotcC)MM
LSMBCRR) 1.SM(ECHM)) log (PC3))))
SMO RM) KAGAIN PPobCCBM)
L.MMPUbCCBM)
L.bel Roo CBM)
(CIBM)
log 103lad
((CUM) (eg(FCRN
log ahel--ffe CBM)
Igg(PoooCCMM
Figure 10: The computational efficiency analysis of factor graphs with different sizes.
D.3 ANALYSIS OF INTERVENTION STRATEGIES
We compare the computational overhead associated with different intervention strategies for cor- recting concept activations. Specifically, we use the greedy strategy and the heuristic strategy as baselines for comparison. For the heuristic strategy. we provide the indexes of incorrect concepts, while for the concept intervention strategy, we supply the indexes of factors with potential function of O. We evaluate the computational overhead of these three strategies on the CUB dataset under perturbation of e=32. 5 samples are selected and randomly modified between and 10 concepts in each sample. Figure II illustrates the average number of interventions performed by different strategies across these samples. The experimental results indicate that the number of interventions increases for all three strategies as the number of incorrect concepts rises. When the number of incorrect concepts is fewer than 10, the differences in the number of interventions among the three strategies are minimal. Notably, the number of perturbed wrong concepts is typically small. For instance, in the CUB dataset, the number of wrong concepts is typically only I to 10.
In addition, we theoretically analyze the computational complexity of the intervention strategy of AGAIN, (V,A,=) -} is proved to be sufficient to express all logical relations. Based on this, if two concepts are randomly selected from a set of M concepts, there are at most M(M-1) possible com- 2 binations. The maximum number of distinet logical rules between two concepts is 8, specifically:
22

---

Published as a conference paper at ICLR 2025
80
Greedy Strategy
Nunberr 10 1
Heuristic Strategy
60
Ours
40
eentions
20
0
1
2
3
5 7
9
10
Number of wrong concepts
Figure 11: Comparison of intervention numbers for different intervention strategies.
Table 8: Comparisons of LSM for our AGAIN on the SaanheticMNNSS dataset.
6
Method
clear
c=4
c=8
c=1
c=8
97.8(0.))
92.613.5)
86.315.4
ProbCBM
98.6(1.1)
92.914.2)
87.3(3.4)
92.4(3.2)
86.713.11
PreteCBM
97.3(0.2)
94.510.8)
90.7(1.2)
94.70.7)
88.9(4.3)
BCBMs
97.5(0.7)
92.613.5)
88.3(2.8)
92.6(3.4)
88.7(2.7)
CBM-AT
93.500.8)
93.012.4)
90443.1
92.6 6.2,0)))
86.3(1.3)
ProbCBM-AT
93.5(0.7)
92.912.71 7)
90.4())6
92.4(3.)
86.7(2.1)
ProteCBM-AT
96.1(1.7)
92.1911.)
90.7(1.0)
87.5(1.4)
84.3(3.
BCBMs-AT
95.9K1.4)
90.4(4.2)
89.5(1.7)
8..001.2)
84.7(2.5)
LEN
98.6(0.3)
97.611
95.5
98.6(1.4)
95.5(2.9)
DKAA
98.(1.0)
98..)
96.((188
98.1(0.8)
96.7(1.8)
MORDAA
97.9(0.8)
98.5(0.9)
95.8(1.6
98.5(0.9)
95.8(16)
DecpProblog
97.5(1.4)
97.613.5)
95.633
98.6(3.5)
95.613.5)
98.7(0.1)
97.7(0.7)
95.1(1.2
70.70)
C-HMCNN
98.1(0.4)
96.8(1.2)
94.5(1.6)
96.8(1.2)
94.5(1.6
(Ours)
98.9(1.3)
97.8(14)
95.601.)
97.8(1.))
B;AV -V of factors is 4M - 1). With a maximum of 3 interventions per rule (intervene A, intervene B, and both), the total number of interventions is 12M (M 1), resulting in complexity of ((M²).
AA
AV-
- B Thus, the maximum number
D.4 EXPERIMENTAL RESULTS ON SYNTHETIC-MNIST
Comprehensibility of Explanations. Table illustrates the LSM results of our proposed AGAIN in comparison with 10 baselines on the Syathetic-MNIST dataset. The results indicate that the ex- planations produced by AGAIN for the synthetic dataset are equally comprehensible, implying that the performance of AGAIN exhibits generalizability, Specifically, when the unknown perturbation magnitude is 8, the LSM of AGAIN exhibits an average increase of 47..1..8 41.83% compared to the baseline with attributional training. Furthermore, in benign environments, the LSM AGAIN outperforms other baselines due to its ability to maintain optimal model parameters without adjusting them to accommodate the effects perturbations.
Rectification of Interactive Intervention Switch. Figure (c) illustrates the visual representa- tions of the rectified explanations generated by AGAIN. The activated handwritten digit concepts in the explanations align seamlessly with the semanties of synthetic images, confirming the logi- cal completeness and semantic richness of the rectified explanations. This outcome validates the continued effectiveness of interactive intervention switch strategy on the synthetic dataset.
D.5 E-ACC AND P-ACC oN EACC-MIMIC-IIEEN AND SYNTHETIC-MNISS
We present the E-ACC of AGAIN on EACC-MIMIC-IIEWS and CyytthetccMNNS"" in comparison to all baseline methods, as shown in Tables
g and 10 respectively. The experimental results show
that AGAIN is optimal for E-ACC on two datasets.
23

---

Published as a conference paper at ICLR 2025
Secondly, report the P-ACC of AGAIN the two datasets, as shown in Tables II and 12 respectively. Notably, since perturbations do not impact the final predictions, the P-ACC remains consistent across different levels of perturbation. Furthermore, the factor graph does not improve the predictive accuracy of the categories, making the P-ACC of AGAIN comparable to that of the other baselines. Notably, since the methods based knowledge integration are not retrained. Therefore, there is no difference between the effects of known and unknown perturbations on these methods. Therefore, their results under known and unknown perturbations are the same.
Table 9: Comparisons of E-ACC between AGAIN and baselines on MIMIC-II EWS.
5y
Method
clear
c=8
c=16
c=32
c=4
8
c=16
<=32
CRM-AT
97.1(1.2)
92.8(1.1) I
90.3(1.3)
87.2(3.1) 1)
85.4(1.7)
90.5(1 4)
87.7(1.1)
84 84.1(2
76.8(2.1
Hard AR-AT
97.700.4)
92.9(2.1)
89.9
87.4(2.4
84.3(1.4
90.11244
84,100.6
80.3(3.1)
78.866
ICBM-AT
97.8(14)
91. 01.9114)
8..811))
84.5(2.7)
83.2(3.2)
89.61
85.902.3)
82.7(2.1)
(9.1(4.)
PCBM-AT
96.9K1.7)
92.6(1.)
83.200.8)
83.012.1)
90.5(1
84.6(2.7)
81.5(3.2)
77.43.9)
ProbCBM-AT
98.1(0.3)
92.101.)
90.311.1)
87.4(1.3)
84.4(2.4)
90.012.2)
85.5(1.2)
84.2(1.0)
81.663.2)
Label free CBM-AT
97.1(16)
93.1(2.1)
90.3(1.5)
8(35()
84.6/2.3)
91.012.5)
86.7(1.7)
23.7(2.2)
80.6(4.1
ProtoCBM-A
97.8(U.8)
93.4(1.))
92.1(1.0)
86.7(1.4)
k5.6/2.1)
92. 100.6)
89.5(1.2)
85.8
81.2(2.4)
BCTMs-AT
9..0K
9..6(17)
92.7(2.6)
88.3(1.7)
88.1(2.))
90.4(1.))
86.4(2.2)
83.((18) 8)
80.2(3.1)
LEN
97.7(0.2)
92.2 92.2(2.4)
91.6(1.2)
86.7(2.6)
83,2(0.9)
92.212.4
91.6(1.))
86.7(2.6)
83.2(0.9)
DKAA
96.5(1.3)
91.7(13)
92.9(0.8)
87.5(1.4)
85.1(3.2)
91.7(1.3)
92.9(0.8)
87.511 4)
83.1
MORDAA
97.1(1.))
91.5(377)
84.3(1.4)
90.5(3.1)
81.5/2.1)
DeepProblog
96.3(1.2)
90.701.0)
89.7(3.1)
25.3().3)
81.7(13))
90.711.))
89.7(3.1)
85 25.2(1.3)
81.7(13)
MBM
97.901.0
91.9(2.2)
90.160.6)
86.7(0.9)
82.6(1
91.9(2.2)
90.1(0.6)
86.7(0.9)
82.6(13)
C-HMCNN
97.6KU.7)
93.4(1.1)
91.033.1)
87.4(2.))
84.9(2.8)
93.4(1.1)
91.03311)
87.4(2.4) 4)
84.9288)
AGAIN
97.3(0.1)
93.0(1.3)
93.211.3)
93.001.4)
93.0(1.2)
93.0(1.7)
93.611.6)
93.3(19)
93.211.2)
Table 10: Comparisons of E-ACC between AGAIN and baselines on Synthetic- MNIST.
8
6u
Method
clear
c=8
c=
e=32
c=8
c=16
c=32
CBM-AT
98.4(0.1)
(1.((11))
95.2(1.5)
90.3(16)
96.5(2.1)
92.1(1.5)
89.4(2.1)
Hard AR
97.9001)
97.6(0.5)
96.2(0.2)
93.7(1.2)
91.4(1.1)
95.610.5)
93.1(1.2)
91.7(1.0
17.9(21)
ICRM-AT
98.2(0.3)
98 98.210.2)
((..()
(2..5))))
90.5(1.4)
95. 111.1
94.7(1.1)
1
88.2(1.4)
PCBM-TT
97.9(0.2)
97.4(1.1)
96.4(
95 95.5(1.0)
92.3(1.3
95.2(1.3)
94.1(1.2)
91.0(14)
90.1(1.2
ProbCBM-AT
98.1(0.2)
07.910.2)
95.11.0)
93.1 93.1(1.4)
90.5(1 3)
94.3(1
93.1(.2)
91 91.0(0.8)
88.2(1.1)
Label-free CBM-AT
97.5(0.7)
97.710.6)
95.9(1.)
92.501.3)
91.6(1.)
95.
92.9(.11)
90.2(4.2)
87.2(2.1
ProtoCBM-Al
98.800.3
98.5(1.2)
97.1(12)
95.6(2.1)
93.9(2.1)
94.1(1.0
92.5(2.1)
90.2(3.2)
89.3/2.4)
ECBMs- AT
98.3(0.6)
98.1(0
97.1(1.2)
95.3(1.3)
92.1(2.5)
95.2(1.1)
92.5(1.7)
90.1(1.3)
(8.6(1.6)
LEN
98.41.1)
98.4(1.2)
96.9(144
95.4(0.9)
92.8(1.))
98.4(1.2)
96.9K1.4)
95-4(0.9)
92.8(1.2)
DKAA
98.6(0.7)
98 98.4(0.9)
97,2(1.4)
94.1(1.2)
92.5(2.1)
98.4(0.9)
97.2(1.4)
94.1(1.2)
92.5(2.1)
MORDAA
98.3(0.7)
97.9(1.2)
95.2(1.2)
92.1(1.)
90.6(2.6)
97.9(1.2)
95.2(1.2)
91.1(1.4)
90.6(2.6)
Desprrbblog
97.5(1.1)
97.3(2.1)
95.3(2.4)
92.712.1)
89.4(3.2)
97.3(2.1)
95.3(2.4)
92.7(2.1)
89.443.)
MRM
98.7(1.3)
97.4(1.5)
96.4(1.3)
91.6(13)
89.4(2.2)
97.4(1.5)
96.421.3)
89.4(2.2)
C-IMCNN
98.8(0.9)
96.7(2.1)
95.1(2.4)
93.2(2.5)
90.5(2.5)
96.7(2.1)
95.1(2.4)
93.2(2.5)
90.5(2.5)
AGAIN
98.1(0.5)
98.1(0.5)
97.3(1.)
97.3(2.9)
97,2(2.1)
98.2(0.9)
97.4(1.0)
97.1(1.)
96.8(1.4)
Table 11: Comparisons of P-ACC between AGAIN and basclines on MIMIC-III EWS.
8
6"
Method
clear
c=4
c=8
e=16
e=32
c=4
e=8
c=16
e=32
CBM-AT
49.7(1.2)
49.6(1.2)
49.1(1.1)
49.2(1.4)
49.6/1.3) 13
49.7(1.2
49.3(2.4)
49.8(1.1)
49.5(1.4)
Hard AR-AT
48.5(2.1)
49.1r1.5)
((1.2)
49.5(1.1)
49.6
49. 111 ()
49.1(2.3)
48.9(11)
49.0(1.6)
ICBM AT
49.1(1.6)
49.61147
(9.1(1.4)
49 (9.5(1.2)
49.4(1.1)
48.9(1.1)
49.4(1.4)
41.7(1.7)
49.9(1.4)
PCBM-AT
1..6(144
49.9r1.3) 3)
49,8(2.1)
50.1(1.3)
49.8(1.))
49.7(1.2)
49.1(1.3)
49.3(14)
10.1(11)
PT
49,8(0.8)
49.8:1.0)
51.4/2.1)
50.2(1.3)
49.6(1
50.2(1.1)
49.4(1.3)
49.6(1.1)
49.7(0.9)
Label-free CBM-AT
49.7(1.3)
49.7(1.2)
49.7(111
49.611.4
49.6(1.1)
49.5(1.5)
49.611.2)
49.7(1.1
ProtoCBM-AT
49.6(1.2)
49.5(1.1)
48.9(2.0)
49.1(1.4)
49.3(1.3)
49.3(1.2)
48.0(2.1)
49.0(1.3)
49.3(1.2)
BCBMs-AI
50.9(1.5)
50.2(1.3)
49.6(2.0)
49.8(1.3)
49,8(1.6
49.5(0.7)
49.6(2.5)
49.3(1.2)
49.8(1.2)
49.6(1.6)
49.7(1.3)
8(.4)
49.5(1.6)
49.711.3)
49.4(1.4)
40../).4)
49.5(1.61
DKAA
49.4K1.1)
49.8(0.1)
50.500.2)
49.701.2
49,8(1.)
49.8(0.1)
50,500.2)
49.7(1.2)
49.8(1.2)
MORDAA
49.5(0.1)
49.8(1.2)
49.7(1.3)
40.4(1.4
49.4(1.6
49.8(1.2)
49.7(1.3)
10.
49 40.4(166)
Deeprrobiog
48.41.2)
48.7(1.3)
48.4112)
48.5(1.))
48.2(1.)
48.7(1.3)
48.4(12))
48.5(1.1)
48.2(1.1)
MBM
49.4K1.3)
50.100.6)
49.3(1.7)
49.400.5)
49.3(1.7)
50.110.6)
49.3(1.7)
49.4(0.5)
49.341.7)
C-HMONN
49.4(0.8)
49.5(1.1)
49.4(1.1)
49.2(1.5)
49.2(1.2)
49.5(1.1)
49,4C1.1)
49.2(1.5)
49.2(1.2)
AGAIN
55.1(2.9)
49.7(1.1)
49.1(2.2)
49.4(0.1)
49.3(2.7)
48.0(1.5)
48.3(1.2)
49.4(1.9)
52.5(0.9)
E IMPLEMENTATION DETAILS OF ADVERSARIAL ATTACKS
In this section, we provide implementation details of adversarial attacks against concept-level expla- nations. Specifically, adversarial attacks against concept-level explanations can be categorized into three types: erasure attacks, introduction attacks, and confounding attacks.
24

---

Published as a conference paper at ICLR 2025
Table 12: Comparisons of P-ACC between AGAIN and baselines SyNthetic-MNIST.
8x
6
Method
clear
c=4
t=8
c=16
c=32
c=4
e=8
c=16
c=32
CBM-AT
98,200.1)
98. 2(0.1)
((0.4)
98. (8,,)
98. ((0.5)
51 (1
98.5(0.6)
97.6 97.600.3)
98.2(0.5)
Hard AR-AT
98.4(0.3)
98. (8.3)
98.4(0.2)
98.4(0.2)
98.2(0.1)
98. 1(0.3)
98.8(0.5)
98. 18.160.6)
98.2(0.5)
ICBM-AT
97.5(1.4)
98.3(0.1)
97.400.2
98. 400.8
97.3(0.5)
98.2(1.1)
98.5(0.3)
97.1(0.1)
98.6 98.6)0.3)
PCBM-AT
98.2(0.7)
98.3(0.4)
97. 97.4(2.1)
98.4(1.3)
98.4(0.1)
98.4(0.1)
97.5:0.2)
98.50..3)
97 97.440.8)
ProCBMM AT
98.1(0.2)
98.4(0.2)
98.3(0.7)
98.2(0.1)
98.2(0.7)
98.4(0.8)
98.2(0.5)
98.1(
98.2(0.6)
Label-free CBM-AT
98.200.5)
98.4(0.2)
98.40.66
98.5(0.2)
98.8(0.6)
98.300.5)
98.2/0.2)
97.910.8)
98.2(0.4)
90.6007)
98.4(0.7)
98.4(0.4)
98.3(0.2)
98.4(0.))
98.6(
BCBMs AT
99.1(0.2)
98.5(0.4)
98.6(0.5)
98.3(0.7)
97.2(0.9)
98.3(0.7)
98.2(0.7)
98.5(0.8)
98.3((16)
LEN
98.2(0.4)
98.4(0.3)
98.2(0.1)
98.4(0.5)
98.3(0.2)
98.2(0.1)
98.4(0.5)
98.3(0.2)
DKAA
98.4(0.))
98.6(0.2)
97.900.1)
98.200.2)
98.3(0.1)
98.610.2)
97.9K0.1)
98.2(0.2)
98.3(0.1)
MORDAA
98.620.1)
98 30..)
97.9(0.2)
90.5.
98.1(0.3)
98.3().2)
97.9(0.2)
98.510.7)
98, (00.)
DeepProbiug
97.2(0.1)
97.2(0.4)
97. 17.1)
97 9(..(
97.1(0.4)
97.210.4)
97.1(0.3)
97.5(0.1)
97. 77.10.4)
MBM
98.2/0.1)
98 20.3)
98. 1(0.
98 50..)
98. ((0.4)
98. 3)
98. 98.1(0.2)
98. 98.5(0.1)
98. ((0.4
C-HMONN
98.2(0.1)
98.2(0.2)
18.1(
98.5(0.6)
98. (0.2))
98.2(0.2)
(0.0(
98.5(0.6)
((0.)
AGAIN
98.3(0.5)
98.2(0.6)
98,1(0.7)
98.2(0.1)
98.2(0.6)
98.5(0.3)
98.7(0.4)
98.2(0.7)
98.3(111)
Erasure Attacks. Erasure attacks attempt to subtly remove a specific concept without changing category prediction results. Gaps in perception and missing concepts are confusing for analyst and very difficult to detect. For CBMs, we typically have a pre-set threshold ?y for determining
whether concept is activated concept. Specifically, for a sample x, We use he (m)
(x:) to denote
the activation value of the m-th concept output by the concept predictor he(-). ! (x) - Y > O,
then the m-th concept is an activated concept. Attackers learn adversarial perturbations for executing crasure attacks by the following objective function:
MAX Σ
(I'T- (1m)
(x + +8]] 1o
II[7 he ho (m))
(23)
mEMg argmax hy (he (x + 0)) = argmax hy (he (x))
S.L.
where II(-) denotes the indicator function, hc(-) is the concept predictor. and (.) is the category predictor (task predictor). 5 is the learnable perturbation. Me denotes the concept set that the attacker wishes to delete away.
Introduction Attacks. The purpose of introduction attacks is manipulate the presence of irrel- evant concepts without modifying the classification results. Such attacks hinder accurate analysis of model explanations. Attackers attempt to introduce new irrelevant concepts that do not previously exist in the concept set of the original sample. Unlike erasure attacks, introduction attacks require to raise the activation value of irrelevant concepts above the threshold y. Therefore, the objective function of introduction attacks is represented as follows:
MAX Σ (AITA, (m)) (I + - I[he (m)) (x:) (m) -1))
mEM, argmax by (he (x + 6)) argmax by (he (z)).
(24)
S.t.
where M, denotes the set of concepts that the attacker wishes to introduce.
Confounding Attacks. Confounding attacks build on introduction attacks and erasure attacks. Thee confounding attack simultaneously removes relevant concepts and introduces irrelevant concepts. The confounding attack is a more powerful attack than the crasure attack and the introduction attack as it allows arbitrary tampering with the concept set of the original sample. The objective function of confounding attacks is as follows:
MAX Σ (aILY -oy-he. he (m)
(x + +5] If-- - he (x))) 7]
mEME
+ Σ (aTh, (m²) (xI -1 - -Ilho (m2) (x) - -1l)
(25)
mEM;
s.t. argmax by (he (a + +)) = argmax By (he (x)).
In this paper, confounding attacks are utilized to disrupt AGAIN, which contributes to a more com- plete evaluation the comprehensibility of the explanations generated by AGAIN when concepts are missing and confused.
25


Reviews:
[{"review_id": "8E3wXXrj9e", "reviewer": "Reviewer_XsjD", "summary": "The paper proposed a method using factor graphs to correct errors in concept-based explanations caused by perturbations. \nSpecifically, it constructs a factor graph using predefined logical rules between concepts or between concepts and categories. \nThis graph helps identify logical errors in the output from concept-based interpretable neural networks by evaluating the likelihood of these errors. \nAdditionally, by leveraging the factor graph, it is possible to correct these logical errors in the explanations. \nExperimental comparisons on three datasets demonstrate that the proposed method outperforms existing concept-based approaches.", "questions": "- Defining the logic rule set seems expensive. Would it be difficult to construct a factor graph by connecting concepts and categories with a bipartite complete graph and estimating the weights $w_i$?\n- How do you distinguish between coexistence and exclusion in the factor graph?\n- It would be better to describe the specific estimation algorithm of $w_i$ in the Appendix.\n- Line 336-337: What is \"attributional training\"?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed method enables both the detection of logical errors and the correction of these errors within a single framework.\n- Compared to other concept-based interpretable neural networks, the proposed method achieves higher comprehensiveness in explanations, regardless of whether the perturbations are known or unknown.", "weaknesses": "- While the proposed method assumes that explanations change due to perturbations without affecting predictions, this seems unrealistic. Particularly in interpretable neural networks with a concept bottleneck structure, as assumed in this study, changes in the concepts outputted by the neural network would naturally lead to changes in predictions, which undermines this assumption.\n- The proposed method requires predefined logic rules between concepts and categories. If these rules are already known, wouldn’t it be possible to detect inconsistencies between concepts and predictions without the need for the factor graph? The advantage of using a factor graph is unclear.\n- As noted in the minor comments below, there is room for improvement in the writing.\n\nMinor comments:\n- The explanation of Figure 3 is insufficient.\n- There is no reference to Figure 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nsPWX5z2ZK", "reviewer": "Reviewer_XLZi", "summary": "The paper introduces AGAIN (fActor GrAph-based Interpretable Neural Network), which generates comprehensible explanations for neural network decisions even under unknown perturbations. Unlike traditional adversarial training methods, AGAIN incorporates logical rules through a factor graph to identify and rectify explanatory logical errors during inference. The model is validated on three datasets: CUB, MIMIC-III EWS, and Synthetic-MNIST, demonstrating superior performance compared to existing baselines", "questions": "- Can the authors provide a comparison against one of the methods injecting knowledge (prior or learnt)?\n- Scalability: The two limitations that have been reported (domain knowledge changes, correct prediction categories) are non-negligible. How could this method be extended to face these limitations?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Good presentation and structure** The paper is well-structured and easy to read. The mathematical definition of the proposed method is clear and well-defined. \n- **Nice experimental campaign**: Extensive experiments on three datasets (CUB, MIMIC-III EWS, and Synthetic-MNIST) demonstrate the superior performance of AGAIN over the compared baselines, although these results hold mainly for a metric (LSM) that has been created ad-hoc.", "weaknesses": "## Major Issues\n\n- **Novelty, Related work and Compared methods**: the main issue with the current paper is that it only considers methods injecting knowledge into the models by means of factor graphs. However, the integration of knowledge into model predictions has been fully studied under different point of views: probabilistic (e.g., DeepProblog [1] and all its variants), logic constraints (see the survey of Giunchiglia et al. [2]). Also, it is not the first method defending against adversarial attack with knowledge and without employing adversarial training. Some of these methods have been already employed to defend against adversarial attacks, such as [3-4]. [5] is a survey entirely dedicated to enhancing interpretability and adversarial robustness with prior knowledge. [6] has shown in the context of concept-based models that it can learn logic rules at training time and use them at inference time to defend against adversarial attacks. This is also reflected in the experimental campaign that is extensive but does not consider any methods injecting prior knowledge to defend against adversarial attacks. The only compared methods are CBMs or adversarial-trained versions of the same models. \n\n\n- **Paper positioning and Preliminaries**: the method provides explanations and a defence mechanism that is based on concept predictions; thus, it applies only to concept-based models. Most of the compared methods also belongs to this niche. While this is fully acceptable, explicit references to concept-based models only appears in Section 3-4. Therefore, I think it should state earlier that this is the focus of the paper, as most of the related work mentioned in the paper does not focus on concept-based explanations. Furthermore, there is no explicit mentions to concept-based models in the preliminaries. The “Interpretable Neural Network” paragraph should include citations to this literature and explain concept-based models.\n\n## Minor Issues\n- P.2 “[…] even if the adversarial samples are available, retraining is only effective for at most one perturbation type” I think this statement is quite strong, and although it may have been proved for some attacks in Tan & Tian, 2023, I don’t think it is a general assumption that we can make. I think this sentence should be soften to “retraining is effective only for few perturbation types at the same time”. \n- P.2 “[…] to ensure the expectation of the potential function is in its maximum value”. It is not clear at this point what is the potential function. Consider rephrasing this sentence.\n- P.2 “The explanations that are further regenerated align with the exogenous logic.” Not clear in this case what are the exogenous factors. \n- P.2-3: The term \"defenses against comprehensibility\" seems a bit misleading. It implies that the goal is to prevent explanations from being understandable, which is not the case. Instead, the focus is on defending the comprehensibility of explanations against perturbations. “defenses of comprehensibility” colud be a more appropriate definition.\n\n\n[1] Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, Luc De Raedt: DeepProbLog: Neural Probabilistic Logic Programming. NeurIPS 2018: 3753-3763\n\n[2] Giunchiglia, E., Stoian, M. C., & Lukasiewicz, T. (7 2022). Deep Learning with Logical Constraints. In L. D. Raedt (Ed.), Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22 (pp. 5478–5485). doi:10.24963/ijcai.2022/767\n\n[3] Yin, M., Li, S., Cai, Z., Song, C., Asif, M. S., Roy-Chowdhury, A. K., & Krishnamurthy, S. V. (2021). Exploiting multi-object relationships for detecting adversarial attacks in complex scenes. In proceedings of the IEEE/CVF international conference on computer vision (pp. 7858-7867).\n\n[4] Melacci, Stefano, et al. \"Domain knowledge alleviates adversarial attacks in multi-label classifiers.\" IEEE Transactions on Pattern Analysis and Machine Intelligence 44.12 (2021): 9944-9959.\n\n[5] Mumuni, Fuseini, and Alhassan Mumuni. \"Improving deep learning with prior knowledge and cognitive models: A survey on enhancing interpretability, adversarial robustness and zero-shot learning.\" Cognitive Systems Research (2023): 101188.\n\n[6] Ciravegna, Gabriele, et al. \"Logic explained networks.\" Artificial Intelligence 314 (2023): 103822.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GOAgS16QMF", "reviewer": "Reviewer_PFRk", "summary": "This paper proposes AGAIN (fActor GrAph-based Interpretable Neural Network), a new approach to maintain comprehensible neural network explanations when inputs are affected by unknown perturbations. AGAIN builds factor graphs from explanations, and integrates logical rules through the graphs. The system detects adversaries by identifying violations of real-world logic, and uses an interactive intervention strategy to fix incorrect explanations. Tested on three datasets (CUB, MIMIC-III EWS, and Synthetic-MNIST), AGAIN outperformed existing methods in generating comprehensible explanations under both known and unknown perturbations.", "questions": "How is the reasoning conducted on G? How is the probability estimation implemented in Equation 1? Please provide more details.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper proposes a novel and interesting idea that uses the logical correctness of explanation to detect and defense against noises and adversaries.\n2. The three stage framework is reasonable to me. \n3. The examples used in the paper are intuitive.", "weaknesses": "1. The concept bottleneck model is not easy to scale, as it requires manual annotations of concepts.\n2. The implementation details of Section 4.2 is not very clear (the introduction is too conceptual). For example, is $\\mathcal{G}$ a graph or a model? At the beginning I thought $\\mathcal{G}$ is a graph, but in Line 221 it \"reasons about a conditional probability\".\n3. The datasets used in this work are not very strong. I doubt if the work is applicable to real-world situations. At least, the datasets used cannot reflect adversarial scenarios in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pCGvMcJek0", "reviewer": "Reviewer_SU9s", "summary": "This paper proposes AGAIN, a neural network model that generates comprehensible explanations under unknown perturbations by integrating logical rules directly during inference, rather than relying on adversarial training. Using a factor graph to identify and rectify logical inconsistencies, AGAIN demonstrates superior interpretability and robustness compared to existing methods.", "questions": "1. The authors employed black-box perturbations to test robustness. Adding input noise could provide more convincing evidence.\n2. I am concerned about the potential impact of large factor graphs on computational efficiency. I would like to see an analysis on this problem.\n3. Currently, the algorithm explores all intervention options to select the optimal one. Is it possible to employ a simplified strategy, such as a heuristic or greedy algorithm, to reduce computational load?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents an innovative method, AGAIN, that combines factor graphs with concept-level explanations to improve model interpretability under unknown perturbations.\n2. The authors evaluate AGAIN across multiple datasets and baseline models, providing a broad view of its effectiveness.\n3. The paper is clear and well-structured.", "weaknesses": "1. The factor graph requires predefined logical rules, which could be challenging to construct or generalize across different domains.\n2. The algorithm’s process of exploring all possible intervention options to find the optimal solution could create computational overhead.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lnpjRcGF8K", "reviewer": "Reviewer_YnPb", "summary": "In this paper, the authors propose a novel method called AGAIN to generate comprehensible explanations under unknown perturbations by employing a factor graph approach. The method presents a significant contribution to the field, addressing an important gap with a new perspective. The paper is well-structured, and the authors have provided thorough theoretical justifications and experimental analyses. These analyses effectively demonstrate the superiority of the proposed method over existing ones.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper offers a well-motivated and clear presentation of a unique approach. The use of a factor graph to handle perturbations is innovative and highly relevant to the research community. Furthermore, the authors have provided sufficient theoretical support and empirical evidence to justify the effectiveness of their approach.", "weaknesses": "1. Clarification on Figure 3:\n   In Figure 3, does the final output include hat{c}_{re} and prediction? Does the hat{c}_{re} mean the interpretation? Additionally, the figure lacks clarity on how hc is generated, which is crucial for understanding the method. Adding details on hc generation would make the figure more comprehensive.\n \n2. Comparison Metrics:\n   The authors compare their method with all baselines using the LSM metric. However, they only compare their method with CBM in terms of accuracy (P-ACC and E-ACC). It would be beneficial to extend the accuracy comparison to include all baselines to provide a more complete evaluation of the method’s performance.\n \n3. Figure 7 Interpretation and Readability:\n   In Figure 7, for each example, the authors provide two bar charts. Does the left bar chart represent the initial interpretation, and the right bar chart represent the interpretation combined with the factor graph? Clarification on this aspect would enhance the understanding of the figure. Additionally, some symbols and text in Figure 7 are overlapping.\n \n4. Inconsistency in Line 463 and Table 5:\n   The authors mention in line 463 that their method is compared with two baselines on the Synthetic-MNIST dataset. However, Table 5 lists four baselines. Furthermore, \"ProbCBM\" in Table 5 should be corrected to \"ProCBM\" for consistency. It is recommended that the authors proofread the paper to eliminate such inconsistencies and typographical errors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)