Neural Networks Decoded: Targeted and Robust Analysis of Neural Network Decisions via Causal Explanations and Reasoning

## Abstract

Abstract Despite their success and widespread adoption, the opaque nature of deep neural networks (DNNs) continues to hinder trust, especially in critical applications. Current interpretability solutions often yield inconsistent or oversimplified explanations, or require model changes that compromise performance. In this work, we introduce Tracer , a novel method grounded in causal inference theory designed to estimate the causal dynamics underpinning DNN decisions without altering their architecture or compromising their performance. Our approach systematically intervenes on input features to observe how specific changes propagate through the network, affecting internal activations and final outputs. Based on this analysis, we determine the importance of individual features, and construct a high-level causal map by grouping functionally similar layers into cohesive causal nodes, providing a structured and interpretable view of how different parts of the network influence the decisions. Tracer further enhances explainability by generating counterfactuals that reveal possible model biases and offer contrastive explanations for misclassifications. Through comprehensive evaluations across diverse datasets, we demonstrate Tracer ’s effectiveness over existing methods and show its potential for creating highly compressed yet accurate models, illustrating its dual versatility in both understanding and optimizing DNNs.

## 1 Introduction

Neural networks have demonstrated transformative potential across various applications, notably image classification (Krizhevsky et al., ), medical diagnostics (Esteva et al., ), and complex pattern recognition (LeCun et al., ), even surpassing humans in certain domains (Silver et al., ; Rajpurkar et al., ). Yet, their inherent complexity obscures their decision-making processes, turning them into “black boxes” that raise transparency and trust concerns, thus impeding their adoption in sectors requiring explainability, such as healthcare and cybersecurity (Zeiler & Fergus, ; Castelvecchi, ; Doshi-Velez & Kim, ; Lipton, ; Papernot & McDaniel, ; Zhang et al., ). Neural Network Explainability, pivotal in Explainable AI (XAI), aims to clarify DNN decision-making to ensure trust, ethical application, and bias mitigation. Although various XAI strategies have been proposed, including saliency maps (Zhou et al., ), Grad-CAM (Selvaraju et al., ), LIME (Ribeiro et al., ), and SHAP (Lundberg & Lee, ), they often present inconsistencies, over-simplification, or architectural constraints, underscoring an ongoing challenge in DNN understanding (Baehrens et al., ; Ba & Caruana, ; Rudin, ).

In this paper, we introduce Tracer, a novel approach based on causal inference theory (Pearl, ), to infer the mechanisms through which AI systems process inputs to derive decisions. Recognizing that conventional evaluation metrics based solely on validation datasets may not be indicative of a model’s performance in real-world settings and drawing inspiration from Pearl’s causal hierarchy, our approach reveals how targeted modifications to input features influence the internal states of neural networks, thereby modelling the underlying causal mechanisms. Specifically, Tracer frames the explainability of neural networks as a causal discovery and counterfactual inference problem, where we observe and analyze all intermediate and final outputs of a model, given any sample, its generated set of interventions, and its counterfactuals. Through the aggregation of multiple such instances, we provide interpretability to state-of-the-art models without requiring any re-training or architectural changes, thus preserving their performance. In conjunction with an efficient approach for counterfactuals generation, this offers contrastive explanations for misclassified samples, expanding our understanding of not just what happened, but why it happened, and what could have happened under different conditions, thus enabling the identification of potential model blind spots and biases, and addressing the overarching issue of trust. Our main contributions can be summarized as follows:

- •
We propose Tracer, a framework for estimating the causal mechanisms underpinning DNN decisions, combined with a conditional counterfactual generation method for identifying failure modes, providing actionable insights for improving classifiers.
- •
We perform comprehensive evaluations of Tracer on image and tabular datasets, providing explanations for correct and misclassified samples, while highlighting its effectiveness in discovering the causal maps that describe the key transformation steps involved in decisions.
- •
We demonstrate Tracer’s versatility in both local and global explainability, as well as its ability to outperform prevalent explanation techniques, identify redundancies in neural network architectures, and aid in the creation of optimized, compressed models.

## 2 Related Work

Various techniques have been developed for DNN interpretability, typically categorized by explainability scope, implementation stage, input/problem types, or output format (Adadi & Berrada, ; Angelov et al., ; Vilone & Longo, ). Early endeavours like saliency maps by Zhou et al. (), Grad-CAM (Selvaraju et al., ) and Layer-wise Relevance Propagation (LRP) (Bach et al., ) visually highlighted key features in input data. Such visual explanation methods often produce inconsistent or coarse explanations, or require structural model changes, sometimes compromising performance or overlooking individual nuances crucial for true comprehension (Rudin, ). Model-agnostic approaches, such as LIME (Ribeiro et al., ) and SHAP (Lundberg & Lee, ) offer explanations by approximating model decision boundaries. However, these methods potentially face challenges such as resource intensiveness or inconsistencies in local explanations. While some works attempt to simplify complex DNNs to improve their interpretability (Che et al., ; Frosst & Hinton, ), they often compromise on performance, as simpler models cannot always capture the nuances of complex DNNs. In contrast to the aforementioned methods, rather than merely highlighting influential features, Tracer estimates the causal dynamics that steer DNN decisions, without the need for altering the model or compromising its performance.

Different from associative methods, causal inference techniques probe deeper, seeking to both understand statistical correlations and uncover the true cause-effect relationships between variables. The idea of merging causal inference with AI is an emerging perspective, advocating for a more robust form of explainability. rior works on causal inference for AI have primarily revolved around the use of causal diagrams and structural equation models to gain such associative understanding (Pearl, ; Yang et al., ; Xia et al., ; Kenny et al., ; Chou et al., ; Geiger et al., ; Kelly et al., ). For instance, methods such as those proposed by Kommiya Mothilal et al. () and Chockler & Halpern () perform causal reasoning to explain decisions made by image classifiers, focusing on identifying causal elements in the input space, whereas Tracer extends causal reasoning deeper into the structure of DNNs themselves. By combining causal analysis of both the model’s internal workings and the input-output relationships, our approach enables explainabibilty at both the feature and network-structure level, providing more comprehensive explanations for DNN behavior.

Recent advances in explainability have also emphasized the importance of counterfactual explanations (Feder et al., ), which generate hypothetical instances showing how small changes in inputs would alter a model’s prediction. Deep generative models proposed for this task, such as those based on Variational Autoencoders (VAEs) (Pawelczyk et al., ; Antorán et al., ) or Generative Adversarial Networks (GANs) (Mirza, ; Nemirovsky et al., ), typically focus on producing counterfactuals that minimize the required changes to the input features. Our approach improves on these by introducing a dual objective that not only ensures realism through adversarial training but also aligns generated counterfactuals closely with their nearest neighbors in the target class. This guarantees that the counterfactuals generated by Tracer are both plausible and interpretable.

Our proposed approach sets itself apart in two main aspects: (1) rather than only focusing on input features, our approach performs an intervention-based analysis that additionally examines the causal mechanisms within the DNN architecture, identifying how specific layers causally influence the decision-making process, thereby inferring the critical components (critical layers) within DNNs; and (2) a conditional counterfactual generation method, which synthesizes realistic alternative scenarios to identify model blind spots and biases, while ensuring the generated counterfactuals remain plausible and target specific outcomes through controlled feature changes.

## 3 Theoretical Foundations and Methodology

To understand the internal-workings of DNN architectures, we must consider not only the operations performed by individual layers, but also how they influence one another across the network. Tracer aims to estimate an accurate model of these mechanisms, focusing on the dynamics that govern the network’s decisions. Therefore, our methodology, depicted in Figure 1, is structured around:

Figure: Figure 1: Overview of Tracer. Interventions and counterfactuals are used to determine the effects of individual features on the models’ intermediate and final outputs, leading to the discovery of the mechanisms underpinning the decision-making process.
Refer to caption: x1.png

Causal discovery: We analyze the interactions and dependencies within DNNs by systematically altering input features to observe the resulting changes, enabling an effective mapping of the decision pathways. Through this process, we estimate the causal structures that drive the network’s decisions, providing a clear understanding of how different features and layers contribute to the outcome.

Counterfactual generation: We simulate alternative scenarios by introducing targeted changes to input features, allowing us to explore ‘what-if’ scenarios and observe how specific changes in inputs can lead to different outcomes, providing further insights into the model’s sensitivity and robustness.

### 3.1 Causal Theory

Causal theory provides the means to model cause-effect relationships, offering a departure from mere observational statistics to tackle questions about interventions and counterfactuals (Pearl, ). To this end, the language of Structural Causal Models has been proposed to formalize these relationships.

###### Definition 1 (Structural Causal Model) .

A Structural Causal Model (SCM) $\mathcal{M}$ is a 4-tuple $(U,V,\mathcal{F},P(U))$, where $U$ is a set of exogenous variables determined by factors external to the model; $V=\{V_{1},V_{2},\ldots,V_{n}\}$ is a set of endogenous variables, each influenced by variables within the model; $\mathcal{F}=\{f_{1},f_{2},\ldots,f_{n}\}$ is a set of functions, each $f_{i}$ mapping a subset of $U\cup V$ to $V_{i}$; and $P(U)$ is a probability distribution over $U$. For every endogenous variable $V_{i}$, its value is determined by $V_{i}=f_{i}(\text{pa}(V_{i}),U_{i})$, where $\text{pa}(V_{i})$ represents the parents or direct causes of $V_{i}$, and $U_{i}\subseteq U$.

Pearl’s Causal Hierarchy (PCH), grounded in SCMs, further refines our understanding by categorizing causal knowledge into three distinct levels, which serve as Tracer’s foundations:
1. Association: We extract dependency structures from the DNN activations and outputs $P(Y^{(i)}\ |\ X)$, where $X$ and $Y^{(i)}$ represent the input and the $i$-th layer’s output variables, respectively;
2. Intervention: By selectively manipulating feature values, we estimate the intervention distributions $P(Y^{(i)}\ |\ do(X=x_{j}))$ to understand the effect of particular features on the final decision;
3. Counterfactual: We explore alternative (or hypothetical) input scenarios and compute the counterfactual distributions, $P(Y_{X=x^{\prime}}^{(i)}\mid X=x)$, which quantifies the model’s output distribution if a certain input were set to a particular value, given that we actually observed another input.

By systematically identifying how specific input features and intermediate layer activations influence the model’s final predictions, Tracer provides a unique approach, based on the principles of PCH, for capturing an abstract overview of the distinct computational components driving DNN decisions. This structured approach allows us to produce explanations that clarify both the direct influence of features and how the model’s predictions would change under different input conditions, thus providing a more comprehensive understanding of the decision-making process.

###### Definition 2 (Explanation) .

Given a $d$-dimensional input $\mathbf{X}=\mathbf{x}\in\mathbb{R}^{d}$, an explanation for the output $y$ of a model $\mathcal{F}$ is a masked input $\mathbf{x}_{E}=\mathbf{x}\odot\mathbf{M}\in\{0,1\}^{d}$ for which the following conditions hold:

- C1
(Correctness): The model $\mathcal{F}$, when evaluated on the input $\mathbf{x}$, produces the output $y$.
$(\mathcal{F},\mathbf{x})\models(\mathbf{X}=\mathbf{x})\text{ and }\mathcal{F}(
\mathbf{x})=y.$
- C2
(Sufficiency):
There exists a mask $\mathbf{M}\in\{0,1\}^{d}$ such that the resulting explanation $\mathbf{x}_{E}=\mathbf{M}\odot\mathbf{x}$ produces the same output as the original input: $(\mathcal{F},\mathbf{x}_{E})\models\mathcal{F}(\mathbf{M}\odot\mathbf{x})=
\mathcal{F}(\mathbf{x})=y.$
This condition ensures that the features selected by $\mathbf{M}$ are sufficient to explain $y$.
Let a mask $\mathbf{M}^{\prime}$ be defined such that $\mathbf{M}^{\prime}\not\subseteq\mathbf{M}$, then $(\mathcal{F},\mathbf{x}^{\prime}_{E})\models\mathcal{F}(\mathbf{M}^{\prime}
\odot\mathbf{x})\neq y.$
- C3
(Minimality):
The mask $\mathbf{M}$ is minimal, meaning that no strict subset $\mathbf{M^{\prime}}\subset\mathbf{M}$ satisfies C2. That is, for every mask $\mathbf{M^{\prime}}\subset\mathbf{M}$, the masked input $\mathbf{x^{\prime}}_{E}=\mathbf{M^{\prime}}\odot\mathbf{x}$ is insufficient to produce the same output: $\forall\;\mathbf{M^{\prime}}\subset\mathbf{M},\quad(\mathcal{F},\mathbf{x^{
\prime}}_{E})\not\models\mathcal{F}(\mathbf{M}\odot\mathbf{x})=y.$
C3 guarantees that $\mathbf{M}$ includes the smallest set of features necessary to explain $y$.

Note that (i) in this definition, $y$ can be set to any specific label to produce explanations for misclassifications or rare events; and (ii) partial explanations can be simplified to binary decisions (i.e., whether a feature is relevant or not) when computing feature attributions (defined in Section 3.3).

###### Definition 1 (Structural Causal Model) .

###### Definition 2 (Explanation) .

### 3.2 Causal Discovery

To discover a faithful representation of the causal mechanisms underpinning DNN models, we perform an intervention-based analysis where we systematically change the values of input features and study the effects on a given classifier. Such interventions allow us to prove the models and measure the effects of specific changes on the classifier’s representations. By observing the internal states and outputs of the classifier, we can deduce how specific components contribute to the final decisions, offering an understanding of the model’s causal structure and enabling the identification of key nodes or connections that highly influence the model’s predictions. Furthermore, by collecting the observed effects of all interventions, we establish an abstract causal map to visualize the interplay between different network components, and asses their collective influence on the DNN outputs. Ultimately, these detailed visualizations can potentially be instrumental for debugging or refining classifiers, or for designing more interpretable neural network architectures.

#### 3.2.1 Interventions

In our analysis, interventions are crucial for isolating and understanding the causal significance of specific input features. Given an input vector $x\in\mathbb{R}^{d}$, where $d$ denotes the dimensionality of the input space, an intervention is simulated by replacing a subset of $x$ with a predetermined baseline value $b$. For a specified subset of indices $I\subseteq\{1,\dots,d\}$ corresponding to the features under intervention, the intervened features are given by: $x^{\prime}_{i}=b\cdot\mathbb{1}_{\{i\in I\}}+x_{i}\cdot(1-\mathbb{1}_{\{i\in I
\}})$, where $\mathbb{1}_{\{i\in I\}}$ indicates 1 when $i$ is in the set $I$ and 0 otherwise. Assuming $b$ to be causally independent (e.g., binary mask), all input features, before and after interventions, can be considered exogenous variables in the causal map.

###### Proposition 1 (Causal Isolation of Intervened Samples) .

Let $F:\mathcal{X}\rightarrow\mathcal{Y}$ denote the mapping function of a DNN. For any $x\in\mathcal{X}$, $I\subseteq\{1,\ldots,d\}$, and $b\in\mathbb{R}$, the intervened sample $x^{\prime}$ isolates the causal effect of the features in $I$ on $F$ by setting the values of $x_{i},\forall i\in I$ to $b$. (Proof in Appendix A.1)

By performing such interventions, we effectively isolate and examine the causal impact of specific features on the output. Through these controlled perturbations, we can determine which features are causally pivotal for the model’s decisions, and measure the depth of their influence.

The chosen baseline values can carry significant importance in our intervention framework. Much like in cooperative game theory where Shapley values (Shapley et al., ) use baselines to understand the contribution of each player by averaging their marginal contributions across all possible coalitions, they have been adapted for interpreting machine learning models (Lundberg & Lee, ), inspiring our use of baselines to serve as neutral points of reference. Specifically, in our interventions, the baseline value aim to counteract or neutralize the impacts of the specific features being altered. This allows us to isolate the original input’s influence on the output without the bias introduced by those features. By contrasting the results from such intervened input with the original, we gain deeper insights into the causal relationships between input features and model outputs.

###### Proposition 1 (Causal Isolation of Intervened Samples) .

#### 3.2.2 Causal Abstraction

By systematically collecting intermediate and final outputs of the classifier, given an input sample and its interventions, Tracer enables a focused comparison of representations across network layers and extrapolates an accurate estimation of the causal dynamics driving the network’s decisions. For this analysis, we use Centered Kernel Alignments (CKA), a prevalent approach for quantifying the similarity between high-dimensional embeddings. Let $f_{i}$ and $f_{j}$ denote the activations of two distinct layers in a neural network for a set of input samples, and let their respective kernel matrices be defined as $K_{i}=f_{i}f_{i}^{T}$ and $K_{j}=f_{j}f_{j}^{T}$. Their CKA similarity can then be obtained as:

$$ $\text{CKA}(K_{i},K_{j})=\text{HSIC}(K_{i},K_{j})/\sqrt{\text{HSIC}(K_{i},K_{i} )\times\text{HSIC}(K_{j},K_{j})},$ $$

where $\text{HSIC}(K_{i},K_{j})$ is the Hilbert-Schmidt Independence Criterion (HSIC) for the kernel matrices, and given by $\text{HSIC}(K_{i},K_{j})=(n-1)^{-2}\ \text{Tr}(HK_{i}HK_{j}).$
Here, $H$ is a centering matrix given by $H=I-\frac{1}{n}\mathbf{1}\mathbf{1}^{T}$, with $n$ being the number of samples, $I$ the identity matrix, and $\mathbf{1}$ a vector of ones. $\text{Tr}(\cdot)$ denotes the trace of a matrix.
The use of CKA for evaluating representation similarity offers several advantages, including: (i) Normalization: CKA scores range from 0 (completely dissimilar) to 1 (identical), allowing straightforward comparison across layers; (ii) Flexibility: It accommodates various kernel functions, such as linear or Gaussian, enabling flexibility based on specific requirements of the analysis; and (iii) Robustness: The use of kernels allows CKA to operate in a richer feature space, providing a more comprehensive similarity measure. Given these properties, CKA stands out as a suitable choice for our similarity analysis of feature representations.

Upon obtaining the similarity measures, we establish causality by grouping layers based on their CKA values, where we create a binary matrix $\mathcal{B}(K_{i},K_{j})$, which is defined as $\mathcal{B}(K_{i},K_{j})=1\text{ if }\text{CKA}(K_{i},K_{j})\geq 1-\epsilon,
\text{ and }0\text{ otherwise}$, with $\epsilon$ representing a predetermined threshold that defines the maximum acceptable dissimilarity for two layers to be considered alike. For our causal analysis, such similarity suggests that these layers contribute to a shared causal node representing an endogenous variable and describing a distinct structural equation in our causal model.

###### Definition 3 (Layer Groups) .

Let $F(x)=f_{0}\circ\ldots\circ f_{k}(x)$ denote the compositional form of the neural network classifier, with $f_{i}$ representing the $i$-th layer of the network. And let $\mathcal{B}$ denote the binary CKA matrix. Two distinct layers $f_{i}$ and $f_{j}$ are said to belong to the same layer group if and only if $|i-j|=1$ and $\mathcal{B}(K_{i},K_{j})=1.$

###### Theorem 1 ( Layer Grouping ) .

Let a sequence of layers $\{f_{i},f_{i+1},\ldots,f_{j}\}$ within a neural network $F(x)$ be classified under the same Layer Group if $\mathcal{B}(K_{i},K_{i+1})=\mathcal{B}(K_{i+1},K_{i+2})=\ldots=\mathcal{B}(K_{
j-1},K_{j})=1$, where $\mathcal{B}(K_{i},K_{j})=1$ if $\text{CKA}(K_{i},K_{j})\geq 1-\epsilon$. The collective causal influence of this sequence on $F(x)$’s output is encapsulated by a single composite layer $g_{ij}$: $F^{\prime}(x)=f_{0}\circ\ldots\circ g_{ij}\circ\ldots\circ f_{k}(x)$, where $g_{ij}\equiv f_{j}\circ f_{j-1}\circ\ldots\circ f_{i}\equiv f_{i}$. (Proof in Appendix A.2)

This definition of “Layer Groups” aggregates layers into cohesive groups, where each group estimates a distinct node in the decision mechanism of the network. Through this aggregation, we effectively abstract the composition of layers into single causal nodes when their computations are found to be redundant, allowing for a more streamlined and high-level understanding of the network’s processes.

###### Theorem 2 ( Necessary and Sufficient Conditions for Causal Nodes ) .

Let $F:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$ be a DNN defined by composition as $F=f_{k}\circ\ldots\circ f_{1}$ where each $f_{i}:\mathbb{R}^{d_{i-1}}\rightarrow\mathbb{R}^{d_{i}}$ represents the transformation applied by the $i$-th layer, $d_{0}=n$, and $d_{k}=m$. Let $g=g(r)=\ldots=g(s)=\{f_{i}\}_{i=r}^{s}$ with $1\leq r<s\leq k$ be a subset of consecutive layers. $g$ constitutes a causal node if and only if
$\forall i\in\{r,\ldots,s-1\},\text{CKA}(K_{i},K_{i+1})\geq 1-\varepsilon,$ where $K_{i}$ is the kernel matrix of layer $i$ and $\varepsilon\in(0,1)$ is a predefined similarity threshold. (Proof in Appendix A.3)

###### Definition 4 (Causal Links between Layer Groups) .

Let $g_{a}$ and $g_{b}$ denote two distinct layer groups within a neural network. A causal link between $g_{a}$ and $g_{b}$ is established if either they are adjacent, meaning there exists at least one pair of layers $f_{i}\in g_{a}$ and $f_{j}\in g_{b}$ such that $|i-j|=1$, or if there is at least one pair of layers $f_{i}\in g_{a}$ and $f_{j}\in g_{b}$ for which $\mathcal{B}(K_{i},K_{j})=1$.

By adopting definitions 3 and 4, we capture the internal dependencies of DDNs, leading to the discovery of layer-wise abstractions that describe the structural equations governing our causal model. This enriched perspective allows for more powerful explanatory modelling through better understanding of the interplay between layers, and how they collectively shape the network’s decisions. Consequently, our approach offers valuable insights into the high-level causal mechanisms that shape the network’s behavior, and allows us to provide an abstract, structured, and interpretable view of the causal dynamics that are intrinsic to its operations.

###### Definition 3 (Layer Groups) .

###### Theorem 1 ( Layer Grouping ) .

###### Theorem 2 ( Necessary and Sufficient Conditions for Causal Nodes ) .

###### Definition 4 (Causal Links between Layer Groups) .

### 3.3 Estimation of Causal Effects

To quantify the causal impact of interventions on the neural network’s outputs, we define the Average Causal Effect (ACE) to quantitatively capture both the direction and magnitude of the effect caused by altering the input features.

###### Definition 5 (Average Causal Effect) .

Let $g_{i}^{\prime}(x)=\text{softmax}(g_{i}(x))$ and $g_{i}^{\prime}(x^{\prime})=\text{softmax}(g_{i}(x^{\prime}))$ denote the normalized outputs of a Layer Group $g_{i}$ for a given input $x$ and its intervention $x^{\prime}$. The normalization of these outputs is performed to transform the activation scores into valid probability distributions, with which the Average Causal Effect (ACE) can be defined as the expected value of the product of the signed Kullback-Leibler (KL) divergence between their probability distributions:

$$ $\text{ACE}_{i}=\mathbb{E}_{P(X)}\Big{[}|\Delta_{x}^{i}|\cdot\text{KL}\big{(}P( g_{i}^{\prime}(x)\;|\;do(X=x^{\prime}))\;\|\;P(g_{i}^{\prime}(x)\;|\;do(X=x)) \big{)}\Big{]},$ $$

where $\Delta_{x}^{i}=g_{i}^{\prime}(x)-g_{i}^{\prime}(x^{\prime})$ represents the sign of the change induced by the intervention, and $\text{KL}(\cdot)$ represents the KL divergence quantifying the changes between the probability distributions.

This definition provides a robust estimation of the causal effects, offering a comprehensive view of how specific interventions are reflected within the neural network.

###### Remark .

Any intervention that produces outputs sufficiently similar to those produced by the original input has little to no impact on the Average Causal Effect.
If the intervention on input $x$ to produce $x^{\prime}$ results in minimal change in the output of a Layer Group $g_{i}$, such that $g_{i}^{\prime}(x)\approx g_{i}^{\prime}(x^{\prime})$, then with all other features of $x$ remaining untouched, the change induced by $x^{\prime}$ approaches 0, leading to minimal or negligible contribution. Formally, if $g_{i}^{\prime}(x)\approx g_{i}^{\prime}(x^{\prime})$, then:

$$ $\text{KL}\big{(}P(g_{i}^{\prime}(x)\;|\;do(X=x^{\prime}))\;\|\;P(g_{i}^{\prime }(x)\;|\;do(X=x))\big{)}\approx 0\implies\text{CE}_{i}=0.$ $$

This suggests that interventions which do not substantially alter the output of a Layer Group have a negligible causal impact on the model’s output, as measured by the ACE. Our approach henceforth consists of generating interventions, such that those with no effect according to our definition above, are considered not part of the explanation.

###### Definition 5 (Average Causal Effect) .

###### Remark .

### 3.4 Counterfactual Generation

To improve classification performance and mitigate model biases, we additionally explain misclassified samples through a Tracer analysis of counterfactuals, highlighting specific feature changes that should be applied to samples in order to obtain the desired outputs. Counterfactuals are hypothetical data instances that, if observed, would alter the model’s decision. Crafting such instances is challenging due to the constraint that all counterfactuals should be valid and plausible. Therefore, using generative models such as Generative Adversarial Networks (GANs) (Goodfellow et al., ), we aim to achieve this task by including such constraints into our training process. Particularly, we propose a novel plausibility constraint, whereby the counterfactual generators are trained using both adversarial training to ensure realism, and a proximity-based regularization term to enforce similarity between the generated counterfactuals and real instances from a target class. This ensures that the counterfactuals are realistic and also require minimal changes to the original data. While our proposed constraint can be adapted to various types of generative models (e.g., VAE, GAN, normalizing flows), the model we discuss hereinafter assumes an autoencoder-based GAN architecture.

Given an input $x\in\mathbb{R}^{d}$ and a target output $y^{*}$, our GAN-based counterfactual generation model is defined such that the generator uses an encoder function $E_{x}$ to map the input $x$ to a condensed latent representation $z_{x}=E_{x}(x)$. The desired model output $y^{*}$, typically an integer label, is transformed into a one-hot encoded vector $o(y^{*})\in\mathbb{R}^{k}$, where $k$ is the number of classes, using the Kronecker delta function $o_{i}(y^{*})=\delta_{iy^{*}}$ for $i=1,\ldots,k$. This latent representation $z_{x}$, concatenated with the one-hot encoded target label $o(y^{*})$ to form an augmented latent vector $z=[z_{x};o(y^{*})]$, is processed by the decoder $D$ to generate a counterfactual instance $x^{*}=D(z)$.
To verify the authenticity of the generated counterfactual $x^{*}$, the discriminator $\mathcal{D}$ evaluates whether $x^{*}$ appears realistic and plausible by distinguishing between original data samples and those produced by the generator.

The GAN is optimized using a dual objective: (1) ensure the authenticity of the generated counterfactual $x^{*}$ and (2) maximize its similarity with its nearest neighbour $x_{\text{nn}}$ among real samples of its training dataset whose label correspond to some target class. This objective can be seen as a combination of a conventional GAN loss and a proximity measure $d(x^{*},x_{\text{nn}})$, with $\lambda$ as the balancing coefficient: $\mathcal{L}=(1-\lambda)\ \mathcal{L}_{\text{GAN}}+\lambda\ d(x^{*},x_{\text{nn
}}),$ ensuring that generated counterfactuals remain minimally different from real instances in the target class, thereby preserving plausibility while leading to the desired model prediction. This approach offers a flexible and data-efficient process that aligns the generated counterfactuals closely with the actual data distribution, while conditioning on priors for controlled outputs.

###### Remark .

The regularization distance $d(x^{*},x_{\text{nn}})$, essential for maintaining plausibility, can be implemented using metrics such as $\ell_{1}$, $\ell_{2}$, or perceptual loss. By introducing minor perturbations $\delta_{i}$ to the latent representation $z_{x}$ before decoding, training with this regularization distance enables the generation of multiple distinct plausible counterfactuals $x^{*}=D([z_{x}+\delta_{i};o(y^{*})])$.

Employing generative models for counterfactual generation, rather than relying on nearest neighbors during inference, offers several advantages. First and foremost, relying on real data points as counterfactuals would require storing large datasets, potentially leading to memory constraints. This could be particularly problematic in applications where storage is expensive or limited. To address this, we train the counterfactual generator on a small random subset of the training set (e.g., 10%), which is afterwards discarded, eliminating the need for storage. Moreover, this allows us to generate plausible, novel counterfactuals on-the-fly, avoiding computational costs and latency associated with dataset searches, while enabling broader exploration of the feature space.

###### Remark .

## 4 Experiments

In this section, we evaluate our proposed explainability method, Tracer, emphasizing both its causal discovery and its counterfactual analysis facets. We perform our initial experiments using the well-known MNIST (Deng, ) and ImageNet (Deng et al., ) datasets, which are standards in image classification tasks, and on the CIC-IDS 2017 (Sharafaldin et al., ) network traffic dataset to demonstrate Tracer’s applicability to tabular datasets. The MNIST dataset provides a collection of handwritten digits that is ideal for the scrutiny of our methodology, while the diversity and scale of ImageNet offer a broader context for evaluating our approach’s effectiveness across a wide range of image recognition challenges. We use pre-trained AlexNet (Krizhevsky et al., ) and ResNet-50 (He et al., ) models as our MNIST and ImageNet classifier, respectively, and design a GAN architecture tailored to our counterfactual generation tasks.

Figure: Figure 2: Counterfactual GAN architecture.
Refer to caption: x2.png

This GAN, depicted in Figure 2, consists of a CNN-based Generator for creating plausible, class-conditional counterfactuals, coupled with a CNN-based Discriminator analyzing the authenticity of the generated images, is designed as follows: (1) the encoder uses four convolutional layers to transform the inputs into latent embeddings which are then merged with class information via one-hot label encodings; (2) the decoder uses transposed convolutions to construct the counterfactual inputs from the augmented latent representations produced by the encoder.

The Adam optimizer (Kingma & Ba, ) is used for training, with a learning rate of $10^{-3}$.

Through our experiments, we seek to provide a comprehensive understanding of Tracer’s capabilities and the insights it offers into DNN decision-making processes.

### 4.1 Causal Discovery and Feature Attributions

Figure: Figure 3: Tracer’s causal analysis results for an MNIST sample classified by AlexNet. The causal structure is inferred using CKA similarities between activation outputs from various layers. Nodes in the resulting causal graph symbolize layer groups, while the connections between them capture their causal relationships.
Refer to caption: x3.png

To evaluate the effectiveness of Tracer in uncovering the intricate causal pathways that govern decision-making in DNNs, the relationships between activations of different layers are analyzed using their CKA similarities. Comparing activations produced by the original input and its corresponding interventions illuminates the effect of these interventions on the decisions. As depicted in Figure 3, Tracer discerns layer groups forming causal nodes and identifies the causal links between them. For instance, eight activation outputs from the AlexNet classifier are observed and analyzed, revealing inherent groupings based on similarity patterns across the network layers. This observation has led to the identification of four distinct causal nodes. Notably, the lack of causal connections between non-adjacent layer groups indicated a linear causal chain that informs the network’s decision for the analyzed sample.

To quantitatively assess the reliability of Tracer, we measure how often a given model’s predictions remain consistent when key features identified by our approach are randomly perturbed.
Formally, let $f:\mathcal{X}\rightarrow\mathcal{Y}$ be the classification model, where $\mathcal{X}$ and $\mathcal{Y}$ represent the input and output spaces, respectively. For a dataset $X\subseteq\mathcal{X}$, each sample $x\in X$ is coupled with an explanation mask $M(x)$ generated by an explainability method. Let $P$ denote a perturbation function which modifies $x$ by targeting a proportion $p$ of the significant regions of the explanation. With $x^{\prime}=x\odot P(M(x),p)$ describing the perturbed sample, the reliability score for the explanations can thus be obtained as:
$~{}~{}~{}~{}S=|X|^{-1}\sum\nolimits_{x\in X}\mathbb{1}_{\{f(x)\neq f(x^{\prime
})\}},$ where$|X|$ is the number of samples in the dataset, and
$\mathbb{1}_{\{\cdot\}}$ is an indicator function returning 1 if the predictions before and after applying the explainability mask differ, i.e., $f(x)\neq f(x^{\prime})$.

Figure: Figure 4: Reliability scores of different explainability methods on the MNIST dataset.
Refer to caption: x4.png

This score captures the sensitivity of the model’s predictions to changes in areas deemed critical by the explainability method, thereby providing insights into the reliability of the explanations generated. To assess the robustness of Tracer and compare its performance against that of existing explainability methods, we use this reliability metric on explanations produced by the different approaches when evaluated on all test samples of the MNIST dataset.
The results, depicted in Figure 4, show the average and standard deviation of each method’s scores over 10 trials, demonstrating Tracer’s superior performance and consistency in producing meaningful and reliable explanations.

### 4.2 Counterfactual Analysis

In this experiment, we explore the use of counterfactuals as a means to understand causes for misclassifications, as well as identify the minimal feature changes required to obtain correct outcomes.
Through a comparison of the causal mechanisms uncovered for the misclassified sample with those for its counterfactuals, Tracer highlights structural or functional differences resulting from a model’s learned parameters, suggesting potential strategies for model improvement, such as refining the training set or implementing regularization techniques. In essence, counterfactuals offer both an intuitive understanding of model decisions and actionable insights for model enhancement. Appendix C presents such a counterfactual analysis for a misclassified MNIST sample.

Figure: Figure 5: Tracer vs existing XAI methods using an ImageNet sample classified by ResNet-50. The second row shows feature contributions from different causal nodes, while the bottom row compares the explanations provided by different methods. The sparse explanations given by SHAP and LRP may require high-resolution screens for adequate visualization.
Refer to caption: x5.png

### 4.3 Generalization and Scalability

In this experiment, we highlight the broad adaptability of our approach across various neural network architectures and datasets. To this end, we evaluate Tracer on the ImageNet dataset, as well as on a Network Intrusion Detection dataset, explaining the decisions of both simple and complex NN architectures such as MLP and ResNet-50.

Given the wide variety and realisic nature of the samples in the ImageNet dataset, its classification results with the ResNet-50 architecture provide a solid benchmark for highlighting the limitations of existing explainability methods and comparing their performances to that of Tracer.
For this comparison, we selected LIME, SHAP, LRP, and Grad-CAM as benchmarks, since they are among the most widely adopted and representative explainability methods in the literature. The results, depicted in Figure 5 show that while existing methods struggle to produce consistent explanations, Tracer provides coherent and comprehensive explanations that highlight the most important features and patterns that drive the classifier’s decisions. Further comparison of these methods, discussed in Appendix D.1, highlight more distinctions between Tracer and existing methods, particularly when using DNN architectures that exhibit complex interactions.

Diving deeper into the versatility spectrum, we challenge Tracer with the intricacies of structured data using the CIC-IDS 2017 network traffic dataset. This dataset, reflecting authentic network dynamics, unfolds a distinct set of challenges useful for evaluating explainability methods (e.g., diverse data types and intertwined correlations). For example, in an instance where a DDoS-attack-induced traffic is erroneously classified as benign (see Appendix D.2), Tracer identifies and elucidates features emblematic of the attack through its causal analysis. Specifically, Tracer reveals that features such as port numbers and data transfer dynamics are essential for the detection of such threats. Overall, the granularity and transparency of explanations provided by Tracer, especially in domains such as cybersecurity, accentuate its potential to build trust in critical applications.

### 4.4 Beyond Local Explainability

To evaluate Tracer’s capacity for global explainability, we integrated individual local explanations to form a comprehensive view of a model’s decision logic. For this task, we focus on a random subset of the MNIST dataset, processed through the AlexNet architecture, to derive causal insights underpinning the classifier’s decisions for all class samples. The results of this analysis, detailed in Appendix E, reveal significant redundancies within AlexNet’s architecture for MNIST, allowing us to design compressed representations of the model to optimize the computational efficiency.

The characteristics and comparisons of these compressed models, reported in Table 1, show that the most refined model obtained (C1) exhibits a staggering 99.42% reduction in model size with only a 0.16% drop in accuracy. This highlights Tracer’s potential for catalyzing practical innovations in DNN design and optimization, without undermining the predictive performance of these models.

**Table 1: Comparison of Tracer-assisted compressed models. $\theta$ represents the number of parameters of the models, and Speed indicates the inference time per sample.**
| Model | $\theta$ (M) | Size (MB) | FLOPs (M) | Speed (ms) | Accuracy (%) |
| --- | --- | --- | --- | --- | --- |
| AlexNet | 11.7 | 46.8 | 46.3 | $4.23^{\pm 0.4}$ | 99.64 |
| C3 | 11.5 | 46.3 | 25.0 | $3.21^{\pm 0.3}$ | 99.64 |
| C2 | 9.4 | 37.9 | 22.9 | $2.65^{\pm 0.1}$ | 99.57 |
| C1 | 0.06 | 0.27 | 13.5 | $\mathbf{1.08^{\pm 0.1}}$ | 99.48 |

## 5 Discussions and Limitations

In this study, we focused our evaluations of Tracer on white-box neural networks. However, its flexibility and design extend beyond, making it equally applicable to black-box models where the internal dynamics remain obscured and only the inputs and outputs are accessible. Under such constraints, Tracer remains valuable, offering two distinct avenues of exploration. First, it can analyze and quantify the influence of input features on the model’s prediction. Alternatively, by using a surrogate white-box model, we can effectively approximate the underlying causal mechanisms driving the predictions. This adaptability underscores Tracer’s potential in diverse environments.

While our Tracer approach is highly parallelizable by design, its depth of analysis can require a trade-off between granularity (the precision of the causal analysis determined by the number of interventions generated for each sample) and computational efficiency.

## 6 Conclusion

In this paper, we introduced Tracer, a novel approach for accurately estimating the causal dynamics embedded within deep neural networks. Through seamless integration of causal discovery and counterfactual analysis, our methodology enables a deep understanding of the decision-making processes of DNNs. Our empirical results demonstrate Tracer’s ability to both identify the causal nodes and links underpinning a model’s decisions, and also leverage counterfactuals to highlight the nuances that drive misclassifications, offering clear and actionable insights for model refinement and robustness. Beyond local explanations, we showcased the potential of our approach to capture the global dynamics of DNNs, leading to practical advantages such as novel and effective model compression strategies. Through our foundational principles and findings, we have ascertained that by producing intuitive, human-interpretable explanations, Tracer offers outstanding transparency to neural networks, significantly enhancing their trustworthiness for critical applications.

## Appendix A Proofs

### A.1 Proposition 1 [Causal Isolation of Intervened Samples]

###### Proof.

By our definition of interventions, $x^{\prime}$ is derived from $x$ by replacing the features indexed by $I$ with a constant $b$. Assuming this operation to be independent of the original data generation process of $x$ and controlled externally, for the features in $x^{\prime}$ indexed by $I$, any variation in $F(x^{\prime})$ with respect to changes in these features can be causally attributed to the intervention itself. Hence, within the scope of our analysis, the causal effect of the features in $I$ is indeed isolated by the intervention.
∎

###### Proof.

### A.2 Theorem 1 [Layer Grouping]

###### Proof.

Let $F(x)=f_{0}\circ f_{1}\circ\ldots\circ f_{k}(X)$ represent a DNN, where $X$ is the input set and each $f_{i}$ denotes a layer in the network. Given a sequence of consecutive layers $\{f_{i},f_{i+1},\ldots,f_{j}\}$, let us show that this sequence can be encapsulated into a single composite layer $g_{ij}$ under the condition that these layers belong to the same Layer Group.

According to the definition of Layer Groups, the layers $f_{i},f_{i+1},\ldots,f_{j}$ are grouped together if they exhibit high similarity as measured by their CKA scores. Specifically, the condition $\mathcal{B}(K_{i},K_{i+1})=\mathcal{B}(K_{i+1},K_{i+2})=\ldots=\mathcal{B}(K_{
j-1},K_{j})=1$ holds if:

$$ $\text{CKA}(K_{m},K_{m+1})\geq 1-\epsilon,\quad\forall m\in\{i,i+1,\ldots,j-1\},$ $$

where $K_{m}=f_{m}f_{m}^{T}$ is the kernel matrix for the layer $f_{m}$, and $\epsilon$ is a small threshold allowing minor dissimilarities. Since $\text{CKA}(K_{m},K_{m+1})\approx 1-\epsilon,\;\forall m\in\{i,i+1,\ldots,j-1\}$, the activations of these layers are highly similar and functionally redundant.

Thus, $\{f_{i},f_{i+1},\ldots,f_{j}\}$ can be treated as performing similar transformations, and represented by a composite layer $g_{ij}(x)=f_{j}\circ f_{j-1}\circ\ldots\circ f_{i}(x)\equiv f_{m}\;\forall m
\in\{i,\ldots,j\}$. Therefore, $F(x)$ can be simplified by replacing the sequence $\{f_{i},f_{i+1},\ldots,f_{j}\}$ with the composite layer $g_{ij}$. The resulting simplified network is:

$$ $F^{\prime}(x)=f_{0}\circ\ldots\circ f_{i-1}\circ g_{ij}\circ f_{j+1}\circ \ldots\circ f_{k}(x)\equiv f_{0}\circ\ldots\circ f_{i-1}\circ f_{i}\circ f_{j+ 1}\circ\ldots\circ f_{k}(x).$ $$

Thus, the collective causal influence of the layers in $\{f_{i},f_{i+1},\ldots,f_{j}\}$ on $F(x)$’s output is encapsulated by the single composite layer $g_{ij}$, which we approximate as $f_{i}$ due to the high similarity between the layers.
∎

###### Proof.

### A.3 Theorem 2 [Necessary and Sufficient Conditions for Causal Nodes]

###### Proof.

Let $B$ be a binary matrix whose elements $B(K_{i},K_{j})$ indicate whether the CKA similarity between the activation outputs of layers $i$ and $j$ meet the threshold criterion as defined in Section 3.2.2.

($\Rightarrow$) Necessity: Assume $g=\{f_{i}\}_{i=r}^{s}$ is a causal node. By definition 3, each layer $f_{i}$ in $g$ must contribute to the same overarching function that the node represents, i.e., $g=g_{i}=g_{i+1}\;\forall\;i\in\{r,\ldots,s-1\}$.

Suppose, for contradiction, that there exist at least one pair of consecutive layers $(f_{i},f_{i+1})$ within the subset $\{f_{i}\}_{i=r}^{s}$ for which $B(K_{i},K_{i+1})=0$. This means that $\exists i\in\{r,\ldots,s-1\}\;\text{s.t.}\;\text{CKA}(K_{i},K_{i+1})<1-\varepsilon$. By Tracer’s definition of a layer group, this indicates a significant change in information content between layers $f_{i}$ and $f_{i+1}$. Therefore, $f_{i}$ and $f_{i+1}$ would not be grouped together ($g_{i}\neq g_{i+1}$), thus contradicting our initial assumption that $g=g_{i}=g_{i+1}$.

($\Leftarrow$) Sufficiency: Conversely, assume:

$$ $\forall i\in\{r,\ldots,s-1\},B(K_{i},K_{i+1})=1,$ $$

indicating that each pair of consecutive layers within $\{f_{i}\}_{i=r}^{s}$ sufficiently maintains the same information content between the layers without significant loss or alteration, i.e., $g_{i}=g_{i+1}$. Let $Y_{i}$ and $Y_{i+1}$, respectively define the SCM equations for the causal nodes of layer groups $i$ and $i+1$:

$$ $\displaystyle Y_{i}=\tilde{f}_{i}(pa(Y_{i}),U_{i}),\;Y_{i+1}$ $\displaystyle=\tilde{f}_{i+1}(pa(Y_{i+1}),U_{i+1})$ $\displaystyle=\tilde{f}_{i}(pa(Y_{i+1}),U_{i+1}),$ $$

where $pa$ denotes the parent sets, and $U$ represents potential exogenous variables or internal noise within the layers.

Since $g_{i}=g_{i+1}$, then it must be that

$$ $\displaystyle Y_{i}\approx Y_{i+1}$ $\displaystyle\Rightarrow\tilde{f}_{i}(pa(Y_{i}),U_{i})\approx\tilde{f}_{i}(pa( Y_{i+1}),U_{i+1})$ $\displaystyle\Leftrightarrow\tilde{f}_{i}(pa(Y_{i}),U_{i})\approx\tilde{f}_{i} (pa(Y_{i}),U_{i+1}).$ $$

It follows that $U_{i}\approx U_{i+1}$. Hence, under the assumption of causal sufficiency and the observed similarity, we can conclude that $\tilde{f}_{i}$ and $\tilde{f}_{i+1}$ represent equivalent causal mechanisms. Therefore, $g=\{f_{i}\}_{i=r}^{s}$ functions as a unified entity, consistent with the properties of a causal node.

($\Leftrightarrow$) Thus, the condition $B(K_{i},K_{i+1})=1\;\forall~{}i~{}\in~{}\{r,~{}\ldots,~{}s-1\}$ is both necessary and sufficient for $g$ to be identified as a causal node, validating the causal grouping dictated by the similarity threshold $\varepsilon$, as defined by Theorem 1.
∎

###### Proof.

## Appendix B Feature Attributions at Causal Nodes

Figure 6 below depicts how individual features contribute to the network’s final decision. For every causal node (group of neural network layers), we highlight the top contributing features (top convolution filter output or top-3 feature outputs for linear layers). Positive contributions are distinctly marked in blue, signifying features that positively influence the network’s decision, while negative contributions are depicted in red, pointing out the features that negatively affect the decision.

Figure: Figure 6: Contribution of features within each causal node. Blue and red respectively indicates positive and negative contributions. The overlay on the input sample provides a cohesive visualization of how distinct features of the input affect the final decision via the causal mechanism discovered.
Refer to caption: x6.png

## Appendix C Detailed results: Counterfactual Analysis

The objective of counterfactual generation in the context of our research is to offer interpretable insights into the decision-making processes of deep neural networks, particularly in cases of misclassification. By examining the contrast between the original input and the generated counterfactual, we can uncover subtle features or patterns that influence the model’s decision, thereby pinpointing what changes might rectify misclassifications.

Figure: Figure 7: Illustration of a misclassified MNIST sample and its generated counterfactual.
Refer to caption: x7.png

As illustrated in Figure 7, given an initially misclassified input and a desired target label, our GAN-based counterfactual generator produces an alternative version of the input, which, when fed to the model, results in the desired outcome. The differences between the input and its counterfactual reveal the minimal modifications required for the classifier to produce the correct (desired) decision.

Figure: Figure 8: Comparison of an original misclassified input, the generated counterfactual, and the associated causal mechanisms. The variations between the original and counterfactual inputs highlight the pertinent features influencing the model’s decision-making process. (Blue: Positive contributions; Red: Negative contributions; $\text{G}_{i}$: $i\text{-th}$ Layer Group)
Refer to caption: x8.png

Through a side-by-side analysis of the causal mechanisms obtained from the predictions of the misclassified sample and its counterfactual, Tracer provides clear insights into the primary contributing factors to the initial misclassification, while also highlighting via the counterfactual’s analysis, the optimal neural pathways for the network to yield the correct (and desired) outcome. This detailed causal analysis is visually represented in Figure 8. Upon examination, we discern that a predominant portion of the input features, represented in blue, activate neurons that steer the classifier towards the produced outcome in both cases. However, the causal analysis of the misclassified sample reveals a notably more extensive set of features that oppose the predicted outcome when contrasted with the counterfactual. This observation makes it evident that Tracer not only identifies which parts of the input features support the misclassification (in blue) but also which features contradict this decision (in red). Intriguingly, while the causal graphs remain consistent for both inputs in this instance, the classifier’s activations manifest pronounced differences. This insight suggests that the model’s learned parameters might lack the flexibility to generalize enough to correctly discern the true label of the misclassified sample. To address this, potential avenues might include incorporating such misclassified instances into the training set or fine-tuning the model with regularization techniques to enhance its generalization capabilities.

This causal analysis reveals that our counterfactual generation method serves two main purposes. First, it provides an intuitive visualization for understanding the nuances of model decisions. Secondly, from a model development and refinement perspective, these counterfactuals can highlight potential vulnerabilities or biases in the model, guiding further training or fine-tuning endeavours.

## Appendix D Detailed results: Generalization

### D.1 Image Datasets

Here, we address the question of scalability of Tracer to large-scale image datasets. Given the challenges associated with the explainability of real-world images (e.g., the intricacies of pixel-level interactions, variances in image quality, or scale), we use for this task the MNIST and ImageNet (Deng et al., ) datasets, classified with the AlexNet and ResNet-50 architectures respectively. Using the ImageNet dataset, known for its vastness, diversity, and complexity, we show that Tracer overcomes the limitations of existing explainability methods.
The explanations produced by Tracer and benchmark explainability methods are depicted in Figure 9, showing that while existing methods struggle to produce coherent and comprehensive explanations, Tracer consistently reveals the core features and patterns crucial for classification decisions. The effectiveness of our proposed approach becomes even more apparent when used with complex models like ResNet-50, as it still maintains its precision despite the intricate patterns leveraged by very deep networks, emphasizing its capability to accurately discern the nuances of complex interactions within deeper architectures.

Figure: Figure 9: Comparison of Tracer results against existing explainability methods.
Refer to caption: x9.png

In contrast to Tracer,

- •
Every execution of LIME produces different explanations due to its inherent stochasticity, hindering interpretability.
- •
SHAP and LRP explanations produce misleading results due to their sensitivity to model and dataset complexities, resulting in overly detailed or sparse attributions that do not always intuitively align with the underlying data patterns.
- •
As Grad-CAM explanations are based on the coarse spatial resolution of the final convolutional layer of a DNN, this method often leads to highlighting broader regions rather than precise feature-level contributions to the decision-making.
- •
LRP and Grad-CAM, inherently designed for white-box DNNs, where internal model structures are accessible, face significant restrictions in terms of applicability and utility in scenarios involving black-box or proprietary models.

### D.2 Tabular Datasets

Transitioning from the realm of images, we further explored the efficacy of Tracer in the context of structured (or tabular) data. For this endeavour, we selected the CIC-IDS 2017 (Sharafaldin et al., ) network traffic dataset, which is representative of real-world network behaviors and patterns. This dataset poses its own set of challenges, distinct from image datasets, such as the mix of categorical and numerical attributes, the potential correlations between features, and the variance in feature scales.

Figure: Figure 10: Explainability of tabular datasets with Tracer. A sample from a Network Intrusion Detection dataset is misclassified as benign traffic rather than its correct class (DDoS attack). Negative contributions are shown in red and positive contributions in blue for the top-20 features.
Refer to caption: x10.png

The results presented in Figure 10 illustrate Tracer’s ability to provide detailed and accurate explanations beyond the image domain. For the sample explained in this figure, where a network traffic generated during a DDoS attack is considered as benign traffic by
a multi-layer feed-forward neural network classifier, we observe that the features indicative of an attack negatively contribute to the decision of the classifier. Specifically, the explanations provided tell us which features were found relevant for classifying this network traffic as an attack (i.e., Source/Destination Port numbers, frequency of communication, sizes of transferred data, etc.).

The clarity of the causal explanations obtained by Tracer for such tasks make it particularly suitable given the criticality of network intrusion detection systems in ensuring cybersecurity, where the ability to transparently understand and trust decisions can be indispensable for the practical viability of such systems.

## Appendix E Detailed results: Global Explainability

Given the effectiveness of Tracer in explaining neural network decisions for individual samples, we endeavour to evaluate its potential as a global explainability tool to paint a holistic picture of the model’s decision-making. To this end, rather than solely relying on global explanations, which might overlook individual nuances, we adopt an approach that aggregates local explanations to derive a global perspective. Specifically, using Tracer, we perform local explanations on a strategically selected subset of the dataset, aiming to capture a representative understanding of the overall characteristics. For this experiment, we selected the MNIST dataset classified using the AlexNet architecture as before. While without loss of generality, simply performing random sampling within all classes suffices for this experiment, by using clustering algorithms (Settles, ; Olvera-López et al., ) or Proximally-Connected graphs (Diallo & Patras, ), more optimal sampling policies can also be adopted to identify and select the most influential samples. Our findings for this experiment revealed several remarkable insights into the potential of Tracer, and into the use of AlexNet for MNIST classification.

Figure: Figure 11: Global explainability with Tracer– Generalization of causal mechanisms across samples. The Coverage column indicates the percentage of analyzed samples that can be explained by distinct causal mechanisms.
Refer to caption: x11.png

Specifically, as shown in Figure 11:

- 1.
About 85% of the samples could be concisely explained with a causal mechanism entailing merely 2 intermediate causal nodes. This level of generalization showcases the simplicity of the model’s decision-making processes.
- 2.
With just one additional causal node, the causal mechanism explains  99% of the classifications, bringing the total to 3 intermediate causal nodes.
- 3.
To attain a full coverage, explaining 100% of the classifications, the complexity increases only marginally, requiring 4 intermediate causal nodes.

Encouraged by these insights into the causal dynamics of AlexNet’s decisions on the MNIST dataset, we venture to create compressed representations of the original model. The objective is twofold: preserving the original model’s accuracy while substantially reducing its computational complexity. Leveraging the knowledge distilled from Tracer, we craft the corresponding compressed models and train them on the identical training set as the original model (compressed models C1, C2, and C3, respectively corresponding to initial coverages of C1: 84.6%, C2: 98.8%, and C3: 100%). The results, presented in Table 1, show that the most compressed model achieves a staggering 99.42% reduction in model size, while only sacrificing a negligible 0.16% in accuracy, making it significantly more lightweight and computationally efficient.

By decoding the fundamental causal interactions within neural networks, this experiment shows that Tracer’s capacity to provide global explanations and insights can also inspire practical applications such as model compression, without compromising the integrity of the predictions. Furthermore, it is worth noting that the compressed models derived through our approach remain fully compatible with existing and well-established compression methods such as quantization and pruning, further extending their efficiency and applicability across diverse deployment scenarios.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see the weaknesses...
soundness: 2
strengths: - aims to  estimate the causal mechanisms underpinning DNN decisions

- aims to provide explanations for correct and misclassified samples
summary: This paper (TRACER) aims to analyze the causal dynamics of deep neural network (DNN) decisions without altering their architecture. By intervening on input features, TRACER tries to identify feature importance and constructs a causal map.
weaknesses: -  The paper is poorly structured and difficult to follow, with several instances of sloppiness. For instance, in lines 157-160, it says the first level of PCH is Association; However, it is typically Abduction, which involves inferring exogenous noises.


>Association: We extract dependency structures from the DNN activations and outputs $P(Y^{(i)}| X)$
where $X$ and $Y^{(i)}$ represent the input and the $i$-th layer’s output variables, respectively;

Is $Y^{(i)}$ exogenous noise?  **Please clarify their interpretation of the Association level and how it relates to abduction**

- A clear description of causal variables is missing. 
  >line (204-213) : Assuming b to be causally independent (e.g., binary mask), all input
features, before and after interventions, can be considered exogenous variables in the causal map.

How? **Please justify.**

- Many definitions and theorems lack adequate motivation. For instance, the reasoning behind the Average Causal Effect presented in that specific form is unclear. **Please provide more context or motivation for key definitions and theorems, particularly the Average Causal Effect.**

- >*Upon obtaining the similarity measures, we establish causality by grouping layers based on their CKA values* line (246-251)...causal explanation depends on the grouping based on predetermined threshold $\epsilon$. 

Different thresholds will give different causal structures or explanations for the DNN. The consistency of the explanations is not discussed.
**How does the choice of threshold affect the stability and reliability of causal explanations?**

- There is no clear explanation for selecting $\epsilon$, which is a critical hyperparameter. **How do you select or tune the 
 parameter? Please  include any empirical studies or theoretical justifications for your choice**

- Figure 1 does not adequately illustrate the methodology or framework.

-  misses relevant literature, such as:
> - *Neural network attributions: A causal perspective*, Aditya Chattopadhyay, Piyushi Manupriya, Anirban Sarkar, and Vineeth N Balasubramanian. 
> - *Towards learning and explaining indirect causal effects in neural networks*,  Abbaavaram Gowtham Reddy, Saketh Bachu, Harsh Nilesh Pathak, Ben Godfrey, Vineeth N. Balasubramanian, V Varshaneya, and Satya Narayanan Kar. 

**A comparison with these works is necessary..**

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - It seems not to show a counterfactual generation for a misclassified ImageNet sample? I want to see the quality of such a bit complicated counterfactual.
    
- It is well known GANs have the mode collapse problem. Does it occur in your experiments? If so, how to deal with it?
soundness: 3
strengths: - The paper is well organized and well written.
    
- The proposed method is technically sound.
    
- The problem is of great importance and of interest to the community
summary: In this paper, the authors propose a novel method, called TRACER, to investigate the causal dynamics inside DNNs by intervening on input features. They further use TRACER to generate counterfactuals to improve explainability.
weaknesses: - As we know that DNNs are strong correlation learners, the learned links/parameters between neurons/layers might contain lots of strong spurious connections, rather than causal connections. In this case, we cannot distinguish them even though intervening on input features and monitoring the changes on intermediate outputs. Thus, I guess that it might be difficult to learn a true causal graph.
    
- Many details in the experimental part are missing, e.g., what interventions are performed in each specific experiment? What specific regularization metric is chosen for the counterfactual analysis? etc.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: * You say that you use a pre-trained AlexNet model for MNIST classification, however AlexNet was an ImageNet model. Did you fine-tune it with a new head to classify MNIST digits? If so, why isn't this described anywhere in the paper?
soundness: 2
strengths: * The paper is well-presented with high-quality figures
* The research direction, of using causal inference for explainability, is an interesting and important one.
summary: The paper proposes TRACER, a method for generating "explanations" of neural network behavior with methods from causal inference. TRACER groups neural network layers into "groups" based on their similarity, and then computes a causal effect score for each input dimension in terms of how that feature affects the intermediate network layers and the final output. This is done by making a series of interventions to the input and observing how that affects downstream layer groups and the network output. The authors claim that the method generates more sensible saliency maps than existing methods, as measured by a "reliability score" that they define. They also claim that their method can be used to compress an MNIST classifier by over 99% without harming accuracy.
weaknesses: * The paper uses a lot of vague and flowery language that makes me worry that it may be substantially LLM-written.
* The authors claim that they can reduce the size of an MNIST classifier by 99.42% without substantially harming model performance. This seems too good to be true, and their method for doing this not very clearly described.

Overall, I am having a hard time assessing the method and claims of the paper, so will give a score of 5 for now with low confidence.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: 1. Section 3.2.1 : Why consider a single unique value to replace a subset of $x$, and why would it be different than say 0?

2. Line 241 : « Flexibility: It accommodates various kernel functions, such as linear or Gaussian, enabling flexibility based on specific requirements of the analysis ». Please explain in which situation it would be preferable to have a Gaussian kernel instead of a linear one. Why this current choice of kernel?

3. Line 287 : Why consider the many neurons of a layer as realizations of a random variable? To which probability are associated?

4. Line 292 : How does $P (g′i(x) | do(X = x′))$ and $P (g′i(x) | do(X = x))$ are computed in practice?

5. Line 408 : How does P is found in practice and what does P actually outputs? A mask?

6. Table 1 : How is the simpler model created?
soundness: 2
strengths: The use of the causal paradigm for building an explanability tool ensures it has interesting groundings. 
The related work section is thorough.
A various number of considerably unique experiments are considered.
summary: This paper present method grounded in causal theory to estimate the causal link between the various layers of neural networks. This technique is used for both the explanation of the model and its optimization.
weaknesses: 1. The first major weakness concerns the writing of the paper.

1.1 I see the « proofs » in section A of the appendices as explanations as to why the propositions and theorems presented in the main paper make sense and are coherent, and not formal nor rigorous proofs (especially A.1).

1.2 In section 4.1, no explanation is given about what P is in practice; the proportion p that is used. The layer grouping is explained in Section 3, but no details regarding the inputs themselves are provided. How are the most important features found? Many parts of the approach and the experiments lack clear explanations of the various steps, leading to many unanswered questions (see the Questions section) and questions found in the present section. Figure 1 does not give much information about the presented approach. How are the counterfactuals generated in practice, and how are they used? How is the feature contribution made?

1.3 There are many inconsistencies or errors in the notation and the writing itself, see ** Typos and such**.

2. « Causality » is central in the article, yet I see no causality in the proposed approach.

2.1 For example, let’s consider the way the various layers in the network are compared via Centered Kernel Alignments. This « distance measure », or similarity measure, does not tells us anything about causality, simply about resemblance. 

2.2 The relationship is considered or examined between the various components of a single level (e.g. feature level), which truly undermine the scope of the analysis.

3.1 The approach is based on the fact that the considered networks are straightforward in their nature; the output of a layer is considered to be the input of the next layer, but what if more sophisticated architecture is considered, such as U-nets? How would the approach work in the case of recurrent networks? Or networks with parallel components, such as attention heads?

3.2  I am conceptually unsure of the approach, in that it only considers huge perturbations made by a single layer. What about if the whole network is a composition of many small perturbations, such as in Variational autoencoders?

3.3 It is said that if a group of layers is similar, they can be regrouped in a single transformation $g_{i,j} = fj ◦ fj−1 ◦ . . . ◦ fi = fi$, but how seeing this part of the network as a single transformation helps to reduce the number of parameters? Simply replacing $ fj ◦ fj−1 ◦ . . . ◦ fi$ by $fi$ simply cannot be done when the dimensions of the various layers aren’t the same.

4. The last concern is about the contributions of the paper. Line 532 : « Through our foundational principles and findings, we have ascertained that by producing intuitive, human-interpretable explanations, TRACER offers outstanding transparency to neural networks [...] » The paper proposes a way to compute how big of a transformation is applied at each step of the network, and to generate counterfactual, so I would not say so.

4.1 Concerning the originality of the approach: The use of counterfactual is not a contribution, for the same procedure could be used with a different model



**Typos and such**

-Line 89 : rior → Prior

-Line 96 : explainabibilty → explainability

-Line 161 : What does do() mean?
-Line 172 : Based on the notation only, it seems like it is the masked input that is in $\{0,1\}^d$, where, I guess, it is implied that it is the mask that is in $\{0,1\}^d$.

-Line 175 : Please define the « |= » operator.

-Definition 2 : In both C2 and C3, the notation is wrong, for you consider applying masks of dimension different than d to set of examples of dimension d. For example, in C3, instead of « for all M’ $\subset$ M », you probably meant « For all M’ such that |M’|₁ < |M|₁ : … ».

-Line 231 : Please provide citations for «  … prevalent approach for quantifying the
similarity between high-dimensional embeddings. »

-Line 231 : Notations are introduced for talking about the inner components of neural networks at too many various places in the article; there should be a section where this notation is properly introduced. Please state explicitly the dimensions of $f_i$ and $f_j$. It is not obvious only looking at the notation whether $K_i$ is a matrix of size $nxn$ or else.

-Line 252 : The composition increases the layer number at certain places (line 252, 259) but decreases at other places (line 259, 266). Please be coherent with the mathematical notation.

-Line 197 : The usage of « nodes » in « causal nodes » refers to the concept of layer in this work, but « node » typically refer to a single neuron of a layer. It wold be less ubiquitous to use another term for talking about « causal nodes ».

-Line 359 : when citing MNIST, please consider the following indication: http://citebay.com/how-to-cite/mnist/. 

-Line 378 : The hyperparameters that were used should be found in the appendices.

-Line 408 : This letter P (with this exact calligraphy) is already used for Probability.

-Table 1 : C1, C2 and C3 already refers to concept in Definition 2. Please use distinct notation.

-Line 512 : « In this study, we focused our evaluations of TRACER on white-box neural network. » I consider neither AlexNet nor ResNet-50 to be white-box models.

Meta-review: This paper analyzes neural networks through causal dynamics for improved explainability by making interventions, defining similarity metrics between layers, and also counterfactuals through a GAN. Two of the reviews had low confidence and the final review was short, but positive.  The confident reviewer raised some issues that were partially addressed by the authors in the rebuttal about counterfactual examples. These were added to an appendix.

Given this context, I took a close read of the paper myself and found several concerns:

1) The paper is hard to follow mostly because many definitions and theorems lack adequate motivation (agreeing with Reviewer w9Xg)
2) For input level attribution, there's a pretty big literature on explanation by removing that was not acknowledged and also a literature on how to evaluate them
* Explanation by Removing
   - Explaining by Removing: A Unified Framework for Model Explanation by Covert et al (https://arxiv.org/abs/2011.14878)
   - Have We Learned to Explain?: How Interpretability Methods Can Learn to Encode Predictions in their Interpretations (https://arxiv.org/abs/2103.01890)
 * Evaluations for input level attribution
   - A Benchmark for Interpretability Methods in Deep Neural Networks (https://arxiv.org/abs/1806.10758)
   - Explanations that reveal all through the definition of encoding (https://arxiv.org/abs/2411.02664)

3) Counterfactual were discussed using a GAN, but they weren't evaluated. This was fixed, but placed in an appendix. It also wasn't obvious why something like that was needed because the structural causal equation (the model) is known, moreover, the outputs do not have any extra randomness that could be conditioned on for some type of counterfactual.

4) The writing of the paper could be improved. The reviewers liked the quality of the figures, but thought the text was often vague and hard to follow (from my own read, I agree with this assessment).

5) Too many pointers to the appendix (which looks like it has some interesting results)

The positive part of the paper was on the coalescing of layers using the CKA. Overall, I think there's merit in the work, but needs a good amount of refinement before it's ready for publication.

Decision: Reject