Enhancing Model Interpretability with Local Attribution over Global Exploration

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 19:16:35

Abstract: In the field of artificial intelligence, AI models are frequently described as ‘black boxes’ due to the obscurity of their internal mechanisms. It has ignited research interest on model interpretability, especially in attribution methods that offers precise explanations of model decisions. Current attribution algorithms typically evaluate the importance of each parameter by exploring the sample space. A large number of intermediate states are introduced during the exploration process, which may reach the model’s Out-of-Distribution (OOD) space. Such intermediate states will impact the attribution results, making it challenging to grasp the relative importance of features. In this paper, we firstly define the local space and its relevant properties, and we propose the Local Attribution (LA) algorithm that leverages these properties. The LA algorithm comprises both targeted and untargeted exploration phases, which are designed to effectively generate intermediate states for attribution that thoroughly encompass the local space. Compared to the state-of-the-art attribution methods, our approach achieves an average improvement of 38.21% in attribution effectiveness. Extensive ablation studies within our experiments also validate the significance of each component in our algorithm. Our code is available at: https://anonymous.4open.science/r/LA-2024

Main paper: 
 introduction: 
 Recent years have witnessed the emergence of deep learning, which has significantly advanced the development of artificial intelligence (AI). It has enabled computers to learn from extensive data and achieve remarkable performance in areas such as image recognition and natural language processing [9, 11] , contributing to almost every aspect of our daily life. For instance, in healthcare, deep learning aids doctors in diseases diagnosis and treatments planning [4] . In transportation, it powers autonomous vehicles that navigate cities and highways safely and efficiently [16] . It also enhance customer service by answering inquiries and solving issues [1] . Moreover, AI is becoming pivotal in industries such as finance and manufacturing, where it optimizes operations and boosts efficiency [3, 13, 29] . Deapite AI's increasing practice, its models are often regarded as 'black box', reflecting the transparency and trust concerns in understanding how these models make decisions. It leads to several significant challenges and potential issues. First, it undermines users' trust in AI systems. In critical sectors like healthcare and finance, a transparent decision-making process is essential for users to trust the recommendations [12, 14] . A lack of trust will greatly reduce the practical value of even the state-of-the-art technology [33] . Secondly, it complicates the identification and mitigation of errors or biases. For example, addressing gender or racial bias in AI assisted hiring is challenging without insights into the decision-making criteria [28] . Furthermore, the 'black box' nature of AI poses challenges for legal and ethical responsibility [6] . In cases where AI systems cause harm or disputes, pinpointing responsibility is difficult if the principles behind the behavior cannot be explained. Lastly, this opacity can also hinder the regulation and public oversight of AI technologies, leading to technological developments that deviate from societal ethics and values.To address these challenges, Explainable AI (XAI) becomes a trending topic for research, aiming to increase the transparency and interpretability of artificial intelligence decision-making processes. LIME is one of the earliest methods which approximates the behavior of complex models around given inputs [19] . However, it fails to provide comprehensive and precise insights, and can sometimes produce misleading explanations due to the reliance on simplified assumptions about model behavior. Later approaches, such as Grad-cam [20] and Score-cam [30] , which use gradients information, are limited by model structure and do not produce fine-grained (input-dimension consistent) results. Introduced in Integrated Gradients (IG) [25] with the axiomatic properties, Sensitivity and Implementation Invariance, attribution method marks a significant advancement in XAI. As a more robust approach, it provides high-resolution, fine-grained explainability and are not limited to model structure, allowing for precise attribution of model results based on rigorous axiomatic principles.Current attribution methods generally calculate the importance of each dimension by accumulating gradients over intermediate states [17, 25, 31, 35] , which ensures compliance with axioms of Sensitivity and Implementation invariance [25] . However, they often fail to address the plausibility of these intermediate states.Considering the extensive input space neural networks encounter, it is impractical to accurately assess every potential state. In this work, we firstly investigate this phenomenon and define a space that neural networks are responsible for as the In-Distribution (ID) space, whereas the space they are not responsible for is termed Out-Of-Distribution (OOD) space. We observe that, most intermediate states utilized in current attribution algorithms often fall within the defined OOD space, which has led to attribution errors.To further investigate what types of intermediate states will contribute to attribution results, we discuss two research questions:• RQ1: Does the significance of the attribution results still hold if there is a critical deviation in the features? • RQ2: Can we still accurately assess the importance of remaining features when key features are excluded? In addressing these research questions, we introduce the concept of local spaces, where attribution approves to be valuable and precise. Inspired by MFABA [35] and AGI [17] , we explore a combination of targeted and untargeted adversarial attacks in local spaces. We provide a thorough analysis from an optimization perspective on how these attacks can be integrated and their contribution to attribution exploration. Building on this analysis, we propose the Local Attribution (LA) method, for which we provide the detailed mathematical derivations and proofs demonstrating its compliance with attribution axioms. Our contributions are outlined as follows:• We identify that current attribution methods often assign intermediate states to spaces not contributing to attribution results. To address this issue, we introduce two research questions and the concept of attribution local spaces.  
related work: 
 In this section, we explore different methods used for explaining Deep Neural Networks (DNNs) and provide a critical discussion of these approaches, which are grouped by three types: local approximation methods, gradient-based attribution methods, and adversarial-sample-based attribution methods.Local approximation methods seek to understand the behavior of the original model near specific inputs by constructing an approximate, more interpretable model. A well-known method is LIME [19] , which approximates local explainability by using multiple interpretable structures near the sample. However, LIME's local explainability requires assumptions, which may not always be accurate. Moreover, LIME can be time-consuming for individual samples. While rudimentary for neural network applications, LIME has been foundational in advancing local explainability methods. Following developments include Layer-wise Relevance Propagation [2] and DeepLIFT [21] . DeepLIFT quantifies the importance of features by comparing the differences between input features and predefined reference points. Although DeepLIFT performs well in local explanation of nonlinear models, its high sensitivity to the choice of reference points can lead to inconsistencies in attribution results. Additionally, DeepLIFT does not satisfy the Implementation invariance axiom proposed in IG [25] , leading to potential biases.Training neural networks inherently utilize gradients, which has inspired the gradient-based methods that use model gradient information to explain decisions. Early methods like Saliency Map (SM) [22] identify the most important features for model predictions by calculating the gradients of input features relative to the model output. However, SM is prone to gradient saturation, resulting in unstable attribution results, and it does not meet the Sensitivity axiom mentioned in subsequent IG [25] , meaning it can yield a zero attribution even if the model output changes. Later methods such as Grad-cam [20] and Score-cam [30] use intermediate layer gradient information but cannot provide high-resolution fine-grained explainability results, and thus cannot be considered true attribution methods (refer to Section 3.1 problem definition).The IG method addresses the insufficient gradient issue of SM by integrating gradients along the path from baseline to input, introducing the axioms of sensitivity and implementation invariance, which are fundamental guarantees for attribution algorithms.Our design also provides proofs of compliance with these axioms. However, IG's main challenge lies in its high computational cost, requiring multiple forward and backward passes. To improve computational efficiency, Fast IG (FIG) [8] optimizes the IG method by improving numerical integration techniques to speed up the attribution process. Although this optimization enhances efficiency, the approximate nature of numerical integration might introduce new errors, affecting the accuracy of attribution results. Additionally, Expected Gradients (EG) [5] provides a more stable and consistent assessment of feature importance by considering multiple baselines and averaging their gradients, improving the IG method. However, a limitation of the EG method is its assumption that contributions from different baselines are equal, which may not be suitable for all types of data and model structures, thus affecting the generalizability of its explanations. SmoothGrad (SG) [24] improves the smoothness and stability of attribution results by adding random noise to inputs, reducing the noise in single gradient calculations. Despite these improvements, the addition of noise may also mask understanding of subtle features important to the model's decision-making process, thus reducing the accuracy of explanations. Guided IG (GIG) [10] combines the principles of IG and guided backpropagation by selectively backpropagating gradients to enhance interpretability. However, GIG's limitation is that it may overemphasize features directly related to specific categories while ignoring indirect features that are equally important to model decisions, somewhat limiting its ability to provide comprehensive explanations. This school of attribution algorithms based on IG is limited by the choice of baseline in the attribution path, introducing a significant amount of irrelevant noise.Adversarial-sample-based attribution methods provide deep explanations of models by generating adversarial samples and exploring model decision boundaries, meaning the attribution process no longer relies on manually specified baseline points. Adversarial Gradient Integration (AGI) [17] is a representative work that uses adversarial samples to explore decision boundaries and improves attribution performance through nonlinear path integral gradients. While AGI offers an innovative method of explanation, its performance highly depends on the quality of the adversarial samples, which may not be stable in some cases. Boundary-based Integrated Gradients (BIG) [31] introduces a boundary search mechanism to optimize the baseline selection, thereby obtaining more accurate feature attributions. However, BIG relies on a linear integration path, which may limit its ability to capture the nonlinearity and complexity in model decision. AttEXplore [34] improves feature attribution by combining adversarial attacks with model parameter exploration, emphasizing the ability to transition between different decision boundaries. Although At-tEXplore shows foresight in enhancing the generalization ability of model explanations, its high computational complexity may limit its application on large-scale models and datasets. MFABA (More Faithful and Accelerated Boundary-based Attribution) [35] enhances the accuracy and computational efficiency of explanations through second-order Taylor expansion and decision boundary exploration, particularly suited for complex model explanations. Nonetheless, its reliance on higher-order derivatives may increase the computational burden, especially when dealing with large deep learning models. This class of adversarial-sample-based attribution methods introduces a large number of intermediate states from the OOD space during the adversarial attack process, as shown in Figure . 1, affecting the accuracy of attribution (discussed in Section 3.2). 
method: 
 In this section, we define the attribution task, the local properties of attribution, and the algorithmic procedure of the LA (Local Attribution) method. Ensuring local properties is key to the rationality of attribution results, and within these constraints, it is still possible to achieve results that satisfy the remaining axioms of attribution. We will describe these in detail below and provide rigorous mathematical derivations. Additionally, the LA algorithm consists of two parts: targeted and untargeted attribution, which can be combined under the premise of maintaining local properties.Given neural network parameters 𝑤 ∈ R 𝑛 and a sample 𝑥 ∈ R 𝑛 to be attributed, we aim to use an attribution method to obtain attribution results 𝐴(𝑥) ∈ R 𝑛 , where 𝐴 𝑖 (𝑥) represents the importance of the 𝑖-th feature dimension. The greater the attribution result, the more important the dimension is for the model's decision. We use 𝑓 (𝑥) ∈ R 𝑐 to represent the model output, where 𝑐 denotes the number of classes. of IG [25] , gradients of intermediate variables accumulated from the sample to the baseline are considered. Suppose there is only one intermediate state, and the baseline is 𝑏 = [0, 0, 0, 0]. Thus, the intermediate variable 𝑥 ′ lies between 𝑥 and 𝑏 at 𝑥 ′ = [3, 4, 3, 5] . At this moment, we need to compute the gradient information, but is this gradient information truly valuable? Given the vast input space neural networks face-a space so large it's impossible to traverse fully-it means that models generally need only be responsible for In-Distribution (ID) samples, and most of the space filled with Out-Of-Distribution (OOD) samples is meaningless. Similarly, in the OOD space, gradients will lack instructive significance because it is virtually impossible for the model to encounter 𝑥 ′ = [3, 4, 3, 5] in tasks, and 𝑥 ′ at this point cannot sustain the model's decision, placing 𝑥 ′ within the OOD space. This means that assessing sample feature importance in scenarios where features undergo significant deviations and cannot maintain model decisions introduces too much extraneous information from spaces the model is not responsible for. Other methods like MFABA [35] and AGI [17] use adversarial attacks to obtain intermediate states 𝑥 ′ , but when adversarial attacks are sufficiently frequent, 𝑥 ′ will ultimately fall into OOD space. As shown in Figure . 2, after two or more adversarial attacks, the attack samples are insufficient to maintain the model's decisions, leading to serious attribution biases.We need further definition on what samples are considered to be in the ID space. In Multiplicative Smoothing (MuS) [32] , the explainable method stable can be seen as when enough important features are satisfied, adding additional features will not affect the original model's decision. That is, the model has obtained enough important features to maintain the current decision, meaning these crucial features are key to keeping the target within the ID space. This also leads to RQ2: When important features are disregarded, can the remaining features still be correctly assessed for importance?As shown in Figure . 3, since assessing feature importance is meaningless without important features, we must ensure that these features do not change during the assessment. Unfortunately, we cannot determine which features are important during the assessment phase (importance can only be confirmed after, not during the assessment, as they influence each other). Here, we provide the definition of when an intermediate state is considered to be in the Out-of-Distribution (OOD) space: An intermediate state is in the OOD space if it cannot maintain the same model decision as the original state. The only remaining option is to ensure that all features are assessed within a local space where no significant deviations occur.Next we give the definition of attribution local space: Theorem 3.1 (Local Space). Given a sample 𝑥, the 𝜖-Local Space of 𝑥, denoted as 𝐵 𝜖 (𝑥), is defined as:𝐵 𝜖 (𝑥) = { x | | x𝑖 -𝑥 𝑖 | ≤ 𝜖 𝑖 }(1) where 𝜖 ∈ R 𝑛 and 𝜖 𝑖 = 𝑥 𝑖 𝑠 , with 𝑠 being a hyperparameter that controls the size of the local space (Spatial Range).We assume that the importance assessment within the local space is valid. Notably, each feature's constraint 𝜖 𝑖 on the local space varies; larger feature values usually imply greater activation but also indicate lower sensitivity to changes, warranting a larger local space. We use the mapping 𝜖 𝑖 = 𝑥 𝑖 𝑠 to make the constraints linearly related to the size of features. Our experiments will analyze the difference between using constant 𝜖 𝑖 and linearly related 𝜖 𝑖 .Next, we introduce our Local Attribution algorithm and present our core theorem: Theorem 3.2 (LA). Given parameters 𝑤 ∈ R 𝑛 and corresponding sample 𝑥, the local attribution for dimension 𝑖 can be expressed asEQUATIONwhere 𝐵 𝜖 2 (𝑥) denotes the 𝜖 2 -Local Space of sample 𝑥, and 𝑢 represents the exploration function.We define the importance of each dimension in the sample point based on the expected change in the loss function value caused within the local space. Intuitively, the dimensions that cause greater changes in the loss function within the effective space (local space) are more sensitive. Next, we will present the derivation proof from the expected change in the loss function to Eq. 2.Proof. Consider the expansion of the loss function 𝐿: 𝐿( x; 𝑦, 𝑤) = 𝐿(𝑥; 𝑦, 𝑤) + ( x -𝑥) • 𝜕𝐿(𝑥; 𝑦, 𝑤) 𝜕𝑥EQUATIONwhere O represents higher order infinitesimals. From the property of expectation and the symmetry of the local space, we have:E x∼𝐵 𝜖 2 (𝑥 ) [𝐿 ( x𝑖 ; 𝑦, 𝑤) -𝐿(𝑥; 𝑦, 𝑤)] = E x∼𝐵 𝜖 2 (𝑥 ) ( x -𝑥) 𝜕𝐿(𝑥; 𝑦, 𝑤) 𝜕𝑥 = E x∼𝐵 𝜖 2 (𝑥 ) [( x -𝑥)] • 𝜕𝐿(𝑥; 𝑦, 𝑤) 𝜕𝑥 = 0 (4) because 𝜕𝐿 (𝑥;𝑦,𝑤 ) 𝜕𝑥 is independent of the choice of 𝐵 𝜖 2 (𝑥).Firstly, we perform a first-order Taylor expansion of the loss function calculated for sample 𝑥 to get Eq. 3, which is substituted into Eq. 4 to calculate the expected transformation of the loss function within the local space. (Taking one dimension of 𝑥, replacing 𝑥 with 𝑥 𝑖 in the formula, the derivation process remains unchanged, i.e., E[𝐿 ( x𝑖 ; 𝑦, 𝑤) -𝐿(𝑥 𝑖 ; 𝑦, 𝑤)] = 0). We see that under a first-order approximation, it is not possible to evaluate each feature through a single local space sampling. Introducing higher-order approximations can mitigate this issue, but due to the presence of sampling, it is impractical to introduce finite differences [15] to approximate the Hessian matrix in intermediate computations, which also makes introducing higher-order approximations computationally infeasible.EQUATIONTo enable practical computation, we introduce an exploration function 𝑢, which ensures that the transformed samples remain within the 𝜖-Local Space (proof in Appendix C). Inspired by MFABA [35] and AGI [17] , the function 𝑢 can utilize both untargeted (Eq. 6) and targeted (Eq. 7) adversarial attacks, where the choice of 𝑦 𝑡 is from categories other than the most probable, selected in descending order of confidence. Our experiments will involve an ablation study on the number of categories selected.EQUATION)Red area is important to the cat category.Blue area is less important than red area.The relative importance of the cat category cannot be obtained from these two areas.(a) (b)Figure 3 : After removing the important features, the relative importance of the remaining features is not as significant. As shown in (a), the features in the red area are notably more important for the category of cats compared to those in the blue area. However, as depicted in (b), after the cat features have been removed, it becomes challenging to assess the importance of the remaining features.EQUATION• sign 𝜕𝐿( x; 𝑦 𝑡 , 𝑤) 𝜕 x (7)The direct output of our neural network is defined as 𝑧 = 𝑓 (𝑥) ∈ R 𝑐 , and after passing through a softmax function, 𝑧 becomes a probability distribution 𝑝 = softmax(𝑧) ∈ R 𝑐 with 𝑝 𝑖 ∈ (0, 1). Observing the gradient 𝜕𝑧 𝑖 𝜕𝑥 ∈ R 𝑛 , updating 𝑥 along the direction of 𝜕𝑧 𝑖 𝜕𝑥 increases 𝑧 𝑖 (proof refers to Eq. 3, the Taylor expansion). We examine the gradient information of 𝑧 during the computation of cross-entropy loss as shown in Eq. 8.𝜕𝑧𝑖 = 𝑝 𝑖 -1 if 𝑖 = class of 𝑦 𝑝 𝑖 otherwise (8)Using the chain rule for gradients, we find 𝜕𝑥 , 𝑗 ≠ 𝑖, will be low, since the probability values 𝑝 𝑗 are lower than for the original category. Thus, relying solely on untargeted adversarial attacks to explore the local space might neglect the information from categories other than the original. This necessitates the introduction of targeted attacks for other categories. As shown in Figure . 1, considering that the sign of targeted adversarial attacks is opposite to that of untargeted attacks, analyzing with 1 -𝑝 𝑗 becomes relevant, where 1 -𝑝 𝑗 is large when 𝑝 𝑗 is small, allowing the preservation of gradient information𝜕𝑧 𝑗𝜕𝑥 . From a gradient perspective, it is crucial to incorporate both forms of adversarial attack in the local space, and since 𝜖 𝑖 remains the same under both attack conditions, their effects can be combined additively.Finally, for the sampling process from 𝐵(𝑥) to obtain x, we can approximate it iteratively, using the gradient calculated from the previous sample step to perform a one-step attack from the original sample. If the sign of the gradient in the same dimension changes within a local space, it indicates that the dimension is sensitive and requires further exploration. If the dimension remains unchanged, it implies that maintaining the current dimension does not require alteration, thus reducing the scope of space that random sampling needs to explore. The obtained x still resides within the local space 𝐵(𝑥), and we provide rigorous proof in Appendix D and pseudocode in Appendix E. 
experiments: 
 In this section, we provide a detailed description of the series of experiments conducted using the Local Attribution (LA) algorithm, including the choice of datasets, models, baseline methods, evaluation metrics, and experimental analysis. Our experiments randomly selected 1000 images from the Ima-geNet dataset, following the precedent set by existing methods such as AGI [17] , MFABA [35] , and AttEXplore [34] . Furthermore, we tested the LA algorithm using four different convolutional neural network architectures to assess its effectiveness and generality, namely Inception-v3 [26] , ResNet-50 [7] , VGG16 [23] , and MaxViT-T [27] .To comprehensively evaluate the performance of the Local Attribution (LA) algorithm and ensure the fairness of our assessments, we have compared LA against eleven baseline methods. These baselines cover a wide range of XAI methods, including AGI [17] , AttEXplore [34] , BIG [31] , DeepLIFT [21] , EG [5] , FIG [8] , GIG [10] , IG [25] , MFABA [35] , SG [24] , and SM [22] . These methods represent various technical approaches in the field of model explainability, providing a broad reference standard for evaluation.We employed two traditional metrics, Insertion Score and Deletion Score, to assess the explanatory power of each explainability method. These metrics evaluate the model's dependence on different parts of the input data by analyzing changes in model performance [17] . The Insertion Score evaluates changes in model performance by progressively converting pixels from a baseline state (usually a state containing no meaningful information, such as all-black or all-white images) to the pixels of the original image. This conversion involves incorporating a certain number of the most important pixels, as determined by the explainability method, from the baseline state to their values in the original image. This process is repeated, re-evaluating the model's performance each time until all pixels have been converted from the baseline state to their corresponding pixels in the original image. The Insertion Score is typically quantified by the degree of improvement in model performance, represented by the following formula:InsertionScore = 1 𝑁 𝑁 ∑︁ 𝑖=1 (𝑃 𝑖 -𝑃 0 )where 𝑁 is the total number of steps, 𝑃 𝑖 is the model performance score after step 𝑖, and 𝑃 0 is the initial state performance score (when no significant pixels are inserted).The Deletion Score is calculated by progressively removing the most important pixels from the original image and observing changes in model performance. Each removal involves replacing a certain number of the most important pixels (again determined by the explainability map) with pixels of a baseline state. This process is continuously repeated, re-evaluating the model's performance each time, until all pixels deemed important have been removed. The Deletion Score is quantified by the degree of reduction in model performance, represented by the following formula:DeletionScore = 1 𝑁 𝑁 ∑︁ 𝑖=1 (𝑃 full -𝑃 𝑖 )where 𝑃 full is the model performance score under the full image, and 𝑃 𝑖 is the model performance score after step 𝑖 when important pixels have been removed.We identified implementation biases in the Insertion Score and Deletion Score in the open-source codes of RISE [18] , MFABA [35] , BIG [31] , and AGI [17] . Previous works performed importance ranking by sorting each channel of the image separately, but the actual evaluation process should treat each input dimension equivalently. Therefore, we corrected this bias in this paper. To ensure consistency in experimental results, we also replicated the experiments In this series of experiments, we set the number of sampling to 30 for the MaxViT-T model and 20 for the other models. The spatial range 𝑠 was consistently set to 20 across all experiments.As shown in Tab. 1, our LA method achieved significant improvements. Compared to other methods, the average increase in Insertion across the four models was 0.31758, and the average reduction in Deletion was 0.028883. Specifically, the average improvements in Insertion for the Inception-v3, ResNet-50, VGG16, and MaxViT-T models were 0.30948, 0.36262, 0.28626, and 0.31197 respectively; while the reductions in Deletion were 0.019774, 0.017429, 0.025277, and 0.053052 respectively. Compared to the latest attribution methods like AGI, MFABA, and AttEXplore, LA showed clear advancements. Notably, LA not only improved in terms of Insertion but also reduced Deletion, indicating a comprehensive enhancement in explainability performance compared to these methods. While some methods slightly outperformed LA in terms of Deletion, their Insertion scores were substantially lower than LA. As previously mentioned, the significance of Insertion outweighs that of Deletion, show the attribution results of our LA method versus other methods on the Inception-v3 and MaxViT-T models. It is evident that our LA method more accurately and concisely captures the key features in images, while the outputs from other methods appear more dispersed and blurred.4.6.1 Impact of Constant vs. Linear Space Constraints on Effectiveness. This section discusses the impact of Constant and Linear space constraints on the effectiveness of LA. We fixed the number of samples and the spatial range 𝑠 at 20. As shown in Figure . 6, across different models, the attribution performance under Linear space constraints was comprehensively better than under Constant constraints, with significantly higher Insertion Scores and lower Deletion Scores under Linear constraints. 4.6.2 Impact of Attack Type on Effectiveness. We discuss the impact of targeted and untargeted attacks on the effectiveness of LA. The number of samples and the spatial range 𝑠 were kept constant at 20. As depicted in Figure . 7, across different models, the Insertion from Untargeted Attacks was higher than from Targeted Attacks. However, the Deletion Scores were relatively similar, showing little variation. 4.6.4 Impact of Spatial Range 𝑠 on Effectiveness. In this section, we explore the impact of the spatial range 𝑠 on the effectiveness of LA. The number of samples was fixed at 20. As illustrated in Figure. 9, with an increase in 𝑠, the Insertion Score initially increased and then decreased, peaking when 𝑠 was at 10. Conversely, the Deletion Score increased with larger 𝑠 values, but the rate of increase gradually weakened. 
conclusion: 
 In this paper, we identifies the challenge of ineffective intermediate states in current attribution algorithms, which has significantly impacted the attribution results. To better investigate this issue, we introduces the concept of Local Space to ensure the validity of intermediate states during the attribution process. With these findings, we propose the LA algorithm, which can comprehensively explore the Local Space using both targeted and untargeted adversarial attacks, thereby achieving state-of-the-art attribution performance in comparison with other methods. We provide rigorous mathematical derivations and ablation study to validate the significance of each component in our algorithm. We anticipate this work will facilitate the attribution method in XAI research.  
 
 Appendix:  


Decision: This paper contends that existing attribution algorithms suffer from a notable limitation: they generate numerous intermediate states that fall into the out-of-distribution (OOD) space, thereby compromising attribution accuracy. To tackle this challenge, the authors propose the Local Attribution (LA) method, which uses targeted and untargeted adversarial attacks to explore the local input space. The paper includes comprehensive theoretical proofs and derivations for the proposed method, and empirically demonstrates its state-of-the-art performance.

All reviewers found the paper interesting with clear motivation and presentation. The experimental results are sound. I would encourage the authors to incorporate the comments and suggestions in the final version.

Reviews: review: paper_summary: paper_summary: This paper investigates the problem of attributing a model's output w.r.t. its input features, one of the core problems in explainable AI. The paper identifies an important issue overlooked by previous studies in this area, namely, the intermediate states used in attribution may be out-of-distribution (OOD) and thus lead to unreliable attribution results. The paper then proposes to search for intermediate states in a _local_ region of inputs (i.e., local space). The notion of local space is proposed by this work, while the search function (i.e., exploration function) is instantiated to be targeted or untargeted adversarial attacks, which is also used in prior work such as MFABA and AGI.
strength_weakness: strength_weakness: - The problem identified by the paper (OOD intermediate states) is intuitive and well-motivated.

- The proposed notion of local space and the corresponding algorithm is simple and seems easy to implement.

- Experimental results are generally good with sufficient ablations.
limitations: limitations: - I think the proof in Appendix A about the _sensitivity_ axiom is incorrect: according to the original definition of sensitivity in IG [1], sensitivity refers to "if for every input and baseline that differ in one feature but have different predictions then the differing feature should be given a non-zero attribution" [1, Section2.1]. The authors of [1] also give a counterexample suggesting why vanilla gradients violate this axiom: the neural network may be "flat" around the input (but not flat around the baseline), leading to zero gradients at the input and hence zero attribution. Since your method also searches over a **local** region of the input, it may also encounter such a "flatness" issue which leads to zero attribution. E.g., when $\epsilon\to 0$, the expected gradient $\mathbb{E}(\frac{\partial L(\tilde{x}; y, w)}{\partial \tilde{x}})$ for $\tilde{x}\sim B_{\epsilon / 2}(x)$ would be essentially the same as the vanilla gradient $\frac{\partial L(x; y, w)}{\partial x}$. So in this scenario, your method would also lead to zero attribution if $\frac{\partial L(x; y, w)}{\partial x}\approx 0$.

- Besides the selectivity axiom, [1] also introduces another axiom of _completeness_, which is not covered in the proof. It would be great if the authors could discuss whether your method satisfies this axiom or whether satisfying this axiom is necessary.

- The authors state that "An intermediate state is in the OOD space if it cannot maintain the same model decision as the original state." (L375-377), which does make sense to me. However, such a criterion is not used in the definition of the local space nor explored in the ablation; instead, the authors resort to manually tuning the "size" of the local region by introducing a hyperparameter. Why not directly introduce the model decision as a criterion in searching for non-OOD intermediate states?

- To be honest, I didn't really get the relation between the RQ2 in the paper and the notion of local space in your method. The paragraph starting from Line 368 to Line 379 seems quite confusing in connecting those two concepts.

- The writing of the paper can be improved. E.g., I think the first several paragraphs in the introduction are kind of lengthy and contain some contents that are not directly related to the paper (e.g., the application of AI in various fields). Also the notion of "intermediate states" can be introduced in a way that is easier to understand, e.g., you can include the formula of IG [1] in Section 3.2 to show how intermediate states contribute to attribution before giving the example.


[1] Axiomatic Attribution for Deep Networks.
paper_summary: paper_summary: The proposed Local Attribution (LA) algorithm aims to provide precise explanations by focusing on local space properties rather than exploring the entire sample space, which can lead to intermediate states in the out-of-distribution (OOD) space. The LA algorithm incorporates both targeted and untargeted adversarial attacks to explore local spaces effectively. Experimental results demonstrate a significant improvement in attribution effectiveness compared to state-of-the-art methods.
strength_weakness: strength_weakness: 1. The concept of local space for attribution and the dual-phase exploration approach of LA is a novel contribution that enhances the precision and reliability of model interpretability.
2. The use of targeted and untargeted adversarial attacks to comprehensively explore local spaces is technically sound and innovative. The detailed mathematical derivations and proofs of compliance with attribution axioms strengthen the methodological foundation.
3. The paper provides extensive experiments comparing LA with various state-of-the-art methods across multiple models and metrics.
4. The paper is well-structured, clearly explaining the proposed method.
limitations: limitations: The proposed method involves computationally intensive adversarial attacks and multiple phases, which increases the training overhead and complexity. This aspect is not discussed in detail in the paper.
paper_summary: paper_summary: This paper argues that existing attribution algorithms suffer from a significant drawback of generating numerous intermediate states that reside in the out-of-distribution (OOD) space, leading to compromised attribution results. To address this issue, the authors propose a Local Attribution (LA) method, which leverages targeted and untargeted adversarial attacks to explore the local space. The paper provides comprehensive theoretical proofs and derivations for the proposed method and empirically demonstrates its state-of-the-art performance.
strength_weakness: strength_weakness: 1. The paper effectively highlights the limitations of current attribution algorithms and presents theoretical evidence supporting the efficacy of the proposed Local Attribution (LA) method. 

2. The writing is coherent, and the logical flow is well-maintained. Furthermore, the experimental validation substantiates the effectiveness of the proposed approach.

3. The idea introduced in the paper exhibits novelty and is logically sound.
limitations: limitations: 1. The authors utilize Figure 3 to illustrate that the assessment of feature importance becomes meaningless when no significant features are present. However, it would be beneficial to provide theoretical justification for this claim, as it may relate to the distribution of image data.

2. In the ablation experiments, the authors investigate the impact of different conditions, such as Space Constraints and Attack Type, on the effectiveness of the LA method. However, further explanations or insights into the experimental results are lacking.

3. The figures included in the paper appear somewhat simplistic, lacking the expected academic quality. Enhancing the visual representations would enhance the overall presentation. Also, the state-of-the-art results achieved by the LA method might to be emphasized in Table 1 (e.g., by bolding the relevant values).
paper_summary: paper_summary: The focus on local spaces to avoid OOD issues is innovative and addresses a critical problem. The paper is well-executed, with strong experimental results and an interesting approach to improving attribution accuracy. While the paper provides mathematical derivations and proofs of the Local Attribution (LA) method’s compliance with attribution axioms, it could benefit from a more in-depth theoretical analysis. Specifically, exploring the limitations and potential failures of the method in different OOD scenarios would add significant value.
strength_weakness: strength_weakness: 1.	The paper addresses the critical issue of OOD impacts in model interpretability, which is highly relevant in the field of AI. Understanding and mitigating OOD effects is essential for the robustness and reliability of AI models.

2.	The experimental results are impressive, demonstrating a significant improvement in attribution effectiveness compared to state-of-the-art methods. The reported 38.21\% average improvement is notable and suggests that the proposed LA algorithm is highly effective.

3.	The idea of focusing on local spaces for attribution rather than global exploration is intriguing. This approach helps to avoid the pitfalls of OOD spaces, potentially leading to more accurate and meaningful attributions.
limitations: limitations: 1.	The theoretical aspects of the paper are somewhat trivial. While the concept of local spaces and their properties is well-presented, the paper does not provide a deep theoretical analysis of the method's generalization capabilities in OOD scenarios. A more rigorous theoretical exploration could strengthen the paper’s contribution.

2.	The paper does not sufficiently analyze how the proposed Local Attribution method generalizes to various OOD problems. A thorough discussion on the method's limitations and potential generalization issues in different OOD contexts would provide a more comprehensive understanding of its applicability and robustness.