Splitting & Integrating: Out-of-Distribution Detection via Adversarial Gradient Attribution

Authors: Jiayu Zhang, Xinyi Wang, Zhibo Jin, Zhiyu Zhu, Jiahao Huang, Xinyi Zhang, Huaming Chen
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Out-of-Distribution Detection, Adversarial Gradient Attribution, Safety

Abstract:
Out-of-distribution (OOD) detection is essential for enhancing the robustness and security of deep learning models in unknown and dynamic data environments. Gradient-based OOD detection methods, such as GAIA, analyse the explanation pattern representations of in-distribution (ID) and OOD samples by examining the sensitivity of model outputs w.r.t. model inputs, resulting in superior performance compared to traditional OOD detection methods. However, we argue that the non-zero gradient behaviors of OOD samples do not exhibit significant distinguishability, especially when ID samples are perturbed by random noise in high-dimensional spaces, which negatively impacts the accuracy of OOD detection. In this paper, we propose a novel OOD detection method called **S \& I** based on layer **S**plitting and gradient **I**ntegration via Adversarial Gradient Attribution. Specifically, our approach involves splitting the model's intermediate layers and iteratively updating adversarial examples layer-by-layer. We then integrate the attribution gradients from each intermediate layer along the attribution path from adversarial examples to the actual input, yielding true explanation pattern representations for both ID and OOD samples. Experiments demonstrate that our S \& I algorithm achieves state-of-the-art results, with the average FPR95 of 29.05\% (38.61\%) and 37.31\% on the CIFAR100 and ImageNet benchmarks, respectively. Our code is available at: https://anonymous.4open.science/r/S-I-F6F7/.

Decision: reject
Reviewer scores: 3, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730132115055

thread_type:
official_review

<thread_object>
creator: reviewer_wckd

<content>
summary:
Due to the non-zero gradient behaviors of OOD samples, which do not exhibit significant distinguishability, this negatively impacts the accuracy of OOD detection. Considering this issue, the paper proposes a novel OOD detection method called S & I based on layer Splitting and gradient Integration via Adversarial Gradient Attribution. The proposed method incorporates adversarial attacks into attribution to explore the distributional characteristics of ID and OOD samples. Experiments demonstrate that S & I algorithm achieves state-of-the-art results.

strengths:
The proposed method has been evaluated on multiple datasets.  
The paper provides theoretical proof to ensure the rationality of the method.

weaknesses:
1. The experiments conducted in the paper do not effectively support the proposed arguments.  
a) The paper mentions that "we argue that the non-zero gradient behaviors of OOD samples do not exhibit significant distinguishability, especially when ID samples are perturbed by random noise in high-dimensional spaces, which negatively impacts the accuracy of OOD detection." However, the experiments in the paper are just some general evaluations and do not highlight the points argued above. There is no experiment to discuss detecting the non-zero gradient behaviors of OOD samples.  
b) There is a lack of ablation studies to verify the importance of each part.

2. The distributional differences caused by adversarial attacks are completely different from the differences between various datasets. Why can introducing adversarial attacks enhance OOD detection across different datasets?

3. Can the proposed OOD detection be applied to distinguish between adversarial examples and clean examples?

4. Is OOD detection still important in current deep learning? For example, in classification tasks with adversarial examples, one approach to avoid outputting the incorrect labels is to use detectors to refuse adversarial examples and only allow clean examples to be classified. However, these methods are gradually being abandoned because a robust system should be able to output correct labels for any input, not just simply refuse to output labels for adversarial examples. Therefore, under the settings of this paper, I am curious about which specific scenarios the OOD method is applicable or necessary to at present.

5. Due to the presence of adversarial attacks and layer-splitting technology, will this lead to the consumption of a large amount of computational cost in the whole process?

Since I am not completely familiar with this task, I have set the confidence as 2.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730172877688

thread_type:
official_review

<thread_object>
creator: reviewer_rqf8

<content>
summary:
This paper proposes the S&I method to enhance the performance of OOD detection.

strengths:
1 The structure is clear.

weaknesses:
1 The experimental results are suboptimal, with an average improvement of no more than 0.5% (AUROC metric) across different datasets. If the improvement is not significant, it suggests that the problem addressed in this paper may not be highly important. Please provide an experiment that can significantly enhance performance.

2 In Figure 2, the meanings of the x and y axes are not explained; the two histograms on GAIA lack subtitles to distinguish them. Please clarify these details.

3 The first two contributions summarized in the last paragraph of Chapter 1 require an ablation study to demonstrate their effectiveness separately.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730595861459

thread_type:
official_review

<thread_object>
creator: reviewer_8iu4

<content>
summary:
The paper addresses the problem of out-of-distribution (OOD) detection, crucial for enhancing the robustness of deep learning models. The authors propose a novel method called S & I (Splitting & Integrating) that improves on existing gradient-based OOD detection approaches like GAIA by introducing adversarial examples and integrating gradient attribution across split intermediate layers of the neural network. The S & I algorithm smooths gradient fluctuations and identifies true explanation patterns, outperforming state-of-the-art (SOTA) methods on CIFAR100 and ImageNet benchmarks, as shown through extensive experiments.

strengths:
+ The introduction of layer splitting combined with adversarial gradient attribution integration is innovative.
+ The method demonstrates superior performance on CIFAR100 and ImageNet benchmarks, achieving lower FPR95 and higher AUROC compared to baselines.
+ The paper provides a mathematical basis and proofs for the concepts introduced.
+ The authors evaluate their approach against multiple baseline methods, showcasing clear advantages.

weaknesses:
- On CIFAR100, the performance gains over GAIA are not substantial, suggesting limited effectiveness in smaller label space datasets.
- The algorithm's complexity, involving iterative adversarial updates and integration across multiple layers, may hinder scalability or applicability to extremely large models.

questions:
- Have you considered the potential security risks introduced by using adversarial examples as baselines, and how might this affect deployment?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730636618618

thread_type:
official_review

<thread_object>
creator: reviewer_btvy

<content>
summary:
The paper proposes the Splitting and Integration methods for out-of-distribution detection. It is based on the method GAIA  (Chen et al. 2023) and compares the proposed method with GAIA and other algorithms.  Splitting is a simple division of network layers into parts (as indicated in line 118), and integration is based on gradient splitting. The integrated gradient is based on the IG method. The experimental results are comparable to GAIA but do not significantly surpass GAIA's performance in order to be a new state of the art.

strengths:
The paper proposes a modification to GAIA (Chen et al. 2023). GAIA uses image x =  0 as a baseline for Integrated Gradient attribution, whereas the proposed method uses adversarial image $x_{adv}$). The results of the experiments slightly improve to GAIA in few cases.

weaknesses:
The paper experiment results are almost the same as GAIA except for a slight improvement on one dataset (SVHN).
The paper writing in most places is full of authors' assumptions, or at least the sentences give such an impression (see lines 115, 202, 311). These statements rather should be backed by evidence or citations. There are questions about the use of variables and their definition in many equations. (see Question section). Proof  2 appears to be based on the assumption that $x_adv$ of $x_out$ will exhibit overly confidence.

questions:
Define the term "overly confident" or not overly confident. 

Clarify the score OOD score in Algo 1 as $\tau$. This appears for the first time in the paper and is not used anywhere else. 

In Eq. confidence score, $\Omega(x)$ is, and $\xi$ is the threshold. The confidence score is not mentioned in Eq. 1 anywhere else.  This disconnects the latter discussion. This should be referred to elsewhere in order to keep the concept flowing. 

Define the boundary of threshold $\xi$.

The author mentions this is the first time the authors use adversarial attribution.  However, Sec in line 172 suggested that they are using the Integrated Gradient (IG) method by Sundarajan et al. 2017.  Clerfy explicitly whether they are applying IG methods from literature when they take image $x=0$ to $x_{adv}$ as baseline. 

Define indices $x_{(r,s)}$. Are they pixel-wise features or indicate a feature of image x with the same dimension? This could also be related to Fig. 1 for more clarity.

Variable $T$ in line 188 indicates this is the number of intervals; in line 216, it is iteration. Rectify or clarify. 

Figure 2 text should be made legible. 
Figure 2 caption should explain what is in the image rather than stating the method in general. Not explaining the figure properly makes this irrelevant. 

Table 1 text is too small- should be made large, or the Table should be re-arranged.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies out-of-distribution detection. It improves the existing gradient-based OOD detection method GAIA (NeurIPS 2023) by introducing adversarial examples and proposing layer splitting and gradient Integration.

The reviewers share significant concerns about the effectiveness of the proposed method, as its experimental improvement is quite marginal compared to GAIA. On CIFAR100, the performance is almost the same as that of GAIA. On ImageNet, the performance improvements are also quite marginal. Although experimental results are not the only criterion for judgment, the presented experiments do not effectively support the proposed arguments and raise serious concerns about the effectiveness of the proposed method. The authors' responses help to address some other concerns, however, after rebuttal, there is still significant concern about its effectiveness. 

Considering all reviews, the paper does not yet meet the acceptance threshold before these significant issues are adequately addressed.