Title: CausalDiff: Causality-Inspired Disentanglement via Diffusion Model for Adversarial Defense
Venue: NeurIPS 2024
Accepted: yes
Keywords: Adversarial Defense

Diffusion Model

Causal
Abstract: Despite ongoing efforts to defend neural classifiers from adversarial attacks, they remain vulnerable, especially to unseen attacks. In contrast, humans are difficult to be cheated by subtle manipulations, since we make judgments only based on essential factors. Inspired by this observation, we attempt to model label generation with essential label-causative factors and incorporate label-non-causative factors to assist data generation. For an adversarial example, we aim to discriminate the perturbations as non-causative factors and make predictions only based on the label-causative factors. Concretely, we propose a casual diffusion model (CausalDiff) that adapts diffusion models for conditional data generation and disentangles the two types of casual factors by learning towards a novel casual information bottleneck objective. Empirically, CausalDiff has significantly outperformed state-of-the-art defense methods on various unseen attacks, achieving an average robustness of 86.39\% (+4.01\%) on CIFAR-10, 56.25\% (+3.13\%) on CIFAR-100, and 82.62\% (+4.93\%) on GTSRB (German Traffic Sign Recognition Benchmark). The code is available athttps://github.com/CAS-AISafetyBasicResearchGroup/CausalDiff.

CausalDiff: Causality-Inspired Disentanglement via Diffusion Model for Adversarial Defense

Abstract

Despite ongoing efforts to defend neural classifiers from adversarial attacks, they remain vulnerable, especially to unseen attacks. In contrast, humans are difficult to be cheated by subtle manipulations, since we make judgments only based on essential factors. Inspired by this observation, we attempt to model label generation with essential label-causative factors and incorporate label-non-causative factors to assist data generation. For an adversarial example, we aim to discriminate the perturbations as non-causative factors and make predictions only based on the labelcausative factors. Concretely, we propose a casual diffusion model (CausalDiff) that adapts diffusion models for conditional data generation and disentangles the two types of casual factors by learning towards a novel casual information bottleneck objective. Empirically, CausalDiff has significantly outperformed state-of-the-art defense methods on various unseen attacks, achieving an average robustness of 86.39% (+4.01%) on CIFAR-10, 56.25% (+3.13%) on CIFAR-100, and 82.62% (+4.93%) on GTSRB (German Traffic Sign Recognition Benchmark). The code is available at https://github.com/CAS-AISafetyBasicResearchGroup/CausalDiff.

1 Introduction

Neural classifiers, despite their impressive performance in various applications, are susceptible to adversarial attacks [1, 2], which can deceive them into making erroneous judgments on subtly manipulated examples. Such vulnerabilities pose severe threats in safety-critical scenarios such as face recognition [3, 4] and autonomous driving [5]. There has been extensive work on defending against adversarial attacks such as certified defenses [6, 7], adversarial training [8, 9], and adversarial purification Samangouei et al. [10].

CAS stands for Chinese Academy of Sciences

Causalrelation

—→Conditionalgeneration

I→

CausalInference

←←--

Forwarddiffusion

CausalDiff

Training

Inference

S

Z

ConditionalDiffusion

S-based

S

Z

Classification

po(ct-1|xt,s,2)

Xo

+X

XT

Causal

q(ct|ct-1)

Inference

Factor

X

'horse'

Insensitivity

A

CausalInformationBottleneck:

Adversarial

(2'Sx)1-(2)1-(Sx)1+(2x)1=(2'x)

Purification

X

Reconstruction

Disentanglement

Label-causative

factor

Label-non-causative

factor

Image

Label

at timestept

Noisy sample

example

Adversarial

Victim model

Parameter

Although effective, these methods have some limitations. Certified defense methods have limited practicality due to the small certified region that can theoretically guarantee robustness. Adversarial training approaches suffer from a significant decline in robustness against unseen attacks since they take effect by adding adversarial examples into the training set. Purification methods, not designed for specific attacks, struggle to determine the optimal denoising level for unforeseen attacks with differing degrees of perturbation. Consequently, they face challenges in effectively defending against unseen attacks.

In reality, attack behaviors are often unpredictable. Is there a way of strengthening a model to act like humans, i.e., be insensitive to subtle perturbations and robust against various unforeseen attacks? Given an image of an object, we typically identify the key visual features that are necessary to determine its category and disregard other factors such as styles, backgrounds, details, or perturbations. This allows us to make robust judgments. Inspired by this human decision-making process, we would like to learn a model that can disentangle the essential features for determining the category from other non-essential ones.

It is natural to treat the essential features as the causal factors of the label, and both essential and the other features as the causal factors of the entire image. Then, we can learn to disentangle them by modeling the process of data generation and label prediction with a structural causal model (SCM) [11, 12](shown in Fig. 1 (Left)). According to the theoretical results provided by Liu et al. [13], the identifiability of such SCMs can be guaranteed under mild conditions. Although similar SCMs have also been employed in modeling the generation of multi-domain data [14] and adversarial examples [15, 16], they either aim to enhance out-of-distribution robustness or protect the model from a certain type of attack. In contrast, our research focuses on modeling the generation of native in-domain data to enhance adversarial robustness against various unseen attacks.

Fig. 1 (Left) depicts our SCM, where Y denotes the category (e.g., horse) of an input image X ; S denotes the essential semantic features of determining Y (i.e., Y -causative factors), such as the characteristics of eyes, ears, nose, mouth, etc. of horses; and Z represents the other features (i.e., Y -non-causative factors) that are not needed to predict Y but are important to generate X , such as the fur color and the image background. Given an adversarial example produced by an unknown attack, our model aims to disentangle its Y -causative features S from the spurious factors in Z and make a robust prediction. To successfully learn the disentanglement, our SCM is guided by the tasks of data generation and label prediction, where there are three major challenges:

1) How do we model the conditional generation of X given S and Z effectively? To this end, we employ a well-recognized diffusion model with state-of-the-art (SOTA) generative performance and

efficiency, i.e., the Denoising Diffusion Probabilistic Model (DDPM) [17], as the backbone. We further adapt it for conditional generation from latent variables rather than random noise. 2) What training objective should we use to effectively learn the disentanglement of S and Z ? With this regard, we propose a Causal Information Bottleneck ( CIB ) optimization objective. CIB aligns the information in the latent variables ( S, Z ) with observed variables ( X,Y ) with a bottleneck set by the mutual information (MI) between S, Z and X . The derived function will minimize the MI between S and Z while learning the other causal relations, ensuring their disentanglement within the causal framework. 3) Given an adversarial example, what inference strategies shall we adopt to make robust predictions? As shown in Fig. 1 (Right), according to how the adversarial example ˜ X is generated, we first purify it to yield a benign example X ∗ and then infer the Y-causative factor S based on X ∗ for final classification. We name the entire causal defense framework based on diffusion models as CausalDiff.

The experimental results on facing various unseen attacks, encompassing both black-box and whitebox ones, show that CausalDiff has superior performance compared to representative adversarial

0

2

4

6

ε

-Budget

0.0

0.5

1.0

Adversarial Robustness

Discriminative

Generative

Causal

without Disent.

Causal

with Disent.

Table 1: The experimental results of four models on toy data. The variation of latent v and logits p ( y | v ) is measured between clean and adversarial examples. The model margin is estimated by the minimal adversarial perturbation required to flip the label, employing both ℓ 2 and ℓ ∞ norm.

defense baselines including state-of-the-art (SOTA) methods. Specifically, our CausalDiff achieves robustness of 86.39% (+4.01%) on CIFAR10, 56.25% (+3.13%) on CIFAR-100 [18], and 81.79% (+4.93%) on GTSRB [19].

In summary, we highlight our contributions as follows: 1) We propose a novel causal diffusion framework (CausalDiff) to defend against unseen attacks by modeling the generation of native indomain data with the category(Y)-causative factors and the other Y-non-causal factors; 2) We propose a Causal Information Bottleneck (CIB) objective to disentangle Y-causative from Y-non-causative factors during causal model training and an associated inference algorithm for adversarial defense; 3) CausalDiff significantly outperforms SOTA methods in defending against various unseen attacks.

2 Related Work

Adversarial Defense. Adversarial training primarily focuses on optimizing the training algorithm [8, 20, 21], incorporating data augmentation [22, 23, 9], and enhancing acceleration [24, 25]. Despite its effectiveness, the trained models could still be vulnerable to unseen attacks [26, 27]. Adversarial purification, orthogonal to our work, utilizes a generative model to purify adversarial noise from examples before classification. Leveraging diffusion models [28, 17, 29, 30], diffusion-based purification have shown to be effective [31-35, 27].

Causal Learning for Robustness. Causal representation learning [12, 36, 14, 37] focuses on discovering invariant mechanisms within structural causal models and has achieved remarkable performance in improving model transferability [14, 13, 38-40] and interpretability [41, 42]. Moreover, in terms of adversarial robustness, researchers [43, 15, 44, 45, 16, 46-48] have attempted to model the attack behaviors in causal structures to identify the adversarial factors. However, modeling particular attack types will limit the model robustness on other types of attacks [27].

Conditional Diffusion Model. Diffusion models [28, 17, 29, 30, 49] have achieved compelling image generation capabilities. To equip them with the ability of controllable generation, the sampling

process can be guided by 1) classifier confidence to generate images of a certain category [30, 50, 51], and 2) semantic embedding of text content or a certain style [52-55].

3 Pilot Study on Toy Data

y=0

y=1

(a) Discriminative

(b) Generative

(c) Causal without Disent.

(d) Causal with Disent.

To compare our proposed causal model with other representative models in terms of adversarial robustness, we constructed a toy dataset according to the hypothesis that the essential factors are the basis of generating labels and they also determine the generation of data together with label-noncausative features. Pilot studies on this toy data will provide insights into whether our causal model will work and why it works.

3.1 Experimental Settings

Toy Data Construction. We constructed 2000 samples following the causal structure in Fig. 1 (Left). Specifically, for each data point, we randomly sample the vectors s and z from two different normal distributions, project s to a score y s with random weights, obtain its label by comparing y s with the medium score, and generate the representation x by projecting the concatenation [ s ; z ] with another random matrix. Please refer to Appendix B.3 for detailed information.

Models for Comparisons. We investigated four representative models: 1) a Discriminative model for classification, 2) a Generative model that predicts the label of an adversarial example with max y p ( x | y ) [27], 3) a causal model without disentanglement ( Causal without Disent. ) that models the generation of both x and y with the same latent factor v , 4) our causal model with disentanglement ( Causal with Disent. ) that is illustrated in Fig. 1. The concrete structures of the four models and further details are presented in Appendix B.3.

3.2 Experimental Observations

Adversarial Robustness. We evaluate the model's robustness against a 100-step PGD attack with varying ϵ -budgets under the ℓ ∞ bound [8]. Model performance in terms of both clean accuracy (when ϵ = 0 ) and robust accuracy is examined.

As shown in Fig. 2: 1) The Discriminative model exhibits the highest clean accuracy but suffers a rapid decline in robustness, dropping to 0% when the attack budget ϵ reaches 1 . 2 ., which highlights its vulnerability. 2) The Generative model has the lowest clean accuracy while its robustness does not dramatically regress with larger attack budgets. The low clean accuracy may be due to the inconsistency between the generation process it modeled and the way this toy dataset is constructed. The small gap between clean and robust accuracy indicates its decent effectiveness in defending against adversarial attacks. 3) Causal with Disent. obtains the second-best clean accuracy, yet its robustness gradually declines with increasing attack strength, maintaining 61.4% robustness at ϵ = 6 . 4 . 4) Our model, Causal with Disent. , has slightly lower clean accuracy than Causal without Disent. but the best robustness among the four. As the attack strength increases, it maintains at least 71.8% robust accuracy at ϵ = 6 . 4 , indicating that it is promising to enhance model robustness by causal modeling with disentanglement of the label-causative factors from the non-causative ones.

Sensitivity to Perturbations. To delve further into how the model behavior changes when defending against adversarial perturbations, we measure the variation between latent variables of clean and adversarial examples, denoted as △ v = 1 -cosine ( v, v adv ) , where v and v adv represents the latent vector of clean and adversarial example, respectively ( v = s for Causal with Disent. ). We also compute the change of predicted logits, △ p ( y | v ) = p ( y | v ) -p ( y | v adv ) , with y being the true class label. Note that a larger variation in the latent factor for prediction, or the predicted logits, results in increased insensitivity to perturbations.

The experimental results, as shown in Tab. 4, indicate that Causal with Disent. , compared to Causal without Disent. , exhibits less sensitivity in both latent variables and prediction outcomes. This suggests that by disentangling the label-causative factor s , it becomes more challenging for attackers to perturb the model and alter its prediction. The small variation of v in the Generative model is probably attributed to the lack of discriminability in v .

Prediction Margins. To intuitively explain the sensitivity of each model to perturbations, we estimate the minimal adversarial perturbation ∥ δ ∥ under PGD required to flip the correct model prediction y of a sample x . It can be interpreted as the prediction margins in terms of perturbation, denoted as margin( x, y ) = min ∥ δ ∥ , subject to p ( y | x + δ ) < p (¯ y | x + δ ) [56]. We measure the ∥ δ ∥ under ℓ 2 and ℓ ∞ norms. Additionally, we visualized the latent vector space of each model to intuitively observe the margin of the classification boundary.

As shown in Tab. 4, Causal with Disent. has the largest prediction margin. This implies that an attacker would need to add significantly more perturbation to successfully cheat our model. We can draw similar conclusions from Fig. 3, which illustrates the distribution of predictive features extracted by each model for correctly classified clean samples across categories. Again, Causal with Disent. has the largest margin between the classes, indicating that it has high confidence in its prediction and increases the cost and difficulty for an attacker to succeed.

4 Causal Diffusion Model

To enhance model robustness on real-world data, in this section, we propose a Causal Diffusion (CausalDiff) Modeling approach, that couples our previously studied SCM (shown in Fig. 6(d)) with diffusion models. We will introduce the three major components in CausalDiff: conditional diffusion generation, causal information bottleneck optimization, and adversarial example inference. We take the Denoising Diffusion Probabilistic Model (DDPM) [17] as an instance for illustration and it can be easily adapted to other diffusion models.

4.1 Conditional Diffusion Generation

Standard diffusion models generate images based on random noise which do not apply to the conditional generation of X based on S and Z in our SCM. In standard diffusion models, at each time step t during denoising, a UNet ϵ θ ( x t , t ) is employed to decode an image given input x t . While diffusion models are highly effective in image generation, they lack an explicit decoder component for generating images from latent variables. To handle this, we develop a conditional DDPM using latent variables S and Z controlling the generation process. This approach draws inspiration from both the class-conditional diffusion model [51] as well as the style control mechanism introduced in DiffAE [52]. The output h t out at each layer of the UNet depends on t and x t , i.e.,

where h is the feature map of x t , t s and t b are the scale and bias of timestep t .

Inspired by the class-conditional diffusion model [51] and the style control mechanism in DiffAE [52], we adapt the standard diffusion generation to be conditioned on S and Z in addition to timestep t . Specifically, the UNet becomes ϵ θ ( x t , t, s, z ) , the final output h t,s,z out is calculated based on the original h t out , the hidden state s and z (representing causal factor S and Z ) encoded from input x :

where z s and s b are produced from the affine projections of z and s respectively, i.e., z s = Affine z ( z ) and s b = Affine s ( s ) . As such, the label-causative factor S acts as a bias that can affect the direction of the latent vector and change its semantics, while the label-non-causative factor Z can only scale the latent vector in a similar way to style control.

4.2 Causal Information Bottleneck Optimization

To learn the causal factors S, Z in our SCM and disentangle them, we propose a Causal Information Bottleneck (CIB) optimization objective. It maximizes the mutual information between the latent factors S, Z and the observed data sample ( X,Y ) with an information bottleneck that constrains the information retained in S, Z with respect to X .

Specifically, to align the information captured in the latent factors with the observed data ( X,Y ) , we maximize the mutual information between them, denoted as I(X , Y;S , Z) , which can be derived as:

A detailed proof is presented in Appendix A.1. Among the resultant terms, I( X ; Y ) is solely dependent on the observed data, independent of latent variables or the causal model, and thus can be ignored in the learning process. Maximizing I( X ; S, Z ) will urge S and Z to capture ample information about X . I( S ; Y ) indicates that the Y -causative factor S should be correlated with Y . The term, -I( S ; Z ) ensures S and Z to be effectively disentangled.

Existing work on optimizing similar SCMs adapts the Evidence Lower BOund (ELBO) from Variational Autoencoders (VAE) to formulate the causal ELBO objectives [14]. This objective only maximizes the likelihood of ( X,Y ) , and does not consider the latent factors. Consequently, the final optimization goal of causal ELBO differs from I ( X,Y ; S, Z ) in that it does not have -I( S ; Z ) in Eq. (3), which is crucial for disentanglement. Further details are discussed in Appendix A.4.

To avoid X contain too many unimportant details, we constrain the mutual information between X and the latent factors S, Z with an information bottleneck I c . Then, the updated objective becomes:

Employing Lagrange multiplier λ ≥ 0 , we formulate our objective as maxI( X,Y ; S,Z ) -λ (I( X ; S, Z ) -I c ) . Since I c is a constant, it is equal to maximize the Causal Information Bottleneck (CIB):

I ( X ; S, Z ) and -λI ( X ; S, Z ) indicate two opposing optimization directions. Because it is unclear whether (1 -λ ) should be positive or negative, we approximate these terms via two separate lower bounds instead of combining them.

To maximize the Causal Information Bottleneck ( CIB ) in Eq. (5), we derive its lower bound as the concrete training loss function. When using the diffusion model ϵ θ , classifier f y ( s ; θ ) (to estimate p θ ( y | s ) ), and the encoder f s,z ( x ; θ ) (to estimate p θ ( s, z | x ) ), the lower bound of CIB is:

where p θ ( s | z ) is a variational distribution to estimate p ( s | z ) and I θ CLUB ( S ; Z ) = E p ( s,z ) [log p θ ( s | z )] -E p ( z ) E p ( s ) [log p θ ( s | z )] represents the Contrastive Log-Ratio Upper Bound (CLUB) of mutual information proposed by Cheng et al. [57]. p θ ( x | s, z ) represents the likelihood estimated by the conditional diffusion model. D KL refers to the Kullback-Leibler (KL) divergence [58]; and q ( · ) denotes the prior distribution of the latent variable, e.g., a normal distribution N ( 0 , I ) . A detailed proof can be found in Appendix A.2.

Loss Function . Thus, maximizing the lower bound of CIB is equal to minimizing the loss function:

where α, λ, γ, η determine the weighting of each term in the optimization process. A detailed derivation can be found in Appendix A.3.

Algorithm . We pretrain the model with the data reconstruction loss (the first term in Eq. (7)) alone before training the model with the entire loss, so that the model can learn the causal factors, disentanglement, and classification from a decent starting point. For space concern, we illustrate the training process in Appendix B.2 involves the training algorithm (Algorithm 1) and the pretraining algorithm (Algorithm 2).

4.3 Adversarial Example Inference

Guided by the causal generation of an adversarial example ˜ X according to Fig. 1 (Right), we illustrate the process for robust classification. Following a typical attack paradigm, ˜ X is produced by adding an adversarial perturbation to a target clean example X when attacking a model θ . To make a robust prediction on ˜ X , our robust inference process comprises three steps: 1) purifying ˜ X to benign X by the unconditional diffusion model ϵ θ ( x t , t ) , 2) inferring S and Z from X utilizing the causal model ϵ θ ( x t , t, s, z ) , and 3) predicting Y based on S using a classifier f y ( s ; θ ) .

Adversarial Purification (AP). We follow the concept of Likelihood Maximization (LM) [33, 27] to purify the adversarial example ˜ X to a benign X ∗ by maximizing the data log-likelihood log p θ ( x ) :

Concretely, we maximize its lower bound utilizing the unconditional diffusion model ϵ θ ( x t , t ) trained according to Section C.2. Chen et al. [27] suggest using one random timestep t during each purification iteration while we believe that smaller timesteps should be more effective since they retain more information from the original example. Thus, we limit the random selection to within the first 50 timesteps. This way significantly boosts adversarial robustness, which will be discussed in Section 5.3.

Causal Factor Inference (CFI). In order to infer the causal and non-causal factors S and Z , which can reconstruct the original image X , we optimize the latent variables by maximizing the conditional likelihood p θ ( x | s, z ) employing the trained conditional diffusion model ϵ θ ( x t , t, s, z ) :

Similarly to purification, we obtain s ∗ and z ∗ by maximizing the lower bound -E ϵ,t [ w t ∥ ϵ θ ( x t , t, s, z ) -ϵ ∥ ] using the conditional diffusion model ϵ θ ( x t , t, s, z ) . For efficiency concerns, instead of using all the timesteps for estimation as in Chen et al. [27], we sample N purify = 5 timesteps at the same intervals across the entire timesteps.

Latent-S-Based Classification (LSBC). After obtaining s ∗ according to Eq. (9), we use the trained classifier f y ( s ; θ ) to predict label Y :

Within our inference pipeline, both adversarial purification and causal factor inference leverage the diffusion model learned toward the CIB optimization objective while they take effect independently. When we combine these two approaches, adversarial robustness could be enhanced further. The concrete inference algorithm in Algorithm 3 is presented as Appendix B.2.

4.4 Comparison with Adversarial Purification

First, our CausalDiff can be viewed as semantic-level purification . Instead of pixel-level denoising, CausalDiff purifies an image in the latent space, trying to remove the effect of perturbation by putting it to the label-non-causative features. Second, conventional purification inevitably loses information essential for classification during denoising. By disentangling label-causative features from labelnon-causative features, CausalDiff can retain essential information in the label-causative features to a large extent. Third, unlike pixel-level purification which does not know the optimal denoising level for various attacks, CausalDiff acts adaptively on different attacks by the causal inference of S and Z . Fourth, they can be combined to further enhance adversarial robustness.

5 Experiments

In this section, we introduce the experimental settings in Section 5.1. Section 5.2 presents the main results defending against unforeseen attacks on CIFAR-10, CIFAR-100, and GTSRB datasets. We then evaluate the effectiveness of individual components of CausalDiff in Section 5.3. Due to space constraints, we provide analyses of core components during training and inference in Appendix C.1 and Appendix C.2. Additionally, we showcase examples in Appendix C.3.

5.1 Experimental Settings

Datasets and Model Architecture. Our experiments utilize the CIFAR-10, CIFAR-100 [18] and GTSRB[19] datasets. CIFAR-10 and CIFAR-100 each consists of 50,000 training images, categorized into 10 and 100 classes, respectively. GTSRB comprises 39,209 training images (each histogram equalized and resized to 3 × 32 × 32 ) of German traffic signs, categorized into 43 classes. We use DDPM [17] as the diffusion model. Further details are available in Appendix B.2.

Attack Evaluation Method. For the CIFAR-10 dataset, we utilize seven types of attack strategies with both ℓ ∞ and ℓ 2 norm bounds for evaluation. These strategies include StAdv attack [60], BPDA+EOT, and AutoAttack [2] (AA), which comprises white-box attacks such as APGD-ce, APGD-t, FAB-t, and a black-box Square Attack. For CIFAR-100, we follow the setting of Chen et al. [27], evaluating against the ℓ ∞ threat model with ϵ = 8 / 255 . For the GTSRB dataset, we utilize four types of attacks as well as fog corruptions . Attack methods include AutoAttack, which comprises APGD-ce, APGD-t, FAB-t, and Square Attack.

5.2 Comparisons on Unseen Attacks

CIFAR-10 . From the experimental results for CIFAR-10 presented in Table 2, we have several observations: 1) The adversarial training methods perform robustly on the same type of attacks they use for training but poorly on other unseen attacks. The only exception is that when the models employ

the most effective attack (i.e., ATℓ ∞ ) for adversarial training, the robustness regarding ℓ 2 is not hurt much. 2) In contrast, purification methods, especially the ones based on more powerful diffusion models (DDPM and EDM), have decent robustness on each unseen attack and much better average robustness. This is reasonable since they learn to purify the adversarial samples without targeting any specific attacks. 3) Our CausalDiff performs the best not only regarding the average robustness but also on each type of attack. Remarkably, the average robust accuracy (86.39%) is only 3.84% lower than the clean accuracy (90.23%) and it boosts the robustness on the most challenging attack ( ℓ ∞ ) to 83.01%. Notably, the reported CausalDiff is based on DDPM, which acts less effectively than EDM, which can be seen from AT and LM purification. Grounded on a stronger backbone, CausalDiff could achieve even better performance. Table 4 shows the performance under the BPDA + EOT (EOT = 20) attack. Under this attack, models whose gradients are not accessible and also be compared (e.g., ADP [31]). The results again confirm the superior performance of CausalDiff across different attack types.

airplane

bird

deer

frog

ship

automobile

cat

6op

horse

truck

label-non-causative

concat [s;z]

label-causativefactors

factorz

Table 4: Clean and robust accuracy on CIFAR-10 against BPDA+EOT against ℓ ∞ ( ϵ = 8 / 255 ) threat model.

DDPM. We believe more augmented data and a stronger diffusion backbone like EDM (used by RDC [27]) could further enhance the performance.

GTSRB . The left part of Table 3 show that CausalDiff also has compelling robustness on traffic sign classification, in terms of not only unforeseen adversarial attacks but also natural corruptions like fog. Evaluations based on different types of tasks, and different numbers of classes (10, 100, and 43) have all shown the efficacy of CausalDiff.

5.3 Ablation Study

As mentioned in Section 4.3, our CausalDiff contains Adversarial Purification (AP), Causal Factor Inference (CFI), and Latent-S-Based Classification (LSBC). The last block of Table 2 shows the individual effect of the AP and CFI in CausalDiff. We can see: 1) Our DDPM-based purification (CausalDiff w/o CFI, achieved by AP plus a standard classifier) performs similarly to LM-EDM and is significantly better than LM-DDPM, showing that the strategy of sampling within small timesteps for purification is much more effective than entire timesteps. (We show more analysis on this in Appendix C.2. ) 2) The core component of CasualDiff - causal disentanglement (CausalDiff w/o AP) outperforms all the baselines in terms of average robustness except RDC which incorporates an extra LM purification step. It shows that modeling the generative of native in-domain data can enhance the model's inherent robustness and thus effectively defend against various types of attacks.

5.4 Visualization of Latent Factors

To understand how the latent causal factors S and Z in CausalDiff take effect during adversarial classification, we visualize S , Z , and their concatenation using t-SNE in Fig. 4. We randomly sampled 5000 correctly classified test samples from CIFAR-10 for visualization. We find that the Y-causative factor S of samples in each category are located in the same cluster, with clear margins between different clusters except the categories - dog, cat, and bird. These three categories share more commonalities than the others and are not surprising to have blurred boundaries. Additionally,

the S vectors of airplanes (dark blue) are near those of birds (green), and trucks have S vectors near automobiles. In contrast to the S vectors, the vectors of Z do not exhibit correlations with the categories. This also aligns with our objective to extract the Y-non-causative factors to Z . The vectors of their concatenation, i.e., [ s ; z ] , also display in clusters but with much more blurred boundaries. These observations are consistent with our commonsense knowledge, showing that CausalDiff has learned reasonable Y-causative factors by S and Y-non-causative factors by Z .

6 Conclusion

We develop a causal model based on diffusion model to improve adversarial robustness. A pilot study on toy data suggests that the model defends against adversarial attacks by leveraging label-causative features to resist perturbations and expand the model's margin. Moreover, on the CIFAR-10, CIFAR100 and GTSRB datasets, our model appears to capture semantic features consistent with the human decision-making process and surpass all baseline models, achieving state-of-the-art performance in adversarial robustness, particularly against unseen attacks.

Acknowledgments and Disclosure of Funding

This work is supported by the Strategic Priority Research Program of the Chinese Academy of Sciences, Grant No. XDB0680101, CAS Project for Young Scientists in Basic Research under Grant No. YSBR-034, the Innovation Project of ICT CAS under Grant No. E261090, the National Natural Science Foundation of China (NSFC) under Grants No. 62302486, the Innovation Project of ICT CAS under Grants No. E361140, the CAS Special Research Assistant Funding Project, the project under Grants No. JCKY2022130C039, and the Strategic Priority Research Program of the CAS under Grants No. XDB0680102. We would like to thank the UIBE GVC Laboratory and the Digital Economy Laboratory at the University of International Business and Economics for their provision of computational resources and support.

References

I. J. Goodfellow, J. Shlens, and C. Szegedy, 'Explaining and harnessing adversarial examples,' arXiv preprint arXiv:1412.6572 , 2014.

F. Croce and M. Hein, 'Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,' in International conference on machine learning . PMLR, 2020, pp. 2206-2216.

K. Eykholt, I. Evtimov, E. Fernandes, B. Li, A. Rahmati, C. Xiao, A. Prakash, T. Kohno, and D. Song, 'Robust physical-world attacks on deep learning visual classification,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 1625-1634.

M. Sharif, S. Bhagavatula, L. Bauer, and M. K. Reiter, 'Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition,' in Proceedings of the 2016 acm sigsac conference on computer and communications security , 2016, pp. 1528-1540.

J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt et al. , 'Towards fully autonomous driving: Systems and algorithms,' in 2011 IEEE intelligent vehicles symposium (IV) . IEEE, 2011, pp. 163-168.

N. Carlini, G. Katz, C. Barrett, and D. L. Dill, 'Provably minimally-distorted adversarial examples,' arXiv preprint arXiv:1709.10207 , 2017.

M. Hein and M. Andriushchenko, 'Formal guarantees on the robustness of a classifier against adversarial manipulation,' Advances in neural information processing systems , vol. 30, 2017.

A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, 'Towards deep learning models resistant to adversarial attacks,' arXiv preprint arXiv:1706.06083 , 2017.

Z. Wang, T. Pang, C. Du, M. Lin, W. Liu, and S. Yan, 'Better diffusion models further improve adversarial training,' arXiv preprint arXiv:2302.04638 , 2023.

P. Samangouei, M. Kabkab, and R. Chellappa, 'Defense-gan: Protecting classifiers against adversarial attacks using generative models,' arXiv preprint arXiv:1805.06605 , 2018.

J. Pearl and D. Mackenzie, The book of why: the new science of cause and effect . Basic books, 2018.

B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio, 'Toward causal representation learning,' Proceedings of the IEEE , vol. 109, no. 5, pp. 612-634, 2021.

C. Liu, X. Sun, J. Wang, H. Tang, T. Li, T. Qin, W. Chen, and T.-Y. Liu, 'Learning causal semantic representation for out-of-distribution prediction,' Advances in Neural Information Processing Systems , vol. 34, pp. 6155-6170, 2021.

X. Sun, B. Wu, X. Zheng, C. Liu, W. Chen, T. Qin, and T.-Y. Liu, 'Recovering latent causal factor for generalization to distributional shifts,' Advances in Neural Information Processing Systems , vol. 34, pp. 16 846-16 859, 2021.

Y. Zhang, M. Gong, T. Liu, G. Niu, X. Tian, B. Han, B. Schölkopf, and K. Zhang, 'Causaladv: Adversarial robustness through the lens of causality,' arXiv preprint arXiv:2106.06196 , 2021.

Q. Ren, Y. Chen, Y. Mo, Q. Wu, and J. Yan, 'Dice: Domain-attack invariant causal learning for improved data privacy protection and adversarial robustness,' in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2022, pp. 1483-1492.

J. Ho, A. Jain, and P. Abbeel, 'Denoising diffusion probabilistic models,' Advances in Neural Information Processing Systems , vol. 33, pp. 6840-6851, 2020.

A. Krizhevsky, G. Hinton et al. , 'Learning multiple layers of features from tiny images,' 2009.

J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel, 'The german traffic sign recognition benchmark: a multi-class classification competition,' in The 2011 international joint conference on neural networks . IEEE, 2011, pp. 1453-1460.

H. Zhang, Y. Yu, J. Jiao, E. Xing, L. El Ghaoui, and M. Jordan, 'Theoretically principled trade-off between robustness and accuracy,' in International conference on machine learning . PMLR, 2019, pp. 7472-7482.

C. Qin, J. Martens, S. Gowal, D. Krishnan, K. Dvijotham, A. Fawzi, S. De, R. Stanforth, and P. Kohli, 'Adversarial robustness through local linearization,' Advances in Neural Information Processing Systems , vol. 32, 2019.

S. Gowal, S.-A. Rebuffi, O. Wiles, F. Stimberg, D. A. Calian, and T. A. Mann, 'Improving robustness using generated data,' Advances in Neural Information Processing Systems , vol. 34, pp. 4218-4233, 2021.

S.-A. Rebuffi, S. Gowal, D. A. Calian, F. Stimberg, O. Wiles, and T. Mann, 'Fixing data augmentation to improve adversarial robustness,' arXiv preprint arXiv:2103.01946 , 2021.

A. Shafahi, M. Najibi, M. A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. S. Davis, G. Taylor, and T. Goldstein, 'Adversarial training for free!' Advances in Neural Information Processing Systems , vol. 32, 2019.

H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and T. Zhao, 'Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization,' arXiv preprint arXiv:1911.03437 , 2019.

C. Laidlaw, S. Singla, and S. Feizi, 'Perceptual adversarial robustness: Defense against unseen threat models,' arXiv preprint arXiv:2006.12655 , 2020.

H. Chen, Y. Dong, Z. Wang, X. Yang, C. Duan, H. Su, and J. Zhu, 'Robust classification via a single diffusion model,' arXiv preprint arXiv:2305.15241 , 2023.

J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, 'Deep unsupervised learning using nonequilibrium thermodynamics,' in International Conference on Machine Learning . PMLR, 2015, pp. 2256-2265.

Y. Song and S. Ermon, 'Generative modeling by estimating gradients of the data distribution,' Advances in neural information processing systems , vol. 32, 2019.

Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, 'Score-based generative modeling through stochastic differential equations,' arXiv preprint arXiv:2011.13456 , 2020.

J. Yoon, S. J. Hwang, and J. Lee, 'Adversarial purification with score-based generative models,' in International Conference on Machine Learning . PMLR, 2021, pp. 12 062-12 072.

C. Xiao, Z. Chen, K. Jin, J. Wang, W. Nie, M. Liu, A. Anandkumar, B. Li, and D. Song, 'Densepure: Understanding diffusion models towards adversarial robustness,' arXiv preprint arXiv:2211.00322 , 2022.

W. Nie, B. Guo, Y. Huang, C. Xiao, A. Vahdat, and A. Anandkumar, 'Diffusion models for adversarial purification,' arXiv preprint arXiv:2205.07460 , 2022.

Q. Wu, H. Ye, and Y. Gu, 'Guided diffusion model for adversarial purification from random noise,' arXiv preprint arXiv:2206.10875 , 2022.

J. Wang, Z. Lyu, D. Lin, B. Dai, and H. Fu, 'Guided diffusion model for adversarial purification,' arXiv preprint arXiv:2205.14969 , 2022.

J. Mitrovic, B. McWilliams, J. Walker, L. Buesing, and C. Blundell, 'Representation learning via invariant causal mechanisms,' arXiv preprint arXiv:2010.07922 , 2020.

B. Schölkopf, 'Causality for machine learning,' in Probabilistic and Causal Inference: The Works of Judea Pearl , 2022, pp. 765-804.

Y. Chen, Y. Zhang, Y. Bian, H. Yang, M. Kaili, B. Xie, T. Liu, B. Han, and J. Cheng, 'Learning causally invariant representations for out-of-distribution generalization on graphs,' Advances in Neural Information Processing Systems , vol. 35, pp. 22 131-22 148, 2022.

W. Wang, X. Lin, F. Feng, X. He, M. Lin, and T.-S. Chua, 'Causal representation learning for out-of-distribution recommendation,' in Proceedings of the ACM Web Conference 2022 , 2022, pp. 3562-3571.

R. Wang, M. Yi, Z. Chen, and S. Zhu, 'Out-of-distribution generalization with causal invariant transformations,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 375-385.

R. Moraffah, M. Karami, R. Guo, A. Raglin, and H. Liu, 'Causal interpretability for machine learning-problems, methods and evaluation,' ACM SIGKDD Explorations Newsletter , vol. 22, no. 1, pp. 18-33, 2020.

G. Xu, T. D. Duong, Q. Li, S. Liu, and X. Wang, 'Causality learning: A new perspective for interpretable machine learning,' arXiv preprint arXiv:2006.16789 , 2020.

C. Zhang, K. Zhang, and Y. Li, 'A causal view on robustness of neural networks,' Advances in Neural Information Processing Systems , vol. 33, pp. 289-301, 2020.

K. Tang, M. Tao, and H. Zhang, 'Adversarial visual robustness by causal intervention,' arXiv preprint arXiv:2106.09534 , 2021.

S. Yang, T. Guo, Y. Wang, and C. Xu, 'Adversarial robustness through disentangled representations,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 35, no. 4, 2021, pp. 3145-3153.

B.-K. Lee, J. Kim, and Y. M. Ro, 'Mitigating adversarial vulnerability through causal parameter estimation by adversarial double machine learning,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 4499-4509.

J. Kim, B.-K. Lee, and Y. M. Ro, 'Demystifying causal features on adversarial examples and causal inoculation for robust network by adversarial instrumental variable regression,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 12 302-12 312.

H. Hua, J. Yan, X. Fang, W. Huang, H. Yin, and W. Ge, 'Causal information bottleneck boosts adversarial robustness of deep neural network,' arXiv preprint arXiv:2210.14229 , 2022.

T. Karras, M. Aittala, T. Aila, and S. Laine, 'Elucidating the design space of diffusion-based generative models,' Advances in Neural Information Processing Systems , vol. 35, pp. 26 56526 577, 2022.

P. Dhariwal and A. Nichol, 'Diffusion models beat gans on image synthesis,' Advances in Neural Information Processing Systems , vol. 34, pp. 8780-8794, 2021.

J. Ho and T. Salimans, 'Classifier-free diffusion guidance,' arXiv preprint arXiv:2207.12598 , 2022.

K. Preechakul, N. Chatthee, S. Wizadwongsa, and S. Suwajanakorn, 'Diffusion autoencoders: Toward a meaningful and decodable representation,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 10 619-10 629.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, 'High-resolution image synthesis with latent diffusion models,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 10 684-10 695.

A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen, 'Glide: Towards photorealistic image generation and editing with text-guided diffusion models,' arXiv preprint arXiv:2112.10741 , 2021.

A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, 'Hierarchical text-conditional image generation with clip latents,' arXiv preprint arXiv:2204.06125 , 2022.

G. W. Ding, Y. Sharma, K. Y. C. Lui, and R. Huang, 'Mma training: Direct input space margin maximization through adversarial training,' arXiv preprint arXiv:1812.02637 , 2018.

P. Cheng, W. Hao, S. Dai, J. Liu, Z. Gan, and L. Carin, 'Club: A contrastive log-ratio upper bound of mutual information,' in International conference on machine learning . PMLR, 2020, pp. 1779-1788.

S. Kullback, Information theory and statistics . Courier Corporation, 1997.

R. S. Zimmermann, L. Schott, Y. Song, B. A. Dunn, and D. A. Klindt, 'Score-based generative classifiers,' arXiv preprint arXiv:2110.00473 , 2021.

C. Xiao, J.-Y. Zhu, B. Li, W. He, M. Liu, and D. Song, 'Spatially transformed adversarial examples,' arXiv preprint arXiv:1801.02612 , 2018.

T. Wu, L. Tong, and Y. Vorobeychik, 'Defending against physically realizable attacks on image classification,' arXiv preprint arXiv:1909.09552 , 2019.

M. Hill, J. Mitchell, and S.-C. Zhu, 'Stochastic security: Adversarial defense using long-run dynamics of energy-based models,' arXiv preprint arXiv:2005.13525 , 2020.

M. Boudiaf, J. Rony, I. M. Ziko, E. Granger, M. Pedersoli, P. Piantanida, and I. B. Ayed, 'A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses,' in European conference on computer vision . Springer, 2020, pp. 548-564.

A Proof of Propositions

In this section, we will present the detailed proof for the theoretical results mentioned in the main paper.

A.1 Proof of Causal Information Bottleneck ( CIB )

According to the Structural Causal Model (SCM), we have p ( x, y, s, z ) = p ( s ) p ( z ) p ( x | s, z ) p ( y | s ) . Thus, the Causal Information Bottleneck (CIB) can be represented as:

Therefore, we have proved the result in Eq. (5)

A.2 Proof of the Lower Bound of CIB

According to the result in Eq. (11), we further prove its lower bound shown in Eq. (6) in this section.

As for the lower bound of the recontruction term I( X ; S, Z ) , we have:

As for the upper bound of I( X ; S, Z ) , we have:

Regarding the label prediction term I( Y ; S ) , we can maximize the mutual information between factor S and label Y by maximize E p ( y,s ) [log p θ ( y | s )] as well as employing a cross-entropy loss function, according to Boudiaf et al. [63].

As for the disentangle term I( S ; Z ) , according to the Contrastive Log-Ratio Upper Bound (CLUB) of mutual information proposed by Cheng et al. [57], we have

where p θ ( s | z ) is a variational distribution to estimate p ( s | z ) .

Thus, we have proved the results in Eq. (6) that

A.3 Detailed Derivation of Loss Function

Based on the result in Eq. (15), we can optimize the Causal Information Bottleneck (CIB) I( X,Y ; S, Z ) by maximizing its theoratically lower bound. In this part, we discuss the detailed derivation for the lower bound of CIB in term of designing the loss function proposed in Eq. (7).

Specifically, for the reconstruction term E p ( x,s,z ) [log p θ ( x | s, z )] , we can maximize the log-likelihood estimated by our conditional diffusion model:

where ϵ θ ( x t , t, s, z ) denotes our conditional diffusion model, and constant C is negligible [17].

Regarding the label prediction term E p ( y,s ) [log p θ ( y | s )] , following the results proposed by Boudiaf et al. [63], we can leveraging a cross-entropy loss function to maximize the mutual information between factor S and label Y .

As for the disentanglement term I( S ; Z ) , we following the optimization strategies proposed by Cheng et al. [57], leveraging a predictor p θ to learn the relationship between S and Z so as to estimate I θ CLUB ( S ; Z ) .

Overall, we have proved the loss function of our Causal Information Bottleneck (CIB) proposed in Eq. (7):

Algorithm 1 CausalDiff Algorithm

Require: Dataset D ; The CausalDiff parameterized by θ pretrained by Algorithm 2 involves an UNet ϵ θ ( x t , t, s, z ) with diffusion timestep T , an encoder f ( s,z ) ( x ; θ ) , a classifier f y ( s ; θ ) , and an MI estimater f CLUB ( s, z ; θ ) for the CLUB loss; an optimizer optim( · ) , dropout probability p drop of s, z for simultaneously training an unconditional generation, hyperparameters α, λ, γ, η .

Initialize:

Load f s,z ( x ; θ ) and ϵ θ from the pretrained model θ . Initialize f y ( s ; θ ) and f CLUB ( s, z ; θ ) randomly. for ep=1 to N 2 do Get mini-batch ( x, y ) ∼ D s, z = f ( s,z ) ( x ; θ ) t ∼ Uniform( { 1 , 2 , ..., T } ) , ϵ ∼ N ( 0 , I ) if rand (0 , 1) ≤ p drop then Compute loss L ( x ; θ ) = ∥ ϵ θ ( x t , t ) -ϵ ∥ 2 2 Update ϵ θ with optim( θ, L ( x ; θ )) else Compute loss L ( x, y, s, z ; θ ) according to Eq. (7) Update ϵ θ ( x t , t, s, z ) , f ( s,z ) ( x ; θ ) , and f y ( s ; θ ) by optim( θ, L ( x, y, s, z ; θ )) end if Update f CLUB ( s, z ; θ ) according to the CLUB algorithm [57] end for

A.4 ELBO (Evidence Lower BOund) V.S. MI (Mutual Information)

the Causal Evidence Lower BOund (ELBO) for multi-domain datasets as proposed by Sun et al. [14], we directly formulate the Causal ELBO within our causal structure, as depicted in Fig. 1 (Left). Given that p ( x, y, s, z ) = p ( s ) p ( z ) p ( x | s, z ) p ( y | s ) , we can derive the Causal ELBO in single domain as follows:

where p θ is to learn the ground-truth p and q ψ is variational distribution to mimic p θ .

Specifically, the causal ELBO, compared to our Causal Information Bottleneck (CIB), incorporates the reconstruction term log p θ ( x | s, z ) , the insensitivity term log p θ ( s,z ) q ψ ( s,z | x ) , and the label prediction term log p θ ( y | s ) . However, it overlooks the disentanglement of latent factors, a critical aspect for effectively learning the causal model. Additionally, we have empirically assessed the robustness of models trained with either CIB or causal ELBO (equivalent to η = 0 ) in section C.1. This evaluation aims to investigate the efficacy of the disentanglement term I( S ; Z ) .

B More Implementation Details

B.1 Baselines

We include representative defense methods of adversarial training (AT), purification, and other types (e.g., generative-model-based approach) as baselines. Specifically, we compare with the AT methods [9, 23] that use DDPM or the Elucidating Diffusion Model (EDM) [49] for data augmentation and a theoretical framework TRADES [20] for adversarial training. We also include causality-based AT baselines such as CausalAdv [15] and DICE [16]. Purification baselines include DiffPure [33] (grounded on Score SDE [30]) and diffusion-based Likelihood Maximization (LM)[27] with EDM (as in the original paper) and DDPM (we reproduced). Other defense baselines comprise generative classifiers such as Score-Based Generative Classifier (SBGC) [59], [27], and the causality-based

Algorithm 2 CausalDiff Pretrain Algorithm

Require: Dataset D ; a diffusion model ϵ θ , an encoder f ( s,z ) ( x ; θ ) , a classifier f y ( s ; θ ) ; an optimizer optim( · ) , probability p drop of training samples for unconditional diffusion, diffusion training epoch N 1 .

Initialize: randomly initialize parameter θ .

for ep=1

(

x, y

)

s, z

=

to

N

∼ D

f

(

s,z

1

)

(

do

x

;

θ

)

s, z ← ϕ with probability p drop ▷ randomly mask the latent factors with probability p drop t ∼ Uniform( { 1 , 2 , ..., T } ) , ϵ ∼ N ( 0 , I )

L

(

x, s, z

Update

;

ϵ

end for

Algorithm 3 Adversarially Robust Inference Algorithm

Require: A fully-trained causal model involves diffusion model ϵ θ ( x t , t, s, z ) , encoder f ( s,z ) ( x ; θ ) , and classifier f y ( s ; θ ) ; test image x , purification optimization steps N purify, causal factor inference steps N infer, number of sampling steps N t for inference,optimizer optim purify for purification, optimizer optim infer for causal factor inference.

∗

) end for Stage 2: Causal Factor Inference Initial s ∗ , z ∗ = f ( s,z ) ( x ∗ ; θ ) for iter=1 to N infer do Sample N t timesteps t at equal intervals from 1 to T Sample N t ϵ ∼ N ( 0 , I ) corresponds to each t Compute L ( x ∗ , s ∗ , z ∗ ; θ ) = E ϵ,t ∥ ϵ θ ( x ∗ t , t, s, z ) -ϵ ∥ 2 2 Update s ∗ , z ∗ by optim infer ( s ∗ , z ∗ ; L ( x ∗ , s ∗ , z ∗ ; θ )) end for Stage 3: S-based Classification )

generative model CAMA [43]. Notably, we also include the SOTA method on unseen attacks - Robust Diffusion Classifier (RDC), which incorporates purification and conditional generation based on labels for defense.

B.2 Implementation Details

We use DDPM [17] as our generative model. For the encoder f s,z ( x ; θ ) and the classification model f s ( y ; θ ) , we employ WideResNet-70-16 (WRN-70-16) as the backbone.

Training Strategy. Considering the different complexities in learning f x ( s, z ; θ ) , f ( s,z ) ( x ; θ ) and f y ( s ; θ ) , attributed to the differing difficulty in generation and discrimination task, we segment the training of the entire causal model into two distinct phases. As outlined in Section 4.2, we employ a two-stage training process for our CausalDiff.

θ

θ

) =

and

∥

f

ϵ

(

θ

(

s,z

)

x

t

(

, t, s, z

x

;

θ

)

by

)

-

ϵ

∥

2

2

optim(

θ,

L

(

x, s, z

;

θ

))

X

Y

V

V

S

Z

V

X

Y

X

Y

X

Y

(a)Discriminative

(b) Generative

(c) Causalwithout Disent.

(d) Causalwith Disent.

(a) Discriminative(b) Generative

(c) Causalwithout Disent.

(d) Causalwith Disent.

C

X

Forward Propagation

Enc

Enc

Enc

Enc

LCE

a

LRec

LRec

LCE

LRec

LCE

Cls

Dec

Dec

Cls

Dec

Cls

y

In the pretrain phase on CIFAR-10 and GTSRB datasets, focused on generation, primarily trains the conditional diffusion model f x ( s, z ; θ ) along with its corresponding encoder f ( s,z ) ( x ; θ ) according to Algorithm 2, setting N 1 = 1440 . For CIFAR-100, we added 10,000 images generated by EDM [49] to our training set. Considering the computational cost, we only pretrain for N 1 = 500 epochs. Note that the augmented data is used only during the pretraining phase while not in the joint training phase.

Subsequently, we conduct joint training of the whole CausalDiff model for N 2 = 560 , amounting to a total of 2000 epochs. The second phase, targeting discrimination and leveraging label information to guide disentanglement, involves the joint training of the entire causal model. Note that, in order to simultaneously train an unconditional diffusion model for adversarial purification, we follow Ho and Salimans [51] to mask the condition s and z with probability p drop = 0 . 1 . Thus, a single shared model is used for both adversarial purification and causal factor inference.

Both the pretraining and joint training phases utilize a learning rate of 1 e -4 and a batch size of 128. For simplicity, we follow the setting of w t = 1 [17]. We set α = 1 ., γ = 1 e -2 , η = 1 e -5 , λ = 1 e -2 as the weights for the loss function in Eq. (7).

Since we need a standard diffusion model ϵ θ ( x t , t ) for purification during adversarial inference, we apply dropout of s and z with a ratio of p drop = 0 . 1 for conditional diffusion generation as in Ho et al. [17] during both pretraining and training. Thus, the unconditional probability of generating x can also be estimated using the same model by masking s and z .

Inference Strategy. Leveraging the trained CausalDiff, we can infer its label from an adversarial example in accordance with Algorithm 3, following the inference pipeline outlined in Section 4.3. Specifically, we set N purify = 5 and use momentum-SGD as our optim purify , with a learning rate of

88

90

92

94

10

-5

10

-6

10

-7

10

-8

Weight

η

of  I

club

(s;z)

64

66

68

70

0

Clean Accuracy

Robust Accuracy

0.1. For causal factor inference, we choose N t = 10 to sample 10 timesteps per iteration and adopt N infer = 10 with momentum-SGD as optim infer , setting the learning rate to 1 e -5 .

Regarding white-box attacks, we perform a gradient backpropagation throughout the entire pipeline of our CausalDiff approach, which includes purification, causal factors inference, and classification. This implies that the attacker possesses sufficient knowledge of the causal model.

Attack Evaluation. Attack evaluation for CIFAR-10 dataset includes a 100-step StAdv attack [60] with ϵ = 0 . 05 under the ℓ ∞ norm bound, BPDA+EOT (EOT=20) against the ℓ ∞ threat model with ϵ = 8 / 255 , and AutoAttack [2] (AA), which comprises 100-step white-box attacks such as APGD-ce, APGD-t, FAB-t, and a 5000-step black-box Square Attack under both ℓ 2 ( ϵ = 0 . 5 ) and ℓ ∞ ( ϵ = 8 / 255 ) constraints.

For the GTSRB dataset, we utilize four types of attacks as well as two types of corruptions to evaluate robustness against adversarial attacks and the influence of the natural environment. Specifically, the attack methods include AutoAttack, which comprises 100-step white-box attacks such as APGD-ce, APGD-t, FAB-t, and a 5000-step black-box Square Attack under both ℓ 2 ( ϵ = 0 . 5 ) and ℓ ∞ ( ϵ = 8 / 255 ) constraints. The corruptions include Fog and Brightness. Following Nie et al. [33], we randomly sample 512 samples for evaluation.

B.3 Details for Pilot Study

Data Construction . For each data point of the 2000 samples, we sample a vector s with dimension h s = 8 from a normal distribution with mean -1 and variance 1 , i.e., s ∼ N ( -1 , 1) , and a vector z ∼ N (1 , 1) with dimension h z = 8 . Subsequently, we projected the concatenation of s and z , denoted as [ s ; z ] , to a sample x , with a random initialized matrix A x ( A x ∈ R ( h s + h z ) × h x ) , i.e., x = [ s ; z ] · A x . Similarly, we produced the score y s of x with y s = s · A y , where A y ∈ R h s × 1 . To obtain samples with balanced binary labels, we consider the label y of the sample with y s above the median as 1 and the others as 0.

Methods for Comparisons . In the pilot study detailed in Section 3, we conducted an investigation and analysis on four models: 1) Discriminative: a discriminative model that learns to classify the samples with a two-layer perceptron (MLP), 2) Generative [27]: a generative model that learns the generation of the sample x conditioning on its label y and predict the label of an adversarial example by calculating the max y p ( x | y ) , 3) Causal without Disent.: a causal model that models the generation of both x and y with the same causal factor v (Causal modeling without Disentanglement), 4) Causal with Disent.: our model that disentangles the label-causative factor s and another factor z during the generation of x . For the latter two causal models, given an adversarial example, the hidden vectors v or s, z are inferred for Causal without Disent. and with Disent. which are then used for the final label prediction. This section comprehensively presents the designed causal structures in Fig. 5 and the model architectures in Fig. 6.

Regarding implementation, we trained each of the four models for 20 epochs, optimizing the model parameters using the Adam optimizer with a learning rate of 1 e -3 . The latent dimension for each model was set to 64. For evaluation, we employed a 100-step PGD attack with ϵ = 0 . 3 and α = 2 / 255 within the ℓ ∞ norm boundary. The variation in latent variables and predicted logits between adversarial examples and clean images, as presented in Table 4, is measured on adversarial examples generated with ϵ = 10 and α = 0 . 05 within the ℓ ∞ norm boundary, using a 100-step PGD attack.

C More Experimental Results

C.1 Analyses on Core Components of Training

Effect of I θ CLUB ( S ; Z ) in Casual Information Bottleneck (CIB) To examine the impact of our introduced disentanglement term I θ CLUB ( S ; Z ) in CIB (See Equation (6)), we vary η in Equation (7) and evaluate the robustness against the most challenging attack in AutoAttack [2], i.e., with ℓ ∞ norm bounded by ϵ = 8 / 255 . Larger η will cause the diffusion model to collapse, so we do not include the results of larger η . As we mentioned in Section 4.2, our CIB regresses to the ELBO objective in [14] when η = 0 . As shown in Fig. 7, CausalDiff has better clean accuracy as well as robustness when η > 0 and yields the best robustness when η = 10 -5 (69.14% compared to 65.04% when η = 0 ). It confirms that I θ CLUB ( S ; Z ) in the loss function is beneficial to disentangle the Y-causative from Y-non-causative factors and can further enhance both clean accuracy and robustness.

C.2 Analyses on Core Components of Inference

Causal Factor Inference Method . We also evaluate the robustness using the encoder f ( s,z ) ( x ; θ ) to get latent variables instead of inferring s and z through the conditional diffusion model. The results reveal that classification by the encoder (without purification) achieves a clean accuracy of 91.99% but 0.00% against AutoAttack under both ℓ ∞ and ℓ 2 threat models. This decline might be attributed to imprecise modeling around x (i.e., x + δ ), which results in an inability to resist adversarial perturbations on x .

Timestep t Sampling Strategies for Purification As discussed in Section 5.2, our purification method ( CausalDiffw/o Causal Factor Inference in Table 2) markedly surpasses the direct adaptation of likelihood maximization ( LM-DDPM in Table 2), as proposed by Chen et al. [27], applied to DDPM. This improvement stems from a refined strategy in sampling the timestep t .

As demonstrated in Fig. 9, we found that smaller timesteps perform better in distinguishing between the distributions of clean and adversarial samples. Specifically, we presented the negative loglikelihood estimated by expectation E ϵ [ w t ∥ ϵ θ ( x t , t ) -ϵ ∥ ] for the given timestep over 512 examples. This may caused by a larger timestep implies a greater degree of noise addition for estimating likelihood, which might overshadow the adversarial perturbations, unexpected for purification.

C.3 Visualization of Cases

We visualize the generated images leveraging the conditional generation of our CausalDiff, providing an intuitive depiction of the label-causative factor s ∗ and the label-non-causative factor z ∗ . Fig. 8 illustrates that after inferring from the benign example x ∗ , perturbations are alleviated. The image x s ∗ conditioned on s ∗ showcases that s ∗ captures the core predictive features, reflecting the general concept of the category (for instance, a common semantic representation of 'horse' from an image of a brown horse's head), or even enhances the predictive features such as the dog's face or the ship's body, whereas z ∗ retains specific and non-predictive image details.

We also visualize the purified example x ∗ , conditionally generated x ∗ s ∗ and x ∗ z ∗ when encountering an adversarial example ˜ x . These cases demonstrate that, despite the presence of perturbations in the clean images, our CausalDiff effectively captures the correct predictive information of s ∗ , maintaining alignment with the clean data.

KXX

C s*

z*

*

*

米

[;2*]

C

C

0.004

0.002 0.000

Density

random t

1.0

0.5

t=1

clean

adv.

0.3

0.2

0.1

0.0

t=50

0.004

0.002

0.000

t=200

0.015

0.010

0.0050.000

t=500

0.00004

0.00002

t=1000

C.4 Speed Test of Inference Time

We evaluate the computational complexity of CausalDiff and DiffPure [33] as well as a discriminative model (WRN-70-16) by measuring the inference time in seconds for a single sample (average on 100 examples from CIFAR-10 dataset) on two types of GPUs, including NVIDIA A6000 GPU and 4090 GPU (Our experiments leverage 4 A6000 GPUs and 4 4090 GPUs). The results are shown in Table 5.

D Limitation

Although our CausalDiff significantly narrows the gap in classification accuracy between adversarial and clean examples, it requires an inference cost of 1 + N 1 + N 2 NFEs (Number of Function Evaluations), where efficiency improvements are needed. Note that N 1 indicates the purification step (e.g. 5) and N 2 indicates the step of causal factor inference (e.g., 10) and 1 NFE for latent-S-based classification. Furthermore, our CausalDiff represents a new framework, meaning it requires training from scratch. Perhaps in the future, an efficient implementation of robust inference could be achieved by embedding causal mechanisms into the existing models in a plug-and-play manner.

E Broader Impact

Our CausalDiff model, built upon a powerful generative framework, aims to align with human decision-making mechanisms to enhance the stability and trustworthiness of neural networks. This approach holds potential for advancing the field of Machine Learning, particularly in safety-sensitive applications such as autonomous driving and facial recognition.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: see abstract

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Appendix D

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Section 4.2 and Appendix A

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Section 5, Appendix C and Appendix B

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We described our experimental details in Appendix B and we will open source our code in camera-ready version.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Appendix B

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Adversarial attack evaluation incurs high computational costs.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Appendix C.4

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Appendix E

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Appendix E

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Appendix E

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Appendix 4

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: NA

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: NA

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: NA

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an adversarial defense method, CausalDiff, which leverages a causal model of robustness combined with a diffusion model to learn label-relevant causative representations. The effectiveness of CausalDiff is specifically demonstrated against unseen attacks, where it surpasses existing adversarial defense baselines on several image classification datasets.\n\nSTRENGTHS:\n- This paper is well-motivated, presenting a novel framework that extracts the disentangled essential features of an image, achieved by first training a diffusion model and then inferring from it.\n- The pilot study is easy to understand, and the results persuasively demonstrate the effectiveness of the proposed causality-inspired defense framework.\n- A comprehensive comparison with baselines of adversarial defense methods, especially those based on Diffusion Models, makes the results convincing.\n\nWEAKNESSES:\n- Limited discussion about related works. To my knowledgement, neither CausalAdv and DICE limits their modeling of adversarial distribution to a certain type of attack. Therefore, it is necessary to discuss how the proposed method differs from these important baselines in terms of causal modeling. \n- The writing can be improved. The challenges listed from line 53 to 66 could be clearer by focusing on highlighting the actual problems that are solved by the proposed method, rather than just listing how the method is implemented. For example, challenge (2) could be clarified by explaining that applying a *Causal Information Bottleneck* objective aims to minimize the essential factors to their minimal necessary extent.\n- The empirical results of the pilot study are convincing. However, the insight into how to extract the desired features (i.e., S, Z) appears unclear, as detailed in Q2-Q3.\n\nQUESTIONS:\nQ1: Could you provide further insight into the source of CausalDiff’s ability to generalize to unseen attacks? For example, what attributes of the *essential factor* for benign images enable its generalizability to unseen attacks, and are there any specific constraints regarding the types of unseen attacks?\n\nQ2: Could the author further explain the reason for the opposite optimization direction $(1 - \\lambda) I(X;S,Z)$ in Eq. 5? E.g., what is the relationship between the motivation of avoiding \"X containing too many unimportant details\" (line 196) and the \"insensitivity\" for $-\\lambda I(X;S,Z)$ in Fig. 2? These statements seem to imply the existence of a feature that is neither \"essential\" (i.e., S) nor \"non-essential but important for reconstructing X\" (i.e., Z). This kind of feature appears crucial for \"sensitivity\" but is not fully discussed in the paper.\n\nQ3: Any abation study on the parameter $\\lambda$ used in Eq.5, which typically plays a crucial role in IB-style objectives?\n\nQ4: Other minor typos. \n\n- There is a citation format error in line 21, and a method name mistake (with / without) in line 124.\n\nLIMITATIONS:\n- The defense performance is superior against adversarial attacks with  L_p  norm constraints. However, the effectiveness of the proposed method against unbounded attacks (e.g., semantic attacks[1]) remains unclear.\n- Considering the high computational cost and training budget associated with diffusion models tailored to a single image classifier, CausalDiff may lack practicality. Strategies to reduce these costs should be considered. For instance, could a smaller diffusion model be trained to replace the two different diffusion models (unconditional and conditional) through model distillation?\n\n[1] Ghiasi, Amin, Ali Shafahi, and Tom Goldstein. \"BREAKING CERTIFIED DEFENSES: SEMANTIC ADVERSARIAL EXAMPLES WITH SPOOFED ROBUSTNESS CERTIFICATES.\" *International Conference on Learning Representations*.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to promote the trustworthiness of DNNs through purification. To this end, the authors propose a novel causality view to perform a disentanglement approach using diffusion models. Some experiments are conducted to verify the effectiveness of the proposed method.\n\nSTRENGTHS:\n+ This work takes a good step towards practical adversarial defense. For instance, adversarial attack behaviors are unpredictable, leading to the requirement for robustness against unseen attacks. In this context, the authors highlight the challenge and propose a novel approach to address the challenge.\n\n+ The experiment designed on the toy dataset is interesting and provides clear points to depict the motivation of this work. Namely, disentanglement is a promising direction to promote the robustness of DNNs. Moreover, this aligns well with Eq. (3), especially the third term on the right. \n\n+ The experimental results are decent, where the proposed method achieves exciting robustness under various settings. Moreover, the proposed method is evaluated using a SOTA adversarial attack method, i.e., AutoAttack. This makes the results convincing.\n\nWEAKNESSES:\n- Adaptive attacks are lacking in this work, which significantly weakens its contribution and convincingness. Specifically, in the context of the considered white-box attack, what if the adversary generates adversarial examples using the following objective function? \n$\\max_{\\delta} \\ell_{ce}(x+\\delta, y) + \\log p(x+\\delta) + \\log p(x+\\delta|s^+,z)$ with $s^+ = \\arg \\min \\log p(y|s^+)$\n\n- It is hard to accept the conclusion shown in Figure 2. Specifically, DNNs would exhibit vulnerability when increasing the $\\epsilon$-budget. However, Figure 2 shows that the proposed method always makes DNNs robust. It is unclear what would happen if we further improve the budget. The results would be solid evidence for the false robustness if it is not intuitive.\n\n- Adversarial training with specific adversarial examples endows DNNs with robustness against these adversarial examples. This is intuitive. However, the current version of this work lacks a detailed and explicit explanation for the intuition or mechanism of why the proposed method can endow DNNs with robustness against various types of adversarial attacks. Relax; this is just a suggestion to make the work more solid. \n\n- It is known that diffusion models are sensitive to the inference step related to the inference time. Thus, the proposed method would introduce more cost in time. However, the authors seem to overlook the cost.\n\nMinor:\nThe second and third paragraphs of the Introduction lack references. For instance, the authors should provide corresponding works discussing certified defense, adversarial training, purification methods, and the causality and disentangle view.\n\n- The authors claim that CausalDiff can be viewed as semantic-level purification. This shows a relation to a previous work [1]. Specifically, performing a (non-semantic) subspace exploration would endow DNNs with robustness to distribution shift [1]. Thus, if we can abstract s* and z* using CausalDiff, we can further perform a distribution exploration to promote the robustness for OOD generalization. \n\n[1] Moderately Distributional Exploration for Domain Generalization. Dai et al.\n\nQUESTIONS:\nPlease see the weakness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
This paper proposed a novel causal diffusion framework based on causal inference to defend against unseen attacks. The causal information bottleneck is interesting to disentangle the target-causative and target-non-causative factors, and then target-causative factors are used for adversarial defense.

STRENGTHS:
The originality of this paper is good, since the causal diffusion framework and causal information bottleneck are innovative and effective for adversarial defense. The quality and clarity of this paper are also good, where the logic is clear for understanding. The significance of this paper is obvious, because unseen attacks are necessary to address.

WEAKNESSES:
1. Some technical details need to be explained. For example, what is the actual meaning of the constraint in Eq. (4)? And why are the positions of z_s and s_b defined like in Eq. (2)?
 
2. Numerically, the authors could consider comparing their method with more baselines. There are some studies on adversarial defense, even without using diffusion models.

QUESTIONS:
1. Some technical details need to be explained. For example, what is the actual meaning of the constraint in Eq. (4)? And why are the positions of z_s and s_b defined like in Eq. (2)?
 
2. Numerically, the authors could consider comparing their method with more baselines. There are some studies on adversarial defense, even without using diffusion models.

LIMITATIONS:
1. What is the technical drawback of the proposed method? E.g., defense efficiency
2. Does this proposed method work for data other than images?
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-11

comments: SUMMARY:
The authors propose CausalDiff, a causality-inspired disentanglement approach using diffusion models for adversarial defense, which outperforms state-of-the-art methods on various unseen attacks across multiple datasets.

STRENGTHS:
Novel approach combining causal modeling and diffusion models for adversarial defense

Strong performance against unseen attacks on multiple datasets

Thorough pilot study on toy data to validate the approach

Clear theoretical foundation with the proposed Causal Information Bottleneck objective

Practical adaptation of diffusion models for conditional generation

WEAKNESSES:
Limited discussion on computational complexity and training time

Lack of comparison with other causal approaches for adversarial robustness

No analysis of the method's performance on more complex datasets (e.g., ImageNet)

Limited exploration of the interpretability of the learned causal factors

Absence of ablation studies to isolate the impact of individual components

QUESTIONS:
Please see the weaknesses section.

LIMITATIONS:
The authors adequately addressed the limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12