CausalDiff: Causality-Inspired Disentanglement via Diffusion Model for Adversarial Defense
Authors: Mingkun Zhang
Keping Bi
Wei Chen
Quanrun Chen
Jiafeng Guo
Xueqi Cheng
Venue: NeurIPS 2024 poster (2024)
Keywords: Adversarial Defense
Diffusion Model
Causal
Primary area: safety_in_machine_learning
TL;DR: We propose a casual diffusion model (CausalDiff) that adapts diffusion models for conditional data generation and disentangles the two types of casual factors for adversarial defense on image classification task.

Abstract: Despite ongoing efforts to defend neural classifiers from adversarial attacks, they remain vulnerable, especially to unseen attacks. In contrast, humans are difficult to be cheated by subtle manipulations, since we make judgments only based on essential factors. Inspired by this observation, we attempt to model label generation with essential label-causative factors and incorporate label-non-causative factors to assist data generation. For an adversarial example, we aim to discriminate the perturbations as non-causative factors and make predictions only based on the label-causative factors. Concretely, we propose a casual diffusion model (CausalDiff) that adapts diffusion models for conditional data generation and disentangles the two types of casual factors by learning towards a novel casual information bottleneck objective. Empirically, CausalDiff has significantly outperformed state-of-the-art defense methods on various unseen attacks, achieving an average robustness of 86.39\% (+4.01\%) on CIFAR-10, 56.25\% (+3.13\%) on CIFAR-100, and 82.62\% (+4.93\%) on GTSRB (German Traffic Sign Recognition Benchmark). The code is available athttps://github.com/CAS-AISafetyBasicResearchGroup/CausalDiff.

CausalDiff: Causality-Inppired Disentanglement via Diffusion Model for Adversarial Defense
Mingkun Zhang
Keping Bi
CAS Key Laboratory of AI Safety Institute of Computing Technology. CAS zhangmingkun22001cct ac.cn
Key Laboratory of Network Data Science and Technology Institute of Computing Technology, CAS
bikeping@ict.acc an.cn
Wei Chen
Quanrun Chen
CAS Key Laboratory of AI Safety Institute of Computing Technology, CAS chenve12022@ict. ac.
School of Statistics University of International Business and Economics qchen@uibe edu. cn
Jiafeng Guo
Xueqi Cheng
Key Laboratory of Network Data Science and Technology Institute of Computing Technology, CAS guojiafeng@ictt.cccom
CAS Key Laboratory of Al Safety Institute of Computing Technology, CAS cxq@ict.ac. ac.cn
Abstract
Despite ongoing efforts to defend neural classifiers from adversarial attacks, they remain vulnerable, especially unseen attacks. In contrast, humans difficult to be cheated by subtle manipulations, since we make judgments only based on essential factors. Inspired by this observation, we attempt to model label generation with essential label-causative factors and incorporate label-non-causatiyy factors assist data generation. For an adversarial example, we aim discriminate the perturbations as non causative factors and make predictions only based on the label- causative factors. Concretely, we propose casual diffusion model (CausalDiff) that adapts diffusion models for conditional data generation and disentangles the two types of casual factors by learning towards a novel casual information bottleneck objective. Empirically, CausalDiff has significantly outperformed state-of-the-an defense methods on various unseen attacks, achieving an average robustness of 86.39% (+4.01%) on CIFAR-10, 10, 56.25% (+3.13%) on CIFAR- 100, and 82.62% (+4.93%) on GTSRB (German Traffic Sign Recognition Benchmark). The code available attt:///ddiiiooocSSSAaaaaassssss
Introduction
Neural classifiers, despite their impressive performance various applications, are susceptible adversarial attacks 21] which can deceive them into making erroneous judgments on subtly manipulated examples. Such vulnerabilities pose severe threats in safety-critical scenarios such as face recognition 4 and autonomous driving [5]. There has been extensive work on defending
against adversarial attacks such as certified defenses 6 7 adversarial training g 9 and adversarial
purification Samangouei et [0]]
"Corresponding Author. CAS stands for Chinese Academy of Sciences
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Causal relation
Conditional generation
Causal Inference
Forward diffission
CausalDiff
Training
Inference
SCM
S
Conditional Diffinous
Slassd Clasiligtion
Po (,.,
Fretor
'bone'
Prodiction
Insenstivvty
Causal Infommation Bottleneck::
Adversria Purtficaiin
CIB Y. 2) U,;SS, S, [S) 18: AI
Reconsmuction
Disentanglement
Labsl-cnaative
Labsloan-smmniie
Label
Nokey sample
example
Adveneriaal
Vistin model Parametrr
Figure I: Illustration of training (Left) and inference (Right) processes of our proposed Cifs model. During training, the model constructs a structural causal model leveraging a conditional diffusion model, disentangling the (label) Y-causative feature S and the Y-non-causative feature Z through maximization the Causal Information Bottleneck (CIB). In the inference stage, CausalDiff first purifies an adversarial example X, yielded by perturbing X according to the target victim model parameterized by 0, to obtain the benign counterpart X Then, it infers the Y-causative feature S for label prediction. We visualize the vectors of S and Z inferred from perturbed image of a horse using diffusion model. We find that S captures the general concept of horse, even when the input image only shows the head, while Z carries information about the horse's 's skin color.
Although effective, these methods have some limitations. Certified defense methods have limited practicality due to the small certified region that can theoretically guarantee robustness. Adversarial training approaches suffer from significant decline in robustness against unseen attacks since they take effect by adding adversarial examples into the training set. Purification methods, not designed for specific attacks, struggle determine the optimal denoising level for unforeseen attacks with differing degrees of perturbation. Consequently, they face challenges in effectively defending against unseen attacks.
In reality, attack behaviors are often unpredictable. Is there a way of strengthening a model to act like humans, i.e., be insensitive to subtle perturbations and robust against various unforeseen attacks? Given an image of an object, we typically identify the key visual features that are necessary to determine its category and disregard other factors such styles, backgrounds, details, perturbations. This allows us to make robust judgments. Inspired by this human decision- meeiiion-mmking process, we would like learn model that can disentangle the essential features for determining the category from other non-essential ones.
It is natural to treat the essential features the causal factors of the label, and both essential and the other features as the causal factors of the entire image. Then, we can learn to disentangle them by modeling the process of data generation and label prediction with structural causal model (SCM) 11 (shown in Fig. I (Left)). According to the theoretical results provided by Liu et al. [3]. the be guaranteed under mild conditions. Although similar SCMs have also been employed in modeling the generation of multi-domaie data [14] and adversarial examples [6], they either aim to enhance cittof-ddistriuttio robustness protect the model from certain type of attack. In contrast, our research focuses on modeling the generation of native in-domain data to enhance adversarial robustness against various unseen attacks.
identifiability of such SCMs
Fig. [ (Left) depicts our SCM, where Y denotes the category (c.g., horse) of an input image X; S denotes the essential semantic features of determining Y (i.c., Y -cauusative factors), such as the characteristics eyes, ears, nose, mouth, etc. of horses; and Z represents the other features (i.c., Y Y-non-causative factors) that are not needed to predict Y but are important to generate X, such as the fur color and the image background. Given an adversarial example produced by an unknown attack, our model aims to disentangle its -ccuusaatvve features S from the spurious factors in Z and make a robust prediction. To successfully learn the disentanglement, our SCM is guided by the tasks of data generation and label prediction, where there three major challenges:
1) How do we model the conditional generation of given S and effectively? To this end, we employ well-recognieed diffusion model with sate-of-the-ar (SOTA) generative performance and

---

efficiency, i.e., the Denoising Diffusion Probabilistic Model (DDPM) [T7. as the backbone. We further adapt it for conditional generation from latent variables rather than random noise. 2) What training objective should we use to effectively learn the disentanglement of S and Z? With this regard, we propose a Causal Information Bottleneck (CIB) optimization objective. CTB aligns the information in the latent variables (S. Z) with observed variables (X, Y) with a bottleneck set by the mutual information (MI) between S, Z and X. The derived function will minimize the MI between S and Z while learning the other causal relations, ensuring their disentanglement within the causal framework. 3) Given an adversarial example, what inference strategies shall we adopt to make robust predictions? As shown in Fig. (Right), according to how the adversarial example X is generated. first purify it to yield benign example X and then infer the Y-causative factor S based on X for final classification. We name the entire causal defense framework based on diffusion models as CausalDiff.
The experimental results on facing various unseen attacks, encompassing both black-box and white- box ones, show that CausalDiff has superior performance compared to representative adversarial Table 1: The experimental results of four models on toy data. The variation of latent and logits p(yle) is measured between clean and adversarial examples. The model margin is estimated by the minimal adversarial perturbation required to flip the label, employing both &g and fo norm.
Adversarial Robustness
1.0
Discriminative
Causalattouu Disnt.
Generative
Causalooth Disent.
0.1
margin f (E2 loo)
Model
& v. 4 & p(ule) +
E- Buddgt
Discriminative
1.15
0.81
2.14 0.38
Generative
0.06
0.27
1.12 / 0.24
Figure 2: Adversarial robustness of four mod- els against 100-step PGD attack under varying attack strength indicated by e-budget.
Causalwao Disent.
0.29
0.32
9.58 / 5.30
Causal Disent.
0.27
0.22
10.64 / 6.28
defense baselines including of-te--f-hh-art (SOTA) methods. Specifically, our Ciff achieves robustness of 86.39% (+4.01%) () CIFAR 10, 56.25% (+3.13%) on CIFAR-100 [8] and 81.79% (44.93%) on GTSRB 19]
In summary, we highlight our contributions as follows: 1) We propose a novel causal diffusion framework (CausalDiff) to defend against unseen attacks by modeling the generation of native in- domain data with the (rregory(Y)ccaussee
factors and the other Y-non-causal factors; 2) We propose a Causal Information Bottleneck (CIB) objective to disentangle Y-causative from Y-non-causative factors during causal model training and an associated inference algorithm for adversarial defense; 3) CifsalDiff significantly outperforms SOTA methods in defending against various unseen attacks.
2 Related Work
Adversarial Defense. Adversarial training primarily focuses on optimizing the training algorithm
20 21I. incorporating data augmentation [22 23 [9. and enhancing acceleration 24
25). Despite
its effectiveness, the trained models could still be vulnerable to unseen attacks [26 27 Adver-
sarial purification, orthogonal to our work, utilizes generative model to purify adversarial noise from examples before classification. Leveraging diffusion models 28 purification have shown to be effective 35 27).
diffusion-based
Causal Learning for Robustness. Causal representation learning
36 14 37 focuses on discov- 12
ering invariant mechanisms within structural causal models and has achieved remarkable performance in improving model transferability [4]
40) and interpretability [4] 42 Moreover, in terms 46 48) have attempted to model the attack behaviors in causal structures to identify the adversarial factors. However, modeling particular attack
[3
38
of adversarial robustness, researchers 43 15
44
45
16
types will limit the model robustness on other types of attacks (27.
Conditional Diffusion Model. Diffusion models image generation capabilities. To equip them with the ability of controllable generation, the sampling
28
29 49 have achieved compelling
3

---

process can be guided by 1) classifier confidence to generate images of a certain category 30 50 51
and 2) semantic embedding of text content certain style [22 55
3 Pilot Study Toy Data
Discriminative
(b)
Causalittouut Disent
(d) Causalaith Disent.
Figure 3: Visualizations of feature space for the two categories on toy data by T-SNE for (a) discriminative model, (b) generative model, (c) causal model without disentanglement, and (d) causal model with disentanglement.
To compare our proposed causal model with other representative models in terms of adversarial robustness, we constructed toy dataset according to the hypothesis that the essential factors are the basis of generating labels and they also determine the generation of data together with label-non- causative features. Pilot studies on this toy data will provide insights into whether our causal model will work and why it works.
3.1 Experimental Settings
Toy Data Construction. We constructed 2000 samples following the causal structure in Fig. (Left). Specifically, for each data point, we randomly sample the vectors and from two different normal distributions, project a score Y/ with random weights, obtain its label by comparing ys with the medium score, and generate the representation æ by projecting the concatenation [s; with another random matrix. Please refer to Appendix B.3 for detailed information.
Models for Comparisons. We investigated four representative models: 1) Discriminative model for classification, 2) Generative model that predicts the label of an adversarial example with max, p( (27]. 3) causal model without disentanglement (Causal Disent.) that models the generation of both and y with the same latent factor v, 4) our causal model with disentanglement (Causal Disent.) that is illustrated in Fig. The concrete structures of the four models and further details are presented in Appendix B.3
3.2 Experimental Observations
Adversarial Robustness. We evaluate the model's robustness against a 100-step PGD attack with varying e-budgets under the foo bound [8] Model performance in terms of both clean accuracy (when = 0) and robust accuracy is examined.
As shown in Fig- The Discriminative model exhibits the highest clean accuracy but suffers rapid decline in robustness, dropping to 0% when the attack budget reaches 1.2., which highlights its vulnerability. 2) The Generative model has the lowest clean accuracy while its robustness does not dramatically regress with larger attack budgets. The low clean accuracy may be due to the inconsistency between the generation process modeled and the way this toy dataset constructed. The small gap between clean and robust accuracy indicates its decent effectiveness in defending against adversarial attacks. 3) Causal with Disent. obtains the second- best clean accuracy, yet its robustness gradually declines with increasing attack strength, maintaining 61.4% robustness at e = 6.4. 4) Our model, Causalwith Disent., has slightly lower clean accuracy than Causal without Disent. but the best robustness among the four. As the attack strength increases, it maintains at least 71. 71.8% robust accuracy = 6.4, indicating that it is promising enhance model robustness by causal modeling with disentanglement of the label-causative factors from the non-causative ones.

---

Sensitivity to Perturbations. To delve further into how the model behavior changes when defending against adversarial perturbations, we measure the variation between latent variables of clean and adversarial examples, denoted as & = cosine (t, Vadv), where v and Vadv represents the latent vector of clean and adversarial example, respectively (v 8 for Causalwith Disent.). We also compute the change of predicted logits, & p(ylo)) = p(ulo) p(yl Vadv), with y being the true class label. Note that a larger variation in the latent factor for prediction, or the predicted logits, results in increased insensitivity to perturbations.
The experimental results, as shown in Tab. indicate that Causalusth Disent., compared to withol Disent., exhibits less sensitivity in both latent variables and prediction outcomes. This suggests that by disentangling the label-causative factor s, it becomes more challenging for attackers to perturb the model and alter its prediction. The small variation of v in the Generative model probably attributed to the lack of discriminability in v.
To intuitively explain the sensitivity of each model to perturbations, we estimate the minimal adversarial perturbation ||ö]| under PGD required to flip the correct model prediction of sample r. It can be interpreted as the prediction margins in terms of perturbation, denoted as margin y) min ||ō]], subject to p(yl jr + 5) p(g|æ + 6) [56.. We measure the ||ôl| under lg and & norms. Additionally, we visualized the latent vector space of each model to intuitively observe the margin of the classification boundary.
Prediction Margins.
As shown in Tab. Causalwask Dieent. has the largest prediction margin. This implics that an attacker would need to add significantly more perturbation to successfully cheat our model. We can draw similar conclusions from Fig. which illustrates the distribution of predictive features extracted by each model for correctly classified clean samples across categories. Again, Causal Disent. has the largest margin between the classes, indicating that it has high confidence in its prediction and increases the cost and difficulty for an attacker succeed.
4 Causal Diffusion Model
To enhance model robustness on real-world data, in this section, we propose a Causal Diffusion (CausalDiff) Modeling approach, that couples our previously studied SCM (shown in Fig. o(d) with diffusion models. We will introduce the three major components in CausalDiff: conditional diffusion generation, causal information bottleneck optimization, and adversarial example inference. We take the Denoising Diffusion Probabilistic Model (DDPM) [17 as instance for illustration and it can be easily adapted to other diffusion models.
4.1 Conditional Diffusion Generation
Standard diffusion models generate images based on random noise which do not apply to the conditional generation of X based on S and Z in our SCM. In standard diffusion models, each time step during denoising, UNet ea (Zr, () is employed to decode an image given input 2t. While diffusion models are highly effective in image generation, they lack an explicit decoder component for generating images from latent variables. To handle this, we develop a conditional DDPM using latent variables S and Z controlling the generation process. This approach draws inspiration from both the class-conditional diffusion model [51] as well as the style control mechanism introduced in DiffAE (52) The output hout at each layer of the UNet depends on and Te. i.c..
hout ta GroupNorm(h) + tb. where h is the feature map of xt, to and to are the scale and bias of timestep t.
(1)
Inspired by the classcooniiional diffusion model [i and the style control mechanism in DifTAE [52], we adapt the standard diffusion generation to be conditioned on S and Z in addition to timestep t. Specifically, the UNet becomes co(Ie, t, 8, 2),, the final output f.e,* is calculated based on the original M the hidden state and (representing causal factor S and Z) encoded from input x::
h-is. 2/ N + Sh.
where 2, and 84 are produced from the affine projections of = and respectively, i.e., 2, = Affine, and = A Af)nn, (s). As such, the label-causative factor S acts as a bias that can affect the direction of the latent vector and change its semanties, while the label-non-causate factor Z can only scale the latent vector in a similar way to style control.
5

---

4.2 Causal Information Bottleneck Optimization
To learn the causal factors S, Z our SCM and disentangle them, we propose a Causal Information Bottleneck (CIB) optimization objective. It maximizes the mutual information between the latent factors S. Z and the observed data sample (X, Y) with an information bottleneck that constrains the information retained in S, Z with respect to X.
Specifically, to align the information captured in the latent factors with the observed data (X, Y), we maximize the mutual information between them, denoted as I Y. Z), which can be derived as:
IX Y, 2) = I(X; 2) + +(Y I(S: Z) I(X;Y)
Among the resultant terms, I(X is solely dependent on the observed data, independent of latent variables or the causal model, and thus can be ignored in the learning process. Maximizing I,) will urge S and Z to capture ample information about X. I(S; Y) indicates that the Y- -ccausative factor should be correlated with Y. The term, [(; 7) ensures S and Z to be effectively disentangled.
A detailed proof is presented in Appendix A.1
Existing work on optimizing similar SCMs adapts the Evidence Lower BOund (ELBO) from Vari- ational Autoencoders (VAE) to formulate the causal ELBO objectives (14). This objective only maximizes the likelihood of (X, Y), and does not consider the latent factors. Consequently, the final optimization goal of causal ELBO differs from [(X, Y; S, Z) in that it does not have - I(S; Z) in Eq. (3, which is crucial for disentanglement. Further details are discussed in Appendix A.4
To avoid X contain too many unimportant details, we constrain the mutual information between X and the latent factors S, Z with an information bottleneck Le. Then, the updated objective becomes:
max I: Z), s. S, Z) L.
Employing Lagrange multiplier À A 0, we formulate our objective as max LXX,Y -X,)) S, Z) -.). Since Le a constant, it is equal to maximize the Causal Information Bottleneck (CIB);
CIB(X,Y,
I(X; Z) + I(Y;S) - I( S,(
I(; Z) and (X; Z) indicate two opposing optimization directions. Because it is unclear whether (1 A) should be positive or negative, we approximate these terms via two separate lower bounds instead of combining them.
To maximize the Causal Information Bottleneck (CIB) in Eq. (5), we derive its lower bound as the concrete training loss function. When using the diffusion model ca. classifier (s; f) (to estimate po(y)s)), and the encoder fa.= 0) (to estimate po (8, 2|x;) ), the lower bound of CIB is:
E px [log po (x/s, =)] Eppy,.)) Pl (yl s)] TeLUR (S; Z) p()) [DKL (po =lxr)ll((ss
(6)
where Po (s|2) is a variational distribution estimate p())2) and TCT.UB(S Z) Ept.. [log po (s|22)
Eptz) Ep(e) [log (s]=)) represents the Contrastive Log-Ratio Upper Bound (CLUB) of mutual
information proposed by Cheng et al. [57]. (z(2s,, represents the likelihood estimated by the conditional diffusion model. DKLL refers to the Kullback-Leibbe (KL) divergence (58); and g(-) denotes the prior distribution of the latent variable, e.g., a normal distribution (O, I). A detailed proof can be found in Appendix A.2]
Loss Function. Thus, maximizing the lower bound of CIB is equal to minimizing the loss function:
(12.y,8,230) = aE,, t, s, - C
+ I (S; Z) + XDKL 2|x)
where a, A, 77 determine the weighting of each term in the optimization process. A detailed derivation can be found in Appendix A.3
Algorithm. We pretrain the model with the data reconstruction loss (the first term in Eq. alone before training the model with the entire loss, so that the model can learn the causal factors, disentanglement, and classification from decent starting point. For space concern, we illustrate the training process in Appendix B.2 involves the training algorithm (Algorithm) I and the pretraining algorithm (Algorithm 2

---

Table 2:
Clean accuracy and adversarial robustness CIFAR-10 against StAdv under &oc (c 0.05) norm bound and AutoAttack (AA) under l2 (c = 0.5) as well as &oo (c = 8/255) bound. We calculate the average robustness across three attack methods to evaluate the model's robustness against unseen attacks. We use underlining to highlight the best robustness for each attack method within each defense category, and bold font denote the stafe-of-the-art (SOTA) across all methods.
CLEAN ACC (%)
ROBUST ACC(%) STADV
METHOD
BACKBONE
AA lac
AA E2
AVG
23
DDPM
88.87
63.28
64.65
4.88
44.27
AT-12
23
DDPM
93.16
49.41
81.05
5.27
45.24
9
EDM
93.36
70.90
69.73
2.93
47.85
ADV. TRAIN
AT-I2
EDM
95.90
53.32
84.77
5.08
47.72
CAUSALADV-T 15
WRN- 72--76-11 10
83.71
8.76
21.95
75.60
35.44
CAUSALADV-M
15
WRN-76-10
70.22
24.36
49.10
48.60
40.69
DICE [66]
WRN-34-10
82.85
37.51
41.58
82.46
53,85
DIFFPURE
R3
Score SDE
87.50
53.12
75.59
12.89
47.20
PURIFY
LM-DDDM
DDPM
80.47
53.32
63.09
74.22
63.54
LM-EDM
EDM
87.89
71.68
75.00
87.50
78.06
SBGC [59
SCORE SDE
95.04
0.00
0.00
0.00
0.00
OTHERS
CAMA 43
VAE
32.19
3.38
5.53
27.54
12.15
RDC 27
EDM
89.85
75.67
82.03
89.45
82.38
CAUSALDIFF
DDPM
90.23
83.01
86.33
89.84
86.39
OURS
CAUSAL
DDPM
83.20
74.61
75.59
82.23
77.48
CAUSALDIFF AP
DDPM
91.21
69.14
84.96
91.21
81.77
4.3 Adversarial Example Inference
Guided by the causal generation of an adversarial example X according to Fig. 1 (Right), we illustrate the process for robust classification. Following typical attack paradigm, X produced by adding target clean example when attacking a model 6. To make a robust prediction on X, our robust inference process comprises three steps: 1) purifying X to benign X by the unconditional diffusion model ca (zz, t). 2) inferring S and Z from X utilizing the causal model (1 f, 8, 2), and 3) predicting Y based on using classifier (s 0).
an adversarial perturbation
Adversarial Purification (AP). We follow the concept of Likelihood Maximization (LM) 33 to purify the adversarial example X to a benign X* by maximizing the data log-likelihood log Po (æ):
x=
= arg max, log Po (T.).
(8)
Concretely, we maximize its lower bound utilizing the unconditional diffusion model c t) trained according to Section C.2 Chen et al. [27] suggest using one random timestep during each purification iteration while we believe that smaller timesteps should be more effective since they retain more information from the original example. Thus, we limit the random selection to within the first 50 timesteps. This way significantly boosts adversarial robustness, which will be discussed in Section 5
Causal Factor Inference (CFI). In order to infer the causal and non-causal factors S and Z, which can reconstruct the original image X, we optimize the latent variables by maximizing the conditional likelihood Po (z|s, z) employing the trained conditional diffusion model e co é, s,2):
s" 3²,2" = arg max,,; log pe
Similarly to purification, -E,,, [wellco( f, 8, ciency concerns, instead of using all the timesteps for estimation as in Chen al. 27]. sample Npurity = 5 timesteps at the same intervals across the entire timesteps.
by maximizing the lower bound <|]] using the conditional diffusion model co(x,, f, 8,2). For effi-
obtain 8 and
Latent-S- Satent---Based Classification (LSBC). After obtaining s* according to Eq. 9 we use the trained classifier Jn (8; 0) to predict label
8" = arg max, log po (")s")).
(10)

---

Table Clean accuracy and adversarial robustness against AutoAttack (AA) on GTSRB (Left) and CIFAR-100 (Right) dataset. We use = 8/255 as 00 and 0.5 as E norm bound.
Clean Acc
Robust Ace
Method
Clean Ace
Robust Acc
Method
AA 1s0
AA &
Fog
Avg
WRN40-2
78.13
0.00
DOA 61
76.56
31.25
36.72
68.36
45.44
AT-DDPM
23
63.56
34.64
GTSRB-CNE
3
93.95
62.30
74.80
65.43
67.51
AT-EDM 9
75.22
42.67
AT-4
92.58
74.78
80.47
78.13
77.69
DiffPure
33
39.06
7.81
AT-8
91.21
74.02
79.10
73.44
75.52
DC 27
79.69
39.06
AT-
8
89.65
73.24
75.59
69.92
72.92
RDC 27
80.47
53.12
CausalDiff
97.85
80.86
80.86
86.13
82.62
CausalDiff' 65.62
56.25
Within our inference pipeline, both adversarial purification and causal factor inference leverage the diffusion model learned toward the CIB optimization objective while they take effect independently. When we combine these two approaches, adversarial robustness could be enhanced further. The concrete inference algorithm in Algorithm 3 is presented as Appendix B.2
4.4 Comparison with Adversarial Purification
First, our CausalDiff can viewed as semantic-level purification. Instead of pixel-level denoising. CausalDiff purifies image in the latent space, trying remove the effect of perturbation by putting it to the label-non-causative features. Second, conventional purification inevitably loses information essential for classification during denoising. By disentangling label-causative features from label- non-causative features, CausalDiff can retain essential information in the label-causative features to large extent. Third, unlike pixel-level purification which does not know the optimal denoising level for various attacks, CafsalDiff acts adaptively on different attacks by the causal inference of S and Z. Fourth, they can be combined to further enhance adversarial robustness.
Experiments
presents the main results defending against unforeseen attacks on CIFAR-10, CIFAR- 100, and GTSRB datasets. We then evaluate the effectiveness of individual components of CausalDiff in Section 5.3 Due to space constraints, we provide analyses of core components during training and inference Appendix C.I
In this section, we introduce the experimental settings in Section 5.1 Section 5.2
and Appendix C.2 Additionally, we showcase examples in Appendix C.3
5.1 Experimental Settings
Datasets and Model Architecture. Our experiments utilize the CIFAR-10, CIFAR-100 18 and GTSRB [19 datasets. CIFAR- and CIFAR-100) each consists of 50,000 training images, categorized into 10 and 100 classes, respectively. GTSRB comprises 39,209 training images (each histogram equalized and resized to 32 32) of German traffic signs, categorized into 43 classes. We use
DDPM [1 as the diffusion model. Further details are available in Appendix [.22
Attack Evaluation Method. For the CIFAR-10 10 dataset, we utilize seven types of attack strategies with both Lx and E norm bounds for evaluation. These strategies include StAdy attack [60], BPD and AutoAttack (2] (AA), which comprises white-box attacks such APGD-ce. APGD-t, FAB-t, and a black-box Square Attack. For CIFAR-100, we follow the setting of Chen et al. 27,, evaluating against the fox threat model with 8/255. For the GTSRB dataset, utilize four types of attacks as well as fog corruptions. Attack methods include AutoAttack, which comprises APGD-ce, APGD-1, FAB-t, and Square Attack.
5.2 Comparisons on Unseen Attacks
CIFAR-10. From the experimental results for CIFAR-10 presented in Table we have several observations: 1) The adversarial training methods perform robustly on same type of attacks they use for training but poorly on other unseen attacks. The only exception is that when the models employ

---

the most effective attack (i.e., A0) ) for adversarial training, the robustness regarding Ez is not hurt much. 2) In contrast, purification methods, especially the ones based on more powerful diffusion models (DDPM and FDM),, have decent robustness on each unseen attack and much better average robustness. This is reasonable since they learn to purify the adversarial samples without targeting any specific attacks. 3) Our CausalDiff performs the best not only regarding the average robustness but also on each type of attack. Remarkably, the average robust accuracy (86.39%) is only 3.84% lower than the clean accuracy (90.23%) and it boosts the robustness on the most challenging attack (6oo) to 83.01% %. Notably, the reported CausalDiff is based on DDPM, which acts less effectively than EDM, which can be
from AT and LM purification. Grounded achieve even better performance. Table the performance under the BPDA EOT (EOT = 20) attack. Under this attack, models whose gradients not accessible and also be compared (e.g.. ADP 31.. The results again confirm the superior performance of CausalDiff across different attack types.
stronger backbone, Cifs could
CIFAR-100. The experimental results on CIFAR-100 in Table 3 (Right) indicate that CausalDiff achieves the highest robustness among all, even with a much lower clean accuracy. The low clean accuracy is likely due to the insufficient training samples to learn S, Z, and its weaker backbone
Table 4: Clean and robust accuracy on CIFAR-10 against BPDA EOT against to (e = 8/255) threat model.
airplane
bird
deer
frog
ship
automobile
cat
dog
horse
truck
Clcan Acc (%)
Robust
Method
Acc (%)
Purify EBM 62
84.12
54.90
LM DDPM 27
83.20
69.73
ADP 31
86.14
70.01
concat [s:z]
label-causative factor
label-non causative factor z
RDC 27
89.85
75,67
GDMP 35
93.50
76.22
Figure 4: Visualization by T-SNE of the feature space,
DiftPure 33
89.02
81.40
inferred by CaffalDiff, of the label-causative factor s,
CausalDiff
90.23
88.48
factor 2, and their concatenation.
DDPM. We believe more augmented data and stronger diffusion backbone like EDM (used by RDC 27 could further enhance the performance.
GTSRB. The left part of Table 3 show that Ciff also has compelling robustness on traffic sign classification, in terms of not only unforeseen adversarial attacks but also natural corruptions like fog. Evaluations based on different types of tasks, and different numbers of classes (10, 100, and 43) have all shown the efficacy of CausalDiff.
5.3 Ablation Study
As mentioned in Section 4.3 our CausalDiff contains Adversarial Purification (AP), Causal Factor Inference (CFI), and Latent-S-Ban -Based Classification (LSBC). The last block of Table 2 shows the individual effect of the AP and CFI in CausalDiff. We can see: 1) Our DDPM-based purification (CausalDiff w/o CFI, achieved by AP plus a standard classifier) performs similarly to LM-EDM and is significantly better than LM-DDPM, showing that the strategy of sampling within small timesteps for purification is much more effective than entire timesteps. (We show more analysis this in Appendix C 2) The core component CasualDiff causal disentanglement (CausalDiff w/o AP) outperforms all the baselines in terms of average robustness except RDC which incorporates extra LM purification step. It shows that modeling the generative of native in-domain data can enhance the model's 's inherent robustness and thus effectively defend against various types of attacks.
5.4 Visualization of Latent Factors
To understand how the latent causal factors S and Z in CausalDiff take effect during adversarial classification, we visualize S, Z, and their concatenation using t-SNE in Fig. [ We randomly sampled 5000 correctly classified test samples from CIFAR-10 10 for visualization. We find that the Y-causative factor S of samples in each category are located in the same cluster, with clear margins between different clusters except the categories dog, cat, and bird. These three categories share more commonalities than the others and are not surprising to have blurred boundaries. Additionally,

---

the S' vectors of airplanes (dark blue) are near those of birds (green), and trucks have S vectors near automobiles. In contrast the vectors, the vectors of Z do not exhibit correlations with the categories. This also aligns with our objective to extract the Y-non-causative factors to Z. The vectors of their concatenation, i.e., [s; 2]. also display clusters but with much more blurred boundaries. These observations are consistent with our commonsense knowledge, showing that CausalDiff has learned reasonable Y-causative factors by and -nnon-casattvve factors by Z.
Conclusion
develop causal model based on diffusion model to improve adversarial robustness. A pilot study on toy data suggests that the model defends against adversarial attacks by leveraging label-causative features to resist perturbations and expand the model's margin. Moreover, on the CIFAR-10, 10, CIFAR- 100 and GTSRB datasets, model appears to capture semantic features consistent with the human decision-making process and surpass all baseline models, achieving state-of- ofaeeoof-hh-arr performance in adversarial robustness, particularly against unseen attacks.
Acknowledgments and Disclosure of Funding
This work is supported by the Strategic Priority Research Program of the Chinese Academy of Sciences, Grant No. XDB0680101, CAS Project for Young Scientists in Basic Research under Grant No. YSBR-034, the Innovation Project of ICT CAS under Grant No. E261090, the National Natural Science Foundation of China (NSFC) under Grants No. 62302486, the Innovation Project of ICT CAS under Grants No. E361 140, the CAS Special Research Assistant Funding Project, the project under Grants No. ICKY202213CC039 and the Strategic Priority Research Program of the CAS under Grants No. XDB0680102. We would like to thank the UIBE GVC Laboratory and the Digital Economy Laboratory at the University of International Business and Economics for their provision of computational resources and support.
10

---

References
[1] J. Goodfellow, J. Shlens, and C. Szegedy, "Explaining and harnessing adversarial examples," arXiv preprint arXiv:1412.6572 2014.
[2] F. Croce and M. Hein, "Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks," in International conference on machine learning. PMLR, 2020, pp. 2206-2216
[3] K. Eykholt, I. Evtimov, E. Fernandes, Li, A. Rahmati, C. Xiao, Prakash, T. Kohno, and D. Song, "Robust physical-worle attacks on deep learning visual classification," in Proceedings of the IEEE conference on computer vision and pattern recognition. 2018, pp. (625-1634,
[4] M. Sharif, S. Bhagavatula, L. Bauer, and M. K. Reiter, "Accessorize to a crime: Real and stealthy attacks on state-of-the-an face recognition." in Proceedings of the 2016 acm sigsac conference on computer and communications security, 2016, pp. 1528-155
[5] Levinson, J. Askeland, Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer. O. Pink, V. Pratt et al., "Towards fully autonomous driving: Systems and algorithms." in 2011 IFFE intelligent vehicles symposium (IV). IEEF, 2011, pp. 163-168.
[6] N. Carlini, G. Katz, C. Barrett, and D. L. Dill, "Provably minimally-distorta adversarial examples," arXiv preprint arXiv: 1709. 10207, 2017.
[7] M. Hein and M. Andriushchenko, "Formal guarantees on the robustness of classifier against adversarial manipulation," Advances neural information processing systems, vol. 30, 2017.
[8] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, "Towards deep learning models resistant to adversarial attacks," arXiv preprint arXiv: how::706.krr 766806 2017.
[9] Z. Wang. T. Pang, C. Du, M. Lin, W. Liu, and S. Yan, "Better diffusion models further improve adversarial training." arXiv preprint arXiv:2302.04668: 2023.
[10] P. Samangouei, M. Kabkab, and R. Chellappa, "Defense-gar gan: Protecting classifiers against adversarial attacks using generative models," arXiv preprint I 1805.06605 2018.
[11] J. Pearl and D. Mackenzie, The book of why: the new science of cause and effect. 2018.
Basic books,
[12] B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio, "Toward causal representation learning," " Proceedings of the IEEE, vol. 109, no. 5, pp. 612-634, 2021.
[13) C. Liu, X. Sun, J. Wang, H. Tang, Li, T. Qin, W. Chen, and T.-Y. Liu, "Learning causal semantic representation for out-of-distribution prediction," Advances in Neural Information Processing Systems, vol. 34, pp. 6155-6170, 2021.
[14] X. Sun, B. WI, X. Zheng, C. Liu, W. Chen, T. Qin, and T.-Y. Liu, "Recovering latent causal factor for generalization to distributional shifts," Advances in Neural Information Processing Systems, vol. 34, pp. 16 846-16 859, 2021.
[15] Y. Zhang, Gong, T. Liu, G. Niu, X. Tian, B. Han, B. Schölkopf, and K. Zhang, "Causaladv: Adversarial robustness through the lens of causality," arXiv preprint 2XXi:2206600166, 2021.
[16] Q. Ren, Chen, Y. Mo, Q Wu, and Yan, "Dice: Domain-attack invariant causal learning for improved data privacy protection and adversarial robustness." in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, pp. 1483-1492.
[17] J. Ho, A. Jain, and P. Abbeel, "Denoising diffusion probabilistic models," Advances in Neural Information Processing Systems, vol. 33, pp. 6840-6851, 2020.
[18] A. Krizhevsky, G. Hinton et al., "Learning multiple layers of features from tiny images," 2009.
11

---

[19] J. Stallkamp. M. Schlipsing, J. Salmen, and C. Igel, "The german traffic sign recognition benchmark: a multi-class classification competition, in The 2011 international joint conference on neural networks. IEEE, 2011, I, pp. 1453-1460.
[20] H. Zhang, Y. Yu, J. Jiao, E. Xing, L. El Ghaoui, and M. Jordan, "Theoretically principled trade-off between robustness and accuracy," in International conference on machine learning. PMLR, 2019, pp. 7472-7482.
[21] C. Qin, J. Martens, S. Gowal, D. Krishnan, K. Dvijotham, A. Fawzi, S. De, R. Stanforth, and Kohli, "Adversarial robustness through local linearization," Advances in Neural Information Processing Systems, vol. 32, 2019.
[22] S. Gowal, S.-A. Rebuffi, O. Wiles, F. Stimberg, D. A. Calian, and T. A. Mann, "Improving robustness using generated data," Advances in Neural Information Processing Systems, vol. 34, pp. 4218-4233, 2021.
[23] S.-A. Rebuffi, S. Gowal, D. A. Calian, F. Stimberg, O. Wiles, and T. Mann, "Fixing data augmentation to improve adversarial robustness," arXiv preprint arXiv:2
2021.
[24] A. Shafahi, M. Najibi, M. A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. S. Davis, G. Taylor, and T. Goldstein, "Adversarial training for free!" Advances in Neural Information Processing Systems, vol. 32, 2019.
[25] H. Jiang, P. He, W. Chen, X. Liu, Gao, and T. Zhao, "Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization," arXiv preprint arXiv: 1911033311 2019.
[26] C. Laidlaw, S. Singla, and S. Feizi, "Perceptual adversarial robustness: Defense against unseen threat models," arXiv preprint arXiv:2006. 12655, 2020.
[27] H. Chen, Y. Dong. Z. Wang, X. Yang, C. Duan, H. Su, and J. Zhu, "Robust classification via single diffusion model," arXiv preprint arXiv:2305. /5241, 2023.
[28] Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, "Deep unsupervised learning using nonequilibrium thermodynamics," in International Conference on Machine Learning. PMLR, 2015, pp. 2256-2265.
[29] Y. Song and S. Ermon, "Generative modeling by estimating gradients of the data distribution," Advances in neural information processing systems, vol. 32, 2019.
[30] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, "Score-based generative modeling through stochastic differential equations," arXiv preprint arXiv:2011.13456, 2020.
[31] J. Yoon, J. Hwang, and J. Lee, "Adversarial purification with score-based generative models," in International Conference on Machine Learning. PMLR, 2021, pp. 12 1202--12072.
[32] C. Xiao, Z. Chen, K. Jin, J. Wang, W. Nie, M. Liu, A. Anandkumar, B. Li, and D. Song, "Densepure: Understanding diffusion models towards adversarial robustness," arXiv preprint arXiv:2211.008: 00322, 2022.
[33] W. Nie, B. Guo, Y. Huang. C. Xiao, A. Vahdat, and A. Anandkumar, "Diffusion models for 2022.
adversarial purification," arXiv preprint
[34] Q Wu, H. Ye, and Y. Gu, "Guided diffusion model for adversarial purification from random noise," arXiv preprint arXiv:2206.1081 2022.
[35] Wang, Z. Lyu, D. Lin, B. Dai, and H. Fu, "Guided diffusion model for adversarial purification," " arXiv preprini arXiv: 14969, 2022.
[36] J. Mitrovic, B. McWilliams, J. Walker, L. Buesing, and C. Blundell, "Representation learning via invariant causal mechanisms." " arXiv preprint arXiv:2010.07922, 2020.
[37] B. Schölkopf, "Causality for machine learning." " in Probabilistic and Causal Inference: The Works of Judea Pearl, 2022, pp. 765-804.
12

---

[38] Y. Chen, Y. Zhang, Y. Bian, H. Yang, M. Kaili, B. Xie, T. Liu, B. Han, and J. Cheng, "Learning causally invariant representations for out-of distributionn generalization on graphs," Advances in Neural Information Processing Systems, vol. 35, pp. 22 131-22 148, 2022.
[39] W. Wang, X. Lin, F. Feng, X. He, M. Lin, and T.-S. Chua, "Causal representation learning for out-of-distribution recommendation" in Proceedings of the ACM Web Conference 2022, 2022, pp. 3562-3571.
[40] R. Wang, M. Yi, Z. Chen, and S. Zhu, "Out-of-distribution generalization with causal invariant transformations."" in Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2022, pp. 375-385.
[41] R. Moraffah, M. Karami, R. Guo, A. Raglin, and H. Liu, "Causal interpretabilityy for machine paaminngrroblems, methods and evaluation," ACM SIGKDD Explorations Newsletter, vol. 22, learning- problems,
no. 1, pp. 18-33, 2020.
[42] G. Xu, T. D. Duong, Q. Li, S. Liu, and X. Wang, "Causality learning: A new perspective for interpretable machine learning," arXiv preprint arXiv:2006. 16789, 2020.
[43] C. Zhang, K. Zhang, and Y. Li, "A causal view robustness of neural networks," Advances in Neural Information Processing Systems, vol. 33, pp. 289-301, 2020.
[44] K. Tang, M. Tao, and H. Zhang. "Adversarial visual robustness by causal intervention," arXiv preprint arXiv:2106. 09534, 2021.
[45] S. Yang. T. Guo, Y. Wang, and C. Xu, "Adversarial robustness through disentangled representa- tions," " Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 4, 2021, pp.
[46] B.-K. Lee, J. Kim, and Y. M. Ro, "Mitigating adversarial vulnerability through causal param- eter estimation by adversarial double machine learning," " in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 4499-4509.
[47] J. Kim, B.-K. Lee, and Y. M. Ro, "Demystifying causal features on adversarial examples and causal inoculation for robust network by adversarial instrumental variable regression," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 12 302-12312.
[48] H. Hua, J. Yan, X. Fang, W. Huang. H. Yin, and W. Ge, "Causal information bottleneck boosts adversarial robustness of deep neural network," arXiv preprint arXiv: 14229, 2022.
[49] T. Karras, M. Aittala, T. Aila, and S. Laine, "Elucidating the design space of diffusion- based generative models," Advances in Neural Information Processing Systems, vol. 35, pp. 26 565- 26 577, 2022.
[50] P. Dhariwal and A. Nichol, "Diffusion models beat gans on image synthesis," Advances in Neural Information Processing Systems, vol. 34, pp. 8780-8794, 2021.
[51] J. Ho and T. Salimans, "Classifier-free diffusion guidance," arXiv preprint a7X5::2207.12598. 2022.
[52] K. Preechakul, N. Chatthee, S. Wizadwongsa, and S. Suwajanakorn, "Diffusion autoencoders: Toward meaningful and decodable representation," in Proceedings the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, 2022, pp. 10619-10629,
[53] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, "High-resolution image synthesis with latent diffusion models," in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2022, pp. 684-410 695.
[54] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen, "Glide: Towards photorealistic image generation and editing with text-guided diffusion models," arXiv preprint 1rXiv:2112.10741, 2021.
[55] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, "Hierarchical text-conditional image generation with clip latents," arXiv preprint arXiv:2204.0612 2022.
13

---

[56] G. W. Ding. Y. Sharma, K. Y. C. Lui, and R. Huang, "Mma training: Direct input space margin maximization through adversarial training," arXiv preprint arXiv:1812.02637,, 2018.
[57] Cheng, Hao, Dai, J. Liu, Z. Gan, and L. Carin, "Club: contrastive log-ratio upper bound of mutual information," in International conference machine learning. PMLR, 2020, pp. 1779-1788.
[58] S. Kullback, Information theory and statistics.
Courier Corporation, 1997.
[59] R. S. Zimmermann, I.. Schott, Y. Song, B. A. Dunn, and D. A. Klindt, "Score-based generative classifiers," arXiv preprint urXiv:2110.0047: 2021.
[60] C. Xiao, J.-Y. Zhu, B. Li, W. He, M. Liu, and D. Song, "Spatially transformed adversarial examples," arXiv preprint S.kiv:8801.02622 2018.
[61] T. Wu, L. Tong, and Y. Vorobeychik, "Defending against physically realizable attacks on image classification," arXiv preprint 1909..9990 2019.
[62] M. Hill, J. Mitchell, and S.-C. Zhu, "Stochastic security: Adversarial defense using long-run dynamics of meegy-bbssed models," arXiv preprint 2rXiv:2005.13525 2020.
[63] M. Boudiaf, J. Rony, I. M. Ziko, E. Granger, M. Pedersoli, P. Piantanida, and I. B. Ayed, "A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses," in European conference on computer vision. Springer, 2020, pp. 548-564.
14

---

A Proof of Propositions
In this section, we will present the detailed proof for the theoretical results mentioned in the main paper.
A.1 Proof of Causal Information Bottleneck (CIB)
According to the Structural Causal Model (SCM), we have p(x, 8, - p(5)p(=)p((r(s) =)p(uls). Thus, the Causal Information Bottleneck (CIB) can be represented as:
(1,Y;S,22) = Ep(x.y...)) log p(z, y, 8,
(),(, 2)
=)p(s)s)
= =R log
p(z, (),,2)
p( s, 2 (yysss)
Ep(ays,.)) log p(s)p(2)
-Ep(a.y,,,)) log
p(z, v)
p(8,2)
p( ss ()(((ss 1(S:2)
p(........) log
p(z, y)
=E =p((......) log
I(S:Z)
p(y) p(y)=)p(z)pp(1
p(y|s) E ((..))))))) log p(z|s, 2)p(2,2) I(5;2)
(11)
E Bp log
p(y
p(z)p(s,22
=E =b log p(y
I(X; 2) I(S;
p(y(z)
-Rp(a.u.e.s) log p(y
8 p(y)s)p(s)p(() I(X; I(S;2)
-Rp log p(y) + E p() log p(y)s)p(s)
p(s)p(u) T(X; 2) 1(S:2)
p(y(z)
-Ep(r.y...>) log p(y)p(x)
2) 1(S;
()(())
=((XY) + 5S) + 5, Therefore, we have proved the result Eq.
A.2 Proof of the Lower Bound of CIB
According to the result in Eq. (TT). we further prove its lower bound shown in Eq. (6) in this section.
As for the lower bound of the recontruction term (X; we have:
(X) Ep(x,,,=) (log p(z|s, 2) Ept,,s,) [log
p(x)
p(x ss, 2)po (x) 2)
when using variational distribution po(x)),2) to approximate p(x ( 2),
p(x) po
( 2)
+ E (p(rln.)) [log p(zls, 2)
Ep(n,..))logg llog
p(x)
po (x)
po =)
1 E Bpl,,)) (DLL p(rls,)) where DKL(.) represents KL-divergence, po (x(ss
Eppn,,,) Dlg
p(x)
Ep(r...)(logg flog p)(x)8,2)
p(x))
p (x)s, 2]] + Ep [log p(x)
Eptr...)) llg 2]] H(xx), where H()x) indicates entropy of z.
(12)
15

---

As for the upper bound of I(X;; S, 2), we have:
2) E. p(x....)) log p(x ((2),,22
p(r)
Ep(,,,)) flog p( 2) 2)g(8,2)) 2)
when using a prior distribution ((s, 2) to estimate p(s, 2).
p(z)q(s,
(, 1og
(KI,(((s, (0)1(,)))lllsss.. =)l,)(s,))) z)); where (-I.()) represents KI-ddivergence,
a(8, 2)
2(
S (p) Dog
(s, 2)
Ep(c) Bpte Dlog p(3, 2/x]
((,,2)
- (PKt ( =]]]-
(13)
Regarding the label prediction term I(;; 5), we can maximize the mutual information between factor and label Y by maximize Ep(r.)) [log Po (yl s]] well as employing cross-entropy loss function, according to Boudiaf et al. [63
As for the disentangle term 1(S; 2), according to the Contrastive Log-Ratio Upper Bound (CLUB) of mutual information proposed by Cheng et al. [57] we have
I(S; Z) S CCLUB (S; Z) (s)) Po (s|2)] Boo
(14)
where (s(s]z) is a variational distribution to estimate p(s[z). Thus, have proved the results in Eq. (6 that
I(X; 2) I(Y I(S; Z) AI(X:5,2) 2)
2 Ep)....)) Po 2]] Pp...) [log (y|s)] Ier.m (S; Z) A (Po,s 2)]]-
(ppt....))loggo000 =)]
Ep(,,)) [log [=]=]()]=]]
+ Bcce)Bpco PKL (<))(s=))
(15)
A.3 Detailed Derivation of Loss Function
we can optimize the Causal Information Bottleneck (CIB) I(X, Z) by maximizing its theoratically lower bound. In this part, we discuss the detailed derivation for the lower bound of CIB in term of designing the loss function proposed in Eq. 7
Based on the result Eq. 15
Specifically, for the reconstruction term (log (2rls, =)]. we can maximize the log-likelihood estimated by conditional diffusion model:
log ( 2) 2 -E.,, (we t, s, - <lI + C,
(16)
where (x 2) denotes conditional diffusion model, and constant C is negligible [17].
Regarding the label prediction term Ep(r..) [log pe ()]] following the results proposed by Boudiaf et al. (63] we can leveraging a cross-entropy loss function to maximize the mutual information between factor S and label Y.
As for the disentanglement term I(S; Z). we following the optimization strategies proposed by Cheng al. [57], leveraging predictor pe learn the relationship between S and Z so estimate 1LLUB (5; Z).
Overall, we have proved the loss function of our Causal Information Bottleneck (CIB) proposed in Eq.
a E.,t llee (at, L, 8, =) - <ellă + >Lcr(s, + n E ( Po Po (s|2)] - Ep En(s) [log Pa (s(2))) + A (po 211r) [g((s
L(x, [[,,0,8,23
(17)
where Epto. (og (s|2)] - Ep(x) log (s|22) is the estimation of 1LLUB
16

---

Algorithm CausalDiff Algorithm
Require:
Dataset D; The CausalDiff parameterized by 0 pretrained by Algorithm involves UNet co( 8, 2) with diffusion timestep T, an encoder (x; 8), a classifier f (s: f). and an
MI estimater SCLUB (s, 2; 0) for the CLUB loss; an optimizer optim()) ) dropout probability Pdrop of for simultaneously training an unconditional generation, hyperparameters a, X, 7,
Initialize:
Load f=,= (2; 6) and 6g from the pretrained model 0. Initialize (s; 0) and fCLUB 2; 2;0) randomly.
for ep=l to N2 do
Get mini-batch (z. y) D
9,
f(s,2) ( [2; 0)
Uniform 21. TT), 10
if rand(0, 1) S Pdrop then
Compute loss [(x; 0) ||co (z,, t) el1 Update Eo with o0tim(0, ((x;))
else
Compute loss [(x, U, 2; 8) according to Eq. Update co 8, 2), J(s,a) (x; 0). and fs(s; 0) by optim(0, f(x, y, s;
end if
Update fCLUB(S, 2; according the CLUB algorithm [57 end for
A.4 ELBO (Evidence Lower BOund) V.S. MI (Mutual Information)
the Causal Evidence Lower BOund (ELBO) for multi-domain datasets as proposed by Sun et al.
[14, we directly formulate the Causal ELBO within our causal structure, depicted Fig. (Left).
Given that p(x, 8, 2) p(s) (p)ppppp((ss)))) ()((s)) we can derive the Causal ELBO in single domain
as follows:
Po 8,
ELBO Ep(a,y [gn (1,2 log
2v (8, 2(1,,)
(0(s,2) (s x))
log po s) log 2r (2)
Ep(nn)) (x,2(aa)))))
log / |s 2, + log
Io (2) s) s. (18)
where po is to learn the ground-truth p and d is variational distribution mimic po-
Specifically, the causal ELBO, compared to our Causal Information Bottleneck (CIB), incorporates PP((.22
the reconstruction term (e s, s,z). the insensitivity term log {e({_=( and the label prediction
term log (y 8). However, it overlooks the disentanglement of latent factors, critical aspect for effectively learning the causal model. Additionally, we have empirically assessed the robustness of
models trained with either CIB or causal ELBO (equivalent to 17 = 0) in section C.I This evaluation
aims to investigate the efficacy of the disentanglement term 1(S; 2).
B
More Implementation Details
B.1
Baselines
We include representative defense methods of adversarial training (AT), purification, and other types (e.g., generative-monn moweee-ooelbbbbeeeee based approach) as baselines. Specifically, we compare with the AT methods
9 23 that use DDPM or the Elucidating Diffusion Model (EDM) 49 for data augmentation and
theoretical framework TRADES [20] for adversarial training. We also include causality- based AT baselines such as CausalAdy [15] and DICE [16). Purification baselines include DiffPure (33 (grounded on Score SDE 30 and diffusion-based Likelihood Maximization (L with EDM (as in the original paper) and DDPM (we reproduced). Other defense baselines comprise generative classifiers such as Score-Base Generative Classifier (SBGC) [59], (27), and the causality- based
17

---

Algorithm 2 CausalDiff Pretrain Algorithm
Require: Dataset diffiusion model eg, an encoder (x; 0), a classifier (s 0); an optimizer optim(-)) probability párop of training samples for unconditional diffusion, diffusion training epoch Ni.
Initialize: randomly initialize parameter 0.
for ep=1 to N1 do (x, v) ~
8, ((..s)(z;)
8, < with probability Parop
randomly mask the latent factors with probability Parop
t~ Uniform( 2,, T)), 10,
f(x, 2;;)) lea (2t, 8) -el12
Update eg and (x, 0) by optim(0, 8,;1),
end for
Algorithm Adversarially Robust Inference Algorithm
Require: A fully-trained causal model involves diffusion model co I, 8, =). encoder [(..s) (z; 0), and classifier fy (s; 0); test image z, purification optimization steps N purifyy causal factor inference steps Ninter. number of sampling steps N₂ for inferennce.optmmizer optimpunity for purification, opti- mizer ope for causal factor inference.
Initialize: purified image x' = x Stage 1: Adversarial Purification for iter=1 to N parify do Uniform( {1, 2,.......... 50)), ((0, L(x*, ||ea (It, e|p
Update x by optim (x" [(xxx())))))))))))))))
end for
Stage 2: Causal Factor Inference
Initial s" 2" f((e,)) (x:"; :))
for iter=l to Ninter do
Sample N, timesteps at equal intervals from to T
Sample N₂ e ~ NN I) corresponds to each
Compute L(x", "=,",,""==="000 = Re, lleo (xt 1, x) <|2
Update s* 2 by optim (s" ;;; L(x S² ,"'""" ;))
end for
Stage 3: S-based Classification
= 6)) return y
generative model CAMA 43 Diffusion Classifier (RDC), which incorporates purification and conditional generation based on labels for defense.
Notably, we also include the SOTA method on unseen attacks Robust
B.2 Implementation Details
We use DDPM 17 as our generative model. For the encoder fa. 0) and the classification model fo (y; 0), we employ WideResNNet-0-116 (WRN-70-16) as the backbone.
Training Strategy. Considering the different complexities in learning fa 2; 9), f(s.:) 8) and
(x; 0), attributed to the differing difficulty in generation and discrimination task, we segment the training of the entire causal model into two distinct phases. As outlined in Section 4.2 we employ two-stage training process for CausalDiff.
18

---

X
Y
V
V
S
Z
X
Y
X
Y
X
Y
(a) Discriminative
(b) Generative
(c) Causaal Anthon
(d) Causalyuu
Figure 5: SCM of models for pilot study including (a) discriminative model, (b) generative model, (c) causal model without disentanglement, and (d) causal model with disentanglement.
C
Ø
Forward Propagation
Baokward Propagation
Enc
Enc
Enc
Enc
Cen
i
Ce
Beo
i
CCE
Lice
2
Ce
Cls
Dec
Cls
Dec
Cls
y
(a) Discriminative(b) Generative
(c) Causalwitvou Disent
(d) Causalwsh Disent.
Figure 6: Architecture of models for pilot study including (a) discriminative model, (b) generative
model. (c) causal model without disentanglement, and (d) causal model with disentanglement.
In the pretrain phase on CIFAR-10 and GTSRB datasets, focused on generation, primarily trains conditional diffusion model (8, z; 0) along with its corresponding encoder ((=.:) 0) according to Algorithm 2 setting N₁ = 1440. For CIFAR-100 100, we added 10,000 images generated by EDM (49 to our training set. Considering the computational cost, we only pretrain for N1 500 epochs. Note that the augmented data is used only during the pretraining phase while not in the joint training phase.
Subsequently, we conduct joint training of the whole CausalDiff model for N2 = 560, amounting to a total of 2000 epochs. The second phase, targeting discrimination and leveraging label information to guide disentanglement, involves the joint training of the entire causal model. Note that, in order simultaneously train an unconditional diffusion model for adversarial purification, we follow Ho and Salimans [51] to mask the condition and with probability Parop 0.1. Thus, a single shared model used for both adversarial purification and causal factor inference.
Both the pretraining and joint training phases utilize a learning rate of le and a batch size of 128. For simplicity, we follow the setting of we [17]. We a 1., 7 - 17 = le -2 as the weights for the loss function in Eq. 7
Since need standard diffusion model (x), t) for purification during adversarial inference,
apply dropout of s and z with a ratio of Parop 0.1 for conditional diffusion generation as in Ho et al. [77 during both pretraining and training. Thus, the unconditional probability of generating can also be estimated using the same model by masking s and z.
Inference Strategy. Leveraging the trained CausalDiff, can infer its label from adversarial example accordance with Algorithm 3 following the inference pipeline outlined in Section 4.3 Specifically, set Npurity 5 and use momentum-SGD optimpurity- with learning rate of
19

---

94
92
88
Clean Accuracy
70-
Robust Accuracy
68
66
64
10
10
10
10
Weight n of (slub(s:z)
Figure 7: Impact of for disentanglement term in loss function clean accuracy and robust accuracy.
0.1. For causal factor inference, choose N, 10 to sample 10 timesteps per iteration and adopt Ninter 10 with momentum- -SD as optimuterr setting the learning rate to le -5.
Regarding white-box attacks, we perform gradient backpropagation throughout the entire pipeline of our CausalDiff approach, which includes purification, causal factors inference, and classification. This implies that the attacker possesses sufficient knowledge of the causal model.
Attack Evaluation. Attack evaluation for CIFAR-10 dataset includes a 100-step StAdv attack (60) with e = 0.05 under the foo norm bound, BPDA+EOT (EOT=20) against the fo threat model with ε = 8/255, and AutoAttack (2) (AA), which comprises 100-step white-box attacks such as
APGD-ce. APGD-t, FAB-t, and 5000-step black-box Square Attack under both lg (e = 0.5) and foo 8/255) constraints.
For the GTSRB dataset, we utilize four types of attacks as well as two types of corruptions to evaluate robustness against adversarial attacks and the influence of the natural environment. Specifically, the attack methods include AutoAttack, which comprises 100-step white-box attacks such APGD-ce, APGD-t, FAB-t, and a 5000-step black-box Square Attack under both & (e 0.5) and lo (e = 8/255) constraints. The corruptions include Fog and Brightness. Following Nie et al. randomly sample 512 samples for evaluation.
33
B.3 Details for Pilot Study
Data Construction. For each data point of the 2000 samples, we sample a vector 8 with dimension
h 8 from normal distribution with mean -1 and variance 1, i.e., ~ 1), and vector 2 ~ ((1,1) with dimension h; = 8. Subsequently, we projected the concatenation of s and 2, denoted as z]. to sample x, with a random initialized matrix (A E R(h, ), i.c. ss Ag. Similarly, we produced the score ys of x with ys s Ag. where Ag x To obtain samples with balanced binary labels, we consider the label y of the sample with ys above the median as and the others as O.
Methods for Comparisons. In the pilot study detailed Section 3 we conducted an investigation and analysis on four models: 1) Discriminative: discriminative model that learns to classify the samples with a two-layer perceptron (MLP), 2) Generative [27] generative model that learns the generation of the sample conditioning on its label and predict the label of an adversarial example by calculating the max, p(rl))) 3) Causal without Disent.: causal model that models the generation of both x and y with the same causal factor v (Causal modeling without Disentanglementt, Causal with Disent.: our model that disentangles the sabel-causative factor and another factor during the generation of x. For the latter two causal models, given an adversarial example, the hidden vectors or s. are inferred for Causal without Disent. and with Disent. which are then used for the final label prediction. This section comprehensively presents the designed causal structures in Fig. 5 and the model architectures in Fig.
20

---

Regarding implementation, we trained each of the four models for 20 epochs, optimizing the model parameters using the Adam optimizer with a learning rate of 1e-3, The latent dimension for each model was set to 64. For evaluation, we employed 100-step PGD attack with = 2/255 within the for norm boundary. The variation in latent variables and predicted logits between adversarial examples and clean images, as presented in Table is measured on adversarial examples generated with 10 and a attack.
3,
0.3 and
0.05 within the lo norm boundary, using a 100-step PGD
C More Experimental Results
C.1 Analyses on Core Components of Training
Effect of T CCLB (S: Z) in Casual Information Bottleneck (CIB) To examine the impact of our
introduced disentanglement term CCLUB(S: Z) in CIB (See Equation (6)). we vary in Equation (7) and evaluate the robustness against the most challenging attack in AutoAttack 2], i.e. with &oo norm bounded by 8/ 8/255. I Larger will cause the diffusion model to collapse, so we do not include the results of larger 7). we mentioned in Section 4.2 our CTB regresses to the ELBO objective in [14 when 1 = 0. As shown in Fig. 7 CausalDiff has better clean accuracy well as robustness when 1) > and yields the best robustness when 7 = 10 (69.14% compared to 65. 55.04% when v = 0). It confirms that LCLUB(( (S; 2) in the loss function is beneficial to disentangle the Y-causative from Y-non-causative factors and can further enhance both clean accuracy and robustness.
C.2 Analyses on Core Components of Inference
Causal Factor Inference Method. We also evaluate the robustness using the encoder f(s.:) (2; 0) to get latent variables instead of inferring and 2 through the conditional diffusion model. The results reveal that classification by the encoder (without purification) achieves a clean accuracy of 91.99% but 0.00% against AutoAttack under both lo and &g threat models. This decline might be attributed to imprecise modeling around x (i.e., x + 8), which results in an inability to resist adversarial perturbations on s.
Timestep 1. Sampling Strategies for Purification As discussed in Section 5.2 our purification
method (CausalDiffrwlo Causal Factor Inference in Table 2 markedly surpasses the direct adaptation of likelihood maximization (LM-DDPM in Table as proposed by Chen et al. [27]. applied to DDPM. This improvement stems from refined strategy in sampling the timestep t.
As demonstrated in Fig. 9 we found that smaller timesteps perform better in distinguishing between presented the negative log-
the distributions of clean and adversarial samples. Specifically, likelihood estimated by expectation E, [w>][ca t) - ell] for the given timestep over 512 examples. This may caused by larger timestep implies a greater degree of noise addition for estimating likelihood, which might overshadow the adversarial perturbations, unexpected for purification.
C.3 Visualization of Cases
We visualize the generated images leveraging the conditional generation of our CausalDiff, providing an intuitive depiction of the label-causative factor s* and the cauusaion- factor sn, Fig. g illustrates that after inferring from the benign example x", perturbations are alleviated. The image J., conditioned on s* showcases that s* captures the core predictive features, reflecting the general concept of the category (for instance, a common semantic representation of "horse" from an image of a brown horse's 's head), or even enhances the predictive features such as the dog's 's face or the ship's body, whereas 2" retains specific and non-predictive image details.
We also visualize the purified example x", conditionally generated x and x.- when encountering an adversarial example F. These cases demonstrate that, despite the presence of perturbations in the clean images, our CafsalDiff effectively captures the correct predictive information of s", maintaining alignment with the clean data.
21

---

[[]
gg
x
x
D
Figure 8: Reconstruction images []]e;==]] when given clean example where 8 and z inferred from the clean example x by our Ciffal generated images T.- and Tag conditioned on s and =e; respectively; purified image x* utilizing the unconditional diffusion (with 8, masked) when given an adversarial example म; generated images 25- and x conditioned on s* and 2", respectively, where and n are inferred from the purified image x².
random
t=50
t=200
t=500
t=1000
clean
Denatt
-0.004-0:002 0.000
-1.0
-0.5
-0.3
-02
-0.1
0.0
-0.044
-0.002
0.000
-0.005-0011-000000000
050-0.00004
-0.00002
Figure 9: Distribution of likelihood for adversarial and benign examples across various timesteps t,
C.4 Speed Test of Inference Time
We evaluate the computational complexity of CausalDiff and DiffPure 33 as well as a discriminative
model (WRN-70-11 16) by measuring the inference time in seconds for single sample (average on 100 examples from CIFAR-10 dataset) on two types of GPUs, including NVIDIA A6000 GPU and 4090 GPU (Our experiments leverage A6000 GPUs and 4 4090 GPUs). The results are shown in Table 5
Table 5: Comparison of NFEs (Number of Function Evaluations) across different models on GPUs
Ciff w/o Purify
CausalDiff
Cifs
DiffPure
WRN-70-16
w/o Causal Factor Infer.
NFE
N1 + N2 +
N2 + 4.62
N1 +
N1
1
Time (A6000)
4.97
0.29
2.22
0.011
Time (4090)
4.88
4.61
0.25
2.06
0.007
22

---

D Limitation
Although our CausalDiff significantly narrows the gap in classification accuracy between adversarial and clean examples, it requires an inference cost of Ni + N2 NFEs (Number of Function Evaluations), where efficiency improvements are needed. Note that N1 indicates the purification step (e-g. and N2 indicates the step of causal factor inference (e.g.. 10) and NFE for latent-S-based classification. Furthermore, our CafsalDiff represents a new framework, meaning it requires training from scratch. Perhaps in the future, an efficient implementation of robust inference could be achieved by embedding causal mechanisms into the existing models in a plug-and-pla manner.
E Broader Impact
Our CausalDiff model, built upon a powerful generative framework, aims to align with human decision-making mechanisms to enhance the stability and trustworthiness of neural networks. This approach holds potential for advancing the field of Machine Learning, particularly in safety-sensitive applications such as autonomous driving and facial recognition.
23

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: see abstract
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Appendix D
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification: Section 4.2 and Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section Appendix and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification: We described our experimental details in Appendix and we will open source our code in camera-ready version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
public/guides/ oow.le/s//oooeeueebsssssooooooon (osssiiccc)))) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/pp wwcppbblc/guuiicuccccccccoecoopp
http://oooSuuuisss.ooeee Stt.SSbbbiiiiooooiiiii.oooi for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
Adversarial attack evaluation incurs high computational costs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
26

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix C.4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpps:/neurrps. http://aawarg..cc/bbb/b///hcccea
Answer: [Yes]
Justification: Appendix
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix
E
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Appendix
E
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Appendix 4
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "S47EQ68wr3", "reviewer": "Reviewer_t8de", "summary": "The authors propose CausalDiff, a causality-inspired disentanglement approach using diffusion models for adversarial defense, which outperforms state-of-the-art methods on various unseen attacks across multiple datasets.", "questions": "Please see the weaknesses section.", "limitations": "The authors adequately addressed the limitations", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novel approach combining causal modeling and diffusion models for adversarial defense\n\nStrong performance against unseen attacks on multiple datasets\n\nThorough pilot study on toy data to validate the approach\n\nClear theoretical foundation with the proposed Causal Information Bottleneck objective\n\nPractical adaptation of diffusion models for conditional generation", "weaknesses": "Limited discussion on computational complexity and training time\n\nLack of comparison with other causal approaches for adversarial robustness\n\nNo analysis of the method's performance on more complex datasets (e.g., ImageNet)\n\nLimited exploration of the interpretability of the learned causal factors\n\nAbsence of ablation studies to isolate the impact of individual components", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We appreciate the time and effort of the reviewer. In response to the issues raised in the review, we offer the following replies:**\n\n**Q1**: Limited discussion on computational complexity and training time.\n\n**A1**:  We provide a detailed speed test in Appendix C.4, evaluating the inference time of our CausalDiff and baselines. We will include a discussion on efficiency in the methods section and link to the detailed analysis of efficiency in the appendix. For more details, please refer to the global response regarding the discussion of computational complexity and training costs.\n\n**Q2**: Lack of comparison with other causal approaches for adversarial robustness.\n\n**A2**: We have compared our work with representative and well-recognized studies in the field of causal method for adversarial robustness, including the pioneering work and highly cited articles. As mentioned in lines 40 to 44 and in the related work section, we discuss the differences between other causal approaches and our CausalDiff. Specifically:\n\n- **From a modeling perspective**, other causal approaches handling adversarial robustness like CausalAdv [1], Deep CAMA [2], and DICE [3] model adversarial attack behaviors, aiming to **identify adversarial factors** (e.g., a manipulation variable) through causal mechanisms. This requires learning targeted at specific attack types during training, which limits their robustness across unseen attacks. In contrast, **our CausalDiff models the generative mechanisms** of the genuine data itself in order to enhance adversarial robustness against various unseen attacks.\n\n- **Experimentally**, we compared the robustness of relevant causal approaches, including CausalAdv [1], Deep CAMA [2], and DICE [3], against different adversarial attack methods in Table 2, demonstrating the superiority of our CausalDiff.\n\n\nWe hope our response has addressed your concerns. If you think other essential related works should be included, we will add them in the next version.\n\n\n**Q3**: No analysis of the method's performance on more complex datasets (e.g., ImageNet).\n\n**A3**: Due to resource limitations, we have not conducted tests on very large datasets. However, we selected three commonly used datasets for evaluation against various types of attacks and also the robustness against corruption (i.e., fog) in traffic sign recognition. We appreciate your suggestion. We are trying our best to test the effectiveness of CausalDiff on the ImageNet dataset.\n\n**Q4**: Limited exploration of the interpretability of the learned causal factors.\n\n**A4**: We have conducted the following analyses, confirming that S captures the label-causative factor while Z learns the label-non-causative factor:\n\n- **Visualization of distribution of S and Z**: As demonstrated in Figure 4, we leverage T-SNE to display the distributions learned by S and Z. We found that **S aligns with human semantic understanding of categories; semantically similar categories (e.g., cat and dog) are also close in the S space**. In contrast, Z shows a more blurred distinction between categories, which corresponds with our objective to isolate the label-non-causative factors into Z.\n\n- **Visualization of cases**: To analyze what $S$ and $Z$ have learned, we visualized images conditioned on $S$ (with $Z$ masked) and on $Z$ (with $S$ masked) using our trained CausalDiff. As depicted in Figure 1 (right), surprisingly, **$S$ captures the general semantic concept of a horse, even from just the head of the horse**, while $Z$ contains details like the horse’s skin color. For additional cases, see Figure 8 in Appendix C.3.\n\n- **Visualization of $S$ and $Z$ Interpolation from cat to dog**: Additionally, as shown in Figure 2 of the uploaded one-page PDF, we plotted the interpolation of S and Z from a cat to a dog. The evolution process in S shows cat-specific features such as facial characteristics fading, while retaining the animal's body, a commonality between cats and dogs. Conversely, the evolution process in Z includes some unimport information with respect to its category. This evolutionary process aligns with human cognition as expected.\n\n**Q5**: Absence of ablation studies to isolate the impact of individual components.\n\n**A5**: In the last three rows of Table 2, we present the performance of CausalDiff and its individual components, including removing adversarial purification and replacing causal factor inference via a standard discriminator. The corresponding analysis can be found in Section 5.3. The results indicate that each component independently exhibits commendable robustness and is essential; together, they achieve superior robustness.\n\n**We appreciate your efforts and are open to further discussion if you have any additional concerns.**\n\n[1] CausalAdv: Adversarial Robustness through the Lens of Causality, ICLR 2022\n\n[2] A Causal View on Robustness of Neural Networks, NeurIPS 2020\n\n[3] DICE: Domain-attack Invariant Causal Learning for Improved Data Privacy Protection and Adversarial Robustness, KDD 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pnw8SBMAot", "reviewer": "Reviewer_DuQ4", "summary": "This paper proposed a novel causal diffusion framework based on causal inference to defend against unseen attacks. The causal information bottleneck is interesting to disentangle the target-causative and target-non-causative factors, and then target-causative factors are used for adversarial defense.", "questions": "1. Some technical details need to be explained. For example, what is the actual meaning of the constraint in Eq. (4)? And why are the positions of z_s and s_b defined like in Eq. (2)?\n \n2. Numerically, the authors could consider comparing their method with more baselines. There are some studies on adversarial defense, even without using diffusion models.", "limitations": "1. What is the technical drawback of the proposed method? E.g., defense efficiency\n2. Does this proposed method work for data other than images?", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The originality of this paper is good, since the causal diffusion framework and causal information bottleneck are innovative and effective for adversarial defense. The quality and clarity of this paper are also good, where the logic is clear for understanding. The significance of this paper is obvious, because unseen attacks are necessary to address.", "weaknesses": "1. Some technical details need to be explained. For example, what is the actual meaning of the constraint in Eq. (4)? And why are the positions of z_s and s_b defined like in Eq. (2)?\n \n2. Numerically, the authors could consider comparing their method with more baselines. There are some studies on adversarial defense, even without using diffusion models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We appreciate the time and effort of the reviewer. In response to the issues raised in the review, we offer the following replies:**\n\n**Q1**: Some technical details need to be explained. For example, what is the actual meaning of the constraint in Eq. (4)? And why are the positions of z_s and s_b defined like in Eq. (2)?\n\n**A1**: The latent factors should **capture the semantic factors** in $x$, **rather than exactly matching** all the details of $x$ (even adversarial perturbation). Therefore, we constrain the mutual information $I(X; S, Z)$ by $I_c$ in Eq. (4) to limit the sensitivity of $s$ and $z$ to pixel perturbations in $x$.\n\nThe distinct roles of $z_s$ and $s_b$ dictate their different positions in the control mechanism. Inspired by **class-conditional generation** [6], the label-causative factor $s$ **controls semantics related to the category**, thus acting as a bias that affects the direction of the latent vector. In contrast, the label-non-causative factor $z$ **governs style and background, and can only scale the latent vector, similar to style control** [7].\n\nThank you for your feedback, we will add more specific details for explanation in the revision.\n\n**Q2**: Numerically, the authors could consider comparing their method with more baselines. There are some studies on adversarial defense, even without using diffusion models.\n\n**A2**: Indeed, most of the baselines in our experiments involve diffusion models for adversarial defense since they achieve state-of-the-art robustness in the domains of adversarial defense due to the capability of diffusion techniques. Comparing our method with these models provides valuable observations and insights.\n\nWe also compare our approach with discriminator-based adversarial training (AT series), causality-based methods (CausalAdv, DICE, CAMA), and geometry-based methods (DOA) in Tables 2 and 3.\n\nAccording to your valuable feedback, we add the following **non-diffusion defense baselines** with the same setting as Table 2:\n\n|       Method      | Architecture         | Clean acc | $\\ell_\\infty$ | $\\ell_2$ | StAdv | Avg  |\n|-----------------|----------------------|:--------:|:-------------:|:--------:|:-----:|:----:|\n| DecoupledKL [2]   | WideResNet-28-10     | 93.16     | 67.97         | 69.53    | 91.21 | 76.24|\n| RobustArchitecture (RA) [3] | RaWideResNet-70-16 | 93.55 | 71.48     | 68.36    | 91.60 | 77.15|\n| MeanSparse [4]    | RaWideResNet-70-16   | 93.36     | 72.27         | 69.92    | 91.80 | 78.00|\n| AT-scaling [5]    | WideResNet-94-16     | 93.95     | 74.41         | 69.53    | 92.38 | 78.77|\n| **Our CausalDiff**    | Diffusion            | 90.23     | 83.01         | 86.33    | 89.84 | **86.39**|\n\n\n**Q3**: What is the technical drawback of the proposed method? E.g., defense efficiency\n\n**A3**: As discussed in the limitations section of Appendix D, handling unseen attacks does indeed incur certain costs. Please refer to the global response regarding a detailed discussion of computational complexity and training costs. \n\n**Q4**: Does this proposed method work for data other than images?\n\n**A4**: This is an excellent idea! We believe that causal models may be applicable to different types of data while diffusion may not. For instance, transformer architecture might be more suitable for text. For text classification tasks, the label-causative factor **$S$ may encompass the semantics of intent**, while the label-non-causative factor **$Z$ could relate to language style or even language type** (which should not influence the task). Similar to CausalDiff, a causal model could be learned from observed data to enhance the adversarial robustness of text classification tasks. \n\nThank you for the inspiration; this is certainly worth further exploration. Besides, it may demonstrate the generality of our proposed framework in defending against adversarial attacks.\n\n**We appreciate your efforts and are open to further discussion if you have any additional concerns.**\n\n[1] MixedNUTS: Training-Free Accuracy-Robustness Balance via Nonlinearly Mixed Classifiers, Arxiv 2024\n\n[2] Decoupled Kullback-Leibler Divergence Loss, Arxiv 2023\n\n[3] Robust Principles: Architectural Design Principles for Adversarially Robust CNNs, BMCV 2023\n\n[4] MeanSparse: Post-Training Robustness Enhancement Through Mean-Centered Feature Sparsification, Arxiv 2024\n\n[5] Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies, ICML 2024\n\n[6] Classifier-Free Diffusion Guidance, NeurIPS 2021\n\n[7] Diffusion Autoencoders: Toward a Meaningful and Decodable Representation, CVPR 2022\n\n[8] Robust Classification via a Single Diffusion Model, ICML 2024\n\n[9] Diffusion Models for Adversarial Purification, ICML 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QSTCtzZduk", "reviewer": "Reviewer_dVc3", "summary": "This work aims to promote the trustworthiness of DNNs through purification. To this end, the authors propose a novel causality view to perform a disentanglement approach using diffusion models. Some experiments are conducted to verify the effectiveness of the proposed method.", "questions": "Please see the weakness.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ This work takes a good step towards practical adversarial defense. For instance, adversarial attack behaviors are unpredictable, leading to the requirement for robustness against unseen attacks. In this context, the authors highlight the challenge and propose a novel approach to address the challenge.\n\n+ The experiment designed on the toy dataset is interesting and provides clear points to depict the motivation of this work. Namely, disentanglement is a promising direction to promote the robustness of DNNs. Moreover, this aligns well with Eq. (3), especially the third term on the right. \n\n+ The experimental results are decent, where the proposed method achieves exciting robustness under various settings. Moreover, the proposed method is evaluated using a SOTA adversarial attack method, i.e., AutoAttack. This makes the results convincing.", "weaknesses": "- Adaptive attacks are lacking in this work, which significantly weakens its contribution and convincingness. Specifically, in the context of the considered white-box attack, what if the adversary generates adversarial examples using the following objective function? \n$\\max_{\\delta} \\ell_{ce}(x+\\delta, y) + \\log p(x+\\delta) + \\log p(x+\\delta|s^+,z)$ with $s^+ = \\arg \\min \\log p(y|s^+)$\n\n- It is hard to accept the conclusion shown in Figure 2. Specifically, DNNs would exhibit vulnerability when increasing the $\\epsilon$-budget. However, Figure 2 shows that the proposed method always makes DNNs robust. It is unclear what would happen if we further improve the budget. The results would be solid evidence for the false robustness if it is not intuitive.\n\n- Adversarial training with specific adversarial examples endows DNNs with robustness against these adversarial examples. This is intuitive. However, the current version of this work lacks a detailed and explicit explanation for the intuition or mechanism of why the proposed method can endow DNNs with robustness against various types of adversarial attacks. Relax; this is just a suggestion to make the work more solid. \n\n- It is known that diffusion models are sensitive to the inference step related to the inference time. Thus, the proposed method would introduce more cost in time. However, the authors seem to overlook the cost.\n\nMinor:\nThe second and third paragraphs of the Introduction lack references. For instance, the authors should provide corresponding works discussing certified defense, adversarial training, purification methods, and the causality and disentangle view.\n\n- The authors claim that CausalDiff can be viewed as semantic-level purification. This shows a relation to a previous work [1]. Specifically, performing a (non-semantic) subspace exploration would endow DNNs with robustness to distribution shift [1]. Thus, if we can abstract s* and z* using CausalDiff, we can further perform a distribution exploration to promote the robustness for OOD generalization. \n\n[1] Moderately Distributional Exploration for Domain Generalization. Dai et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We appreciate the time and effort of the reviewer. In response to the issues raised in the review, we offer the following replies:**\n\n**Q1**: Adaptive attacks are lacking in this work.\n\n**A1**: **All robustness evaluations in our paper are conducted against adaptive attacks.** This means the attacker has full knowledge of the inference mechanism and model parameters.  The attack objective $\\max_{\\delta} \\ell_{ce}(x+\\delta, y)$ is **adaptively applied to each component of CausalDiff, including purification and causal factor inference.**\n\nSpecifically, in the case of a gradient-based attack, the attacker generates adversarial examples with the objective $\\max_{\\delta} \\ell_{ce}(x+\\delta, y)$ according to the **chain rule**. The gradient for the adaptive attack in each component of CausalDiff is expressed as follows:\n\n$$\\frac{\\partial \\ell_{ce}(x+\\delta, y)}{\\partial (x+\\delta)} = \\frac{\\partial \\ell_{ce}(s^+, y)}{\\partial s^+} \\cdot \\frac{\\partial s^+}{\\partial x_{\\mathrm{purified}}} \\cdot \\frac{\\partial x_{\\mathrm{purified}}}{\\partial (x+\\delta)}$$\n\nThus, the objective $\\max_{\\delta} \\ell_{ce}(x+\\delta, y)$ of the adaptive attack is achieved by optimizing the following three terms:\n- $\\frac{\\partial \\ell_{ce}(s^+, y)}{\\partial s^+}$ (corresponds to the sub-objective $\\min_{s^+} \\log p(y|s^+)$)\n- $\\frac{\\partial s^+}{\\partial x_{\\mathrm{purified}}}$ (corresponds to the sub-objective $\\max_{\\delta} \\log p(x_{\\mathrm{purified}} | s^+, z)$, where $x_{\\mathrm{purified}}$ is generated by $x+\\delta$ in consideration of the likelihood maximization)\n- $\\frac{\\partial x_{\\mathrm{purified}}}{\\partial (x+\\delta)} $(corresponds to the sub-objective $\\max_{\\delta} \\log p(x+\\delta)$)\n\n**This aligns well with the aspects the reviewer mentioned that need to be considered simultaneously.** According to your valuable feedback, we will add a more detailed description of the implementation of the adaptive attack in our paper.\n\n**Q2**: It is hard to accept the conclusion shown in Figure 2. \n\n**A2**: In PGD attack, both $\\epsilon$-budget and step size $\\alpha$ together determine the attack's strength. In our experimental setup, as detailed in Appendix B.3, we use a constant value of $\\alpha = 2/255$ for 100 attack steps. Even as $\\epsilon$ increases, the actual intensity of the attack struggles to rise further due to the small $\\alpha$. However, this does not impact the relative robustness of the models. We will consider replacing this figure with experimental results where $\\alpha$ increases proportionally with $\\epsilon$.\n\n**Q3**: A detailed and explicit explanation for the intuition or mechanism of why the proposed method can endow DNNs with robustness against various types of adversarial attacks. \n\n**A3**: We appreciate your valuable advice.\n\n- **Intuitions** for why CausalDiff works:\nInspired by the robust reasoning abilities of humans, our goal is to construct a **robust feature extractor** that captures label-causative features resistant to perturbations and uses these features to predict label $Y$. Thus, such robust features should only focus on the semantic factor regardless of any type of adversarial perturbations. **If attackers want to succeed, they need to impact the image's semantics, which is more challenging**, because it requires a larger $\\epsilon$-budget. Consequently, our method achieves better robustness.\n\n- **Evidence** for Why CausalDiff works: Results show that CausalDiff smartly finds the robust feature $S$ for robust prediction. Specifically,\n  - as depicted in the case of a horse shown in Figure 1 (right), we visualized images conditioned on $S$ (with $Z$ masked) and on $Z$ (with $S$ masked) using our CausalDiff. Surprisingly, **$S$ captures the general semantic concept of a horse for this case**, even from just the head of the horse, while $Z$ contains details like the horse’s skin color.  \n  - as demonstrated in Figure 4, we visualize the feature space. We found that **$S$ aligns with human semantic understanding of categories**, that is semantically similar categories (e.g., cat and dog) are also close in the $S$ space. \n\n**Q4**: The proposed method would introduce more cost in time. \n\n**A4**: Unlike the image generation task, which requires traversing all timesteps (e.g., 1000 steps) to generate an image from Gaussian noise, our CausalDiff method only needs to sample $N_t$ timesteps to estimate the likelihood via ELBO for inference. Empirically, we have found that $N_t = 10$ is sufficient.\n\nRegarding a **detailed discussion of inference efficiency, please refer to the global response.**\n\n\n**Q5**: The second and third paragraphs of the Introduction lack references. \n\n**A5**: Thank you for your reminder; we will add the relevant references immediately.\n\n\n**Q6**: If we can abstract s* and z* using CausalDiff, we can further perform a distribution exploration to promote the robustness for OOD generalization.\n\n**A6**: We appreciate your valuable advice. We also believe that CausalDiff should naturally address out-of-distribution (OOD) issues if it is trained with a huge amount of data across various domains, as it eliminates the spurious correlation between $z^*$ and label $y$. \n\nAs shown in Table 3, we have tested the robustness against fog corruption for traffic sign recognition and observed the potential for improving OOD robustness. According to our experimental observations (e.g., visualization of feature space in Figure 4 and case study in Figure 8), $s^*$ learns semantics aligned with human perception, which should be robust to distribution shifts. In the future, we would like to extend our model on more data and aim to improve robustness to distribution and adversarial attacks simultaneously.\n\n[1] Robust Classification via a Single Diffusion Model, ICML 2024\n\n[2] Diffusion Models for Adversarial Purification, ICML 2022\n\nWe appreciate your efforts and are open to further discussion if you have any additional concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "afd4KhiujZ", "reviewer": "Reviewer_pU16", "summary": "The paper introduces an adversarial defense method, CausalDiff, which leverages a causal model of robustness combined with a diffusion model to learn label-relevant causative representations. The effectiveness of CausalDiff is specifically demonstrated against unseen attacks, where it surpasses existing adversarial defense baselines on several image classification datasets.", "questions": "Q1: Could you provide further insight into the source of CausalDiff’s ability to generalize to unseen attacks? For example, what attributes of the *essential factor* for benign images enable its generalizability to unseen attacks, and are there any specific constraints regarding the types of unseen attacks?\n\nQ2: Could the author further explain the reason for the opposite optimization direction $(1 - \\lambda) I(X;S,Z)$ in Eq. 5? E.g., what is the relationship between the motivation of avoiding \"X containing too many unimportant details\" (line 196) and the \"insensitivity\" for $-\\lambda I(X;S,Z)$ in Fig. 2? These statements seem to imply the existence of a feature that is neither \"essential\" (i.e., S) nor \"non-essential but important for reconstructing X\" (i.e., Z). This kind of feature appears crucial for \"sensitivity\" but is not fully discussed in the paper.\n\nQ3: Any abation study on the parameter $\\lambda$ used in Eq.5, which typically plays a crucial role in IB-style objectives?\n\nQ4: Other minor typos. \n\n- There is a citation format error in line 21, and a method name mistake (with / without) in line 124.", "limitations": "- The defense performance is superior against adversarial attacks with  L_p  norm constraints. However, the effectiveness of the proposed method against unbounded attacks (e.g., semantic attacks[1]) remains unclear.\n- Considering the high computational cost and training budget associated with diffusion models tailored to a single image classifier, CausalDiff may lack practicality. Strategies to reduce these costs should be considered. For instance, could a smaller diffusion model be trained to replace the two different diffusion models (unconditional and conditional) through model distillation?\n\n[1] Ghiasi, Amin, Ali Shafahi, and Tom Goldstein. \"BREAKING CERTIFIED DEFENSES: SEMANTIC ADVERSARIAL EXAMPLES WITH SPOOFED ROBUSTNESS CERTIFICATES.\" *International Conference on Learning Representations*.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well-motivated, presenting a novel framework that extracts the disentangled essential features of an image, achieved by first training a diffusion model and then inferring from it.\n- The pilot study is easy to understand, and the results persuasively demonstrate the effectiveness of the proposed causality-inspired defense framework.\n- A comprehensive comparison with baselines of adversarial defense methods, especially those based on Diffusion Models, makes the results convincing.", "weaknesses": "- Limited discussion about related works. To my knowledgement, neither CausalAdv and DICE limits their modeling of adversarial distribution to a certain type of attack. Therefore, it is necessary to discuss how the proposed method differs from these important baselines in terms of causal modeling. \n- The writing can be improved. The challenges listed from line 53 to 66 could be clearer by focusing on highlighting the actual problems that are solved by the proposed method, rather than just listing how the method is implemented. For example, challenge (2) could be clarified by explaining that applying a *Causal Information Bottleneck* objective aims to minimize the essential factors to their minimal necessary extent.\n- The empirical results of the pilot study are convincing. However, the insight into how to extract the desired features (i.e., S, Z) appears unclear, as detailed in Q2-Q3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We appreciate the time and effort of the reviewer. In response to the issues raised in the review, we offer the following replies:**\n\n**For Weakness 1**: \n\n- **Similarities**:\nBoth CausalAdv [1] and DICE [2], like our CausalDiff, model the generative mechanisms of clean data, with causal relationships defined as $S \\rightarrow X \\leftarrow Z$, and $S \\rightarrow Y$.\n\n- **Differences**:\nThe defensive strategies vary because they require adversarial examples generated by a specific attack during training, while our CausalDiff does not. Specifically,\n  - CausalAdv [1]: The causal graph explicitly models adversarial perturbations $E$ thus adversarial examples generated by a specific attack are required during training to identify variable $E$.\n  - DICE [2]: In the Pipeline of domain-attack invariant causal learning, DICE is trained on adversarial examples to eliminate spurious correlations between confounders $V$ and labels $Y$ that arise from specific attack behaviors used during training. \n\n**For Weakness 2**: Yes, thank you for your suggestion. We will revise this paragraph to illustrate better the key challenges solved by the proposed method.\n\n**For Question 1**: \n- **What the label-causative feature learned**:\n  - **Visualization of cases**: In Figure 1 (right), we separately visualize the images conditioned on $S$ (with $Z$ masked) and on $Z$ (with $S$ masked) using our CausalDiff. Interestingly, $S$ captures the general semantic concept of a horse, even from just the image of the head, while $Z$ contains the style like skin color. More cases found in Appendix C.3.\n  - **Visualization of feature space**: As shown in Figure 4, the label-causative feature $S$ aligns with the human semantic understanding of categories. Semantically similar categories, such as cats and dogs, are proximate in the $S$ space.\n\n- **Why CausalDiff works**:\n  - Inspired by the robust reasoning abilities of humans, CausalDiff focuses on the semantic factors regardless of any type of adversarial perturbations. If attackers wish to succeed, they must impact the image's semantics, which is inherently more challenging and requires a larger $\\epsilon$-budget. Consequently, our method exhibits enhanced robustness.\n\n- **Attack Types**: We consider CausalDiff to be highly robust against various adversarial attacks. It may face challenges with unseen corruptions such as rotation and translation while it actually extends beyond the scope of our paper.\n\n**For Question 2**: The two opposite optimization directions $(1-\\lambda) I(X;S,Z)$ in Eq.(5) ensure that the mutual information $I(X;S,Z)$ is **optimized to an appropriate level at the information bottleneck**, allowing $S$ and $Z$ to neither be independent of $X$ nor overly sensitive, thus making $S$ and $Z$ appropriate semantic abstractions of $X$.\n\n- The **positive term** $I(X;S,Z)$ aims to encourage $S$ and $Z$ to adequately represent the information in $X$. It is derived from the goal of maximizing the mutual information between observed data and latent variables in Eq.(3).\n- The **negative term** $\\lambda I(X;S,Z)$ aims to avoid exactly matching all details of $X$ and instead encourages $S$ and $Z$ to learn condensed, abstract semantics of $X$. (sorry for the typo in 196, We intended to express \"To avoid S and Z containing too many unimportant details of X.\")\n\nRegarding the causal modeling, any additional information in $X$ beyond what is captured by $S$ and $Z$ would typically be modeled by **exogenous variables** in a causal model, which is generally not explicitly represented.\n\n**For Question 3**: We evaluated the robustness of our model against three white-box attacks (APGD-ce, APGD-t, FAB-t) as $\\lambda$ varied. As $\\lambda$ increases for insensitive to pixel perturbation in $X$, model robustness is initially improved. However, when $\\lambda$ reaches 0.5, it begins to degrade the feature information in latent factors $s$ and $z$, resulting in decreased robustness.\n\n|$\\lambda$|5e-3|1e-2|5e-2|1e-1|5e-1|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|Robustness(%)|82.81|85.16|85.74|86.52|84.38|\n\n**For Question 4**: Thank you for pointing these out. We will fix these typos immediately.\n\n**For Limitation 1**: Facing unbound attacks [4], all models exhibit significant vulnerabilities, but our model shows a slight advantage (+2.7%) compared to adversarial training. \n\n|Method|Clean Accuracy(%)|Robust Accuracy(%)|\n|-|:-:|:-:|\n|VGG 16|93.6|5.7|\n|Adversarial Training|87.3|8.4|\n|CausalDiff|90.23|11.1|\n\nTypically, unbound attacks may not be prevalent in robustness evaluations because once the perturbation magnitude is large, it becomes easily detectable by defenders. This contradicts the basic assumptions of adversarial examples, which are intended to be subtle and undetectable.\n\n**For Limitation 2**: We appreciate your perspective. Since the primary computational overhead of CausalDiff occurs during causal factor inference by estimating likelihood on 10 timesteps, we implemented a **Distilled CausalDiff** by modifying the last layer of the diffusion process to predict noise for 10 timesteps in a single operation, as illustrated in Figure 1 of the one-page PDF. \n\nThe Distilled CausalDiff requires only about 13% of the original inference time while maintaining 95% of the performance of the original CausalDiff, **achieving 82.55% robustness—still more robust than state-of-the-art methods**. For a more details please refer to the global comment.\n\n**We appreciate your efforts and are open to further discussion if you have any additional concerns.**\n\n[1] CausalAdv: Adversarial Robustness through the Lens of Causality, ICLR 2022\n\n[2] DICE: Domain-attack Invariant Causal Learning for Improved Data Privacy Protection and Adversarial Robustness, KDD 2022\n\n[3] beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework, ICLR 2017\n\n[4] Semantic Adversarial Examples, CVPR 2018", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**We appreciate the time and effort of all the reviewers.**

Regarding the inference efficiency mentioned by the reviewers, we provide a detailed speed test in Appendix C.4, evaluating the inference time of our CausalDiff and baselines. We will include a discussion on efficiency in the methods section and link to the detailed analysis of efficiency in the appendix.

We have implemented a distilled version of CausalDiff (marked as Distilled CausalDiff) by modifying the last layer of the diffusion process to predict noise for 10 timesteps in a single operation (as illustrated in Figure 1 of the uploaded one-page PDF).  The comparisons between our methods (CausalDiff and Distilled CasualDiff) and the representative baselines including Robust Diffusion Classifier (RDC) [1] that has SOTA robustness on unseen attacks, in terms of inference time (for a single sample on an NVIDIA A6000 GPU) and average robustness against unseen attacks (consistent with the settings in Table 2) are as follows:


| Method               | CausalDiff | Distilled CausalDiff | RDC [1] | Distilled RDC [1] | DiffPure [2] | WRN-70-16 |
|----------------------|:------------:|:----------------------:|:---------:|:-------------------:|:--------------:|:-----------:|
| Inference Time (s)   | 4.97       | 0.67                 | 19.37   | 15.88             | 2.22         | 0.01      |
| Avg Robustness (%)   | **86.39**      | 82.55                | 82.38   | 78.85             | 47.20        | 0.00      |


**Discussion**: 
- We must acknowledge that addressing unseen attacks incurs certain costs. However, when **considering both computational complexity and robustness, our (Distilled) CausalDiff maintains strong competitiveness**, compared to state-of-the-art methods. For instance, compared to the RDC method, our approach requires only about one-quarter of the inference time to achieve superior robustness.
- The **distilled CausalDiff** requires only about 13% of the original inference time while maintaining 95% of the performance of the original CausalDiff, **achieving 82.55% robustness—still more robust than state-of-the-art methods**.
- Regarding **training costs**, training CausalDiff is comparable to training a standard diffusion model. Modeling a causal structure does not introduce additional training overhead but helps us model the generative mechanisms of genuine data. If CausalDiff is trained with tremendous data (similar to StableDiffusion), we would expect that this classifier will be robust in handling unseen attacks. It does not need to construct adversarial examples of all possible types of attacks, and incorporate them into training, which is costly.

[1] Robust Classification via a Single Diffusion Model, ICML 2024

[2] Diffusion Models for Adversarial Purification, ICML 2022

Meta-review: This paper develops a causal diffusion framework (CausalDiff) to improve the adversarial robustness against unseen attacks. A causal information bottleneck is introduced to disentangle label-causative factors and non-causative ones. The effectiveness of CausalDiff is well illustrated by the pilot study. The experimental results are promising, where CausalDiff achieves state-of-the-art robustness under various settings.

All the reviewers found their concerns were adequately addressed after the rebuttal. I therefore recommend acceptance of this paper. I encourage the authors to revise the paper according to the reviewers' comments.
Decision: Accept (poster)