Title: Carefully Blending Adversarial Training and Purification Improves Adversarial Robustness
Venue: NeurIPS 2024
Accepted: no
Keywords: adversarial robustness

adversarial training

adversarial purification

generative purification

internal representation
Abstract: In this work, we propose a novel adversarial defence mechanism for image classification - *CARSO* - blending the paradigms of *adversarial training* and *adversarial purification* in a synergistic robustness-enhancing way. The method builds upon an adversarially-trained classifier, and learns to map its *internal representation* associated with a potentially perturbed input onto a distribution of tentative *clean* reconstructions. Multiple samples from such distribution are classified by the same adversarially-trained model, and an aggregation of its outputs finally constitutes the *robust prediction* of interest. Experimental evaluation by a well-established benchmark of strong adaptive attacks, across different image datasets, shows that *CARSO* is able to defend itself against adaptive *end-to-end* *white-box* attacks devised for stochastic defences. Paying a modest *clean* accuracy toll, our method improves by a significant margin the *state-of-the-art* for CIFAR-10, CIFAR-100, and TinyImageNet-200 $\ell_\infty$ robust classification accuracy against AutoAttack.

Abstract

In this work, we propose a novel adversarial defence mechanism for image classification - CARSO - blending the paradigms of adversarial training and adversarial purification in a synergistic robustness-enhancing way. The method builds upon an adversarially-trained classifier, and learns to map its internal representation associated with a potentially perturbed input onto a distribution of tentative clean reconstructions. Multiple samples from such distribution are classified by the same adversarially-trained model, and an aggregation of its outputs finally constitutes the robust prediction of interest. Experimental evaluation by a well-established benchmark of strong adaptive attacks, across different image datasets, shows that CARSO is able to defend itself against adaptive end-to-end white-box attacks devised for stochastic defences. Paying a modest clean accuracy toll, our method improves by a significant margin the state-of-the-art for CIFAR-10, CIFAR-100, and TINYIMAGENET-200 ℓ ∞ robust classification accuracy against AUTOATTACK.

1 Introduction

Vulnerability to adversarial attacks [8, 57] i.e. the presence of inputs, usually crafted on purpose,

capable of catastrophically altering the behaviour of high-dimensional models [9] - constitutes a major 16

hurdle towards ensuring the compliance of deep learning systems with the behaviour expected by 17

modellers and users, and their adoption in safety-critical scenarios or tightly-regulated environments. 18

This is particularly true for adversariallyperturbed inputs, where a norm-constrained perturbation 19

20

often hardly detectable by human inspection [48, 5] - is added to an otherwise legitimate input, with

the intention of eliciting an anomalous response [34]. 21

Given the widespread nature of the issue [30], and the serious concerns raised about the safety and 22 reliability of data-learnt models in the lack of an appropriate mitigation [7], adversarial attacks have 23 been extensively studied. Yet, obtaining generally robust machine learning ( ML ) systems remains a 24 longstanding issue, and a major open challenge. 25

Research in the field has been driven by two opposing, yet complementary, efforts. On the one 26

hand, the study of failure modes in existing models and defences, with the goal of understanding 27

their origin and developing stronger attacks with varying degrees of knowledge and control over the 28

target system [57, 21, 44, 60]. On the other hand, the construction of increasingly capable defence 29

mechanisms. Although alternatives have been explored [15, 59, 11, 68], most of the latter is based on 30

adequately leveraging adversarial training [21, 42, 58, 49, 23, 31, 54, 62, 18, 47], i.e. training a ML 31

model on a dataset composed of (or enriched with) adversarially-perturbed inputs associated with 32

their correct, pre-perturbation labels. In fact, adversarial training has been the only technique capable 33

of consistently providing an acceptable level of defence [24], while still incrementally improving up 34

to the current state-of-the-art [18, 47]. 35

Another defensive approach is that of adversarial purification [53, 66], where a generative model is 36 used - similarly to denoising - to recover a perturbation-free version of the input before classification 37 is performed. Nonetheless, such attempts have generally fallen short of expectations due to inherent 38 limitations of the generative models used in early attempts [45], or due to decreases in robust 39 accuracy 1 when attacked end-to-end [25] - resulting in subpar robustness if the defensive structure is 40 known to the adversary [60]. More recently, the rise of diffusion-based generative models [28] and 41 their use for purification have enabled more successful results of this kind [45, 13] - although at the 42 cost of much longer training and inference times, and a much brittler robustness evaluation [13, 38]. 43

44

45

46

47

48

49

50

51

52

53

54

In this work, we design a novel adversarial defence for supervised image classification, dubbed CARSO ( i.e. , Counter-Adversarial Recall of Synthetic Observations). The approach relies on an adversarially-trained classifier (called hereinafter simply the classifier ), endowed with a stochastic generative model (called hereinafter the purifier ). Upon classification of a potentially-perturbed input, the latter learns to generate - from the tensor 2 of (pre)activations registered at neuron level in the former - samples from a distribution of plausible, perturbation-free reconstructions. At inference time, some of these samples are classified by the very same classifier , and the original input is robustly labelled by aggregating its many outputs. This method - to the best of our knowledge the first attempt to organically merge the adversarial training and purification paradigms - avoids the vulnerability pitfalls typical of the mere stacking of a purifier and a classifier [25], while still being able to take advantage of independent incremental improvements to adversarial training or generative modelling.

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

An empirical assessment 3 of the defence in the ℓ ∞ white-box setting is provided, using a conditional [56, 64] variational autoencoder [32, 50] as the purifier and existing state-of-the-art adversarially pre-trained models as classifiers. Such choices are meant to give existing approaches - and the adversary attacking our architecture end-to-end as part of the assessment - the strongest advantage possible. Yet, in all scenarios considered, CARSO improves significantly the robustness of the pre-trained classifier - even against attacks specifically devised to fool stochastic defences like ours. Remarkably, with a modest clean accuracy toll, our method improves by a significant margin the current state-of-the-art for CIFAR-10 [33], CIFAR-100 [33], and TINYIMAGENET-200 [14] ℓ ∞ robust classification accuracy against AUTOATTACK [17].

In summary, the paper makes the following contributions:

The description of CARSO, a novel adversarial defence method synergistically blending adversarial training and adversarial purification ;

A collection of relevant technical details fundamental to its successful training and use, originally developed for the purifier being a conditional variational autoencoder - but applicable to more general scenarios as well;

Experimental assessment of the method, against standardised benchmark adversarial attacks - showing higher robust accuracy w.r.t. to existing state-of-the-art adversarial training and purification approaches.

The rest of the manuscript is structured as follows. In section 2 we provide an overview of selected contributions in the fields of adversarial training and purification-based defences - with focus on image classification. In section 3, a deeper analysis is given of two integral parts of our experimental assessment: PGD adversarial training and conditional variational autoencoders. Section 4 is devoted to the intuition behind CARSO, its architectural description, and the relevant technical details that allow it to work. Section 5 contains details about the experimental setup, results, comments, and limitations. Section 6 concludes the paper and outlines directions of future development.

2 Related work

Adversarial training as a defence The idea of training a model on adversarially-generated examples as a way to make it more robust can be traced back to the very beginning of research in the area.

The seminal [57] proposes to perform training on a mixed collection of clean and adversarial data, 83 generated beforehand. 84

85

86

87

88

89

90

The introduction of FGSM [21] enables the efficient generation of adversarial examples along the training, with a single normalised gradient step. Its iterative counterpart PGD [42] - discussed in section 3 and Appendix A - significantly improves the effectiveness of adversarial examples produced, making it still the de facto standard for the synthesis of adversarial training inputs [24]. Further incremental improvements have also been developed, some focused specifically on robustness assessment ( e.g. adaptive-stepsize variants, as in [17]).

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

The most recent adversarial training protocols further rely on synthetic data to increase the numerosity of training datapoints [23, 49, 62, 18, 47], and adopt adjusted loss functions to balance robustness and accuracy [67] or generally foster the learning process [18]. The entire model architecture may also be tuned specifically for the sake of robustness enhancement [47]. At least some of such ingredients are often required to reach the current state-of-the-art in robust accuracy via adversarial training.

Purification as a defence Amongst the first attempts of purification-based adversarial defence, [25] investigates the use of denoising autoencoders [61] to recover examples free from adversarial perturbations. Despite its effectiveness in the denoising task, the method may indeed increase the vulnerability of the system when attacks are generated against it end-to-end . The contextually proposed improvement adds a smoothness penalty to the reconstruction loss, partially mitigating such downside [25]. Similar in spirit, [39] tackles the issue by computing the reconstruction loss between the last-layers representations of the frozen-weights attacked classifier, respectively receiving, as input, the clean and the tentatively denoised example.

In [52], Generative Adversarial Networks (GANs) [22] learnt on clean data are used at inference time to find a plausible synthetic example - close to the perturbed input - belonging to the unperturbed data manifold. Despite encouraging results, the delicate training process of GANs and the existence of known failure modes [70] limit the applicability of the method. More recently, a similar approach [27] employing energy-based models [37] suffered from poor sample quality [45].

Purification approaches based on (conditional) variational autoencoders include [29] and [53]. Very recently, a technique combining variational manifold learning with a test-time iterative purification procedure has also been proposed [65].

Finally, already-mentioned techniques relying on score[66] and diffusionbased [45, 13] models have also been developed, with generally favourable results - often balanced in practice by longer training and inference times, and a much more fragile robustness assessment [13, 38].

3 Preliminaries

PGD adversarial training The task of finding model parameters robust to adversarial perturbations is framed by [42] as a min-max optimisation problem seeking to minimise adversarial risk . The inner optimisation ( i.e. , the generation of worst-case adversarial examples) is solved by an iterative algorithm Projected Gradient Descent - interleaving gradient ascent steps in input space with the eventual projection on the shell of an ϵ -ball centred around an input datapoint, thus imposing a perturbation strength constraint.

In this manuscript, we will use the shorthand notation ϵ p to denote ℓ p norm-bound perturbations of maximum magnitude ϵ .

The formal details of such method are provided in Appendix A.

(Conditional) variational autoencoders Variational autoencoders ( VAE s) [32, 50] allow the learning from data of approximate generative latent-variable models of the form p ( x , z ) = p ( x | z ) p ( z ) , whose likelihood and posterior are approximately parameterised by deep artificial neural networks ( ANN s). The problem is cast as the maximisation of a variational lower bound.

In practice, optimisation is performed iteratively - on a loss function given by the linear mixture of data-reconstruction loss and empirical KL divergence w.r.t. a chosen prior, computed on mini-batches of data.

Conditional Variational Autoencoders [56, 64] extend VAE s by attaching a conditioning tensor c -132 expressing specific characteristics of each example - to both x and z during training. This allows the 133 learning of a decoder model capable of conditional data generation. 134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

Further details on the functioning of such models are given in Appendix B.

4 Structure of CARSO

The core ideas informing the design of our method are driven more by first principles rather than arising from specific contingent requirements. This section discusses such ideas, the architectural details of CARSO, and a group of technical aspects fundamental to its training and inference processes.

4.1 Architectural overview and principle of operation

From an architectural point of view, CARSO is essentially composed of two ANN models - a classifier and a purifier - operating in close synergy. The former is trained on a given classification task, whose inputs might be adversarially corrupted at inference time. The latter learns to generate samples from a distribution of potential input reconstructions, tentatively free from adversarial perturbations. Crucially, the purifier has only access to the internal representation of the classifier - and not even directly to the perturbed input - to perform its task.

During inference, for each input, the internal representation of the classifier is used by the purifier to synthesise a collection of tentatively unperturbed input reconstructions. Those are classified by the same classifier , and the resulting outputs are aggregated into a final robust prediction .

There are no specific requirements for the classifier, whose training is completely independent of the use of the model as part of CARSO. However, training it adversarially improves significantly the clean accuracy of the overall system, allowing it to benefit from established adversarial training techniques.

The purifier is also independent of specific architectural choices, provided it is capable of stochastic conditional data generation at inference time, with the internal representation of the classifier used as the conditioning set.

In the rest of the paper, we employ a state-of-the-art adversarially pre-trained WIDERESNET model as the classifier, and a purpose-built conditional variational autoencoder as the purifier, the latter operating decoder-only during inference. Such choice was driven by the deliberate intention to assess the adversarial robustness of our method in its worst-case scenario against a white-box attacker, and with the least advantage compared to existing approaches based solely on adversarial training.

In fact, the decoder of a conditional V AE allows for exact algorithmic differentiability [6] w.r.t. its conditioning set, thus averting the need for backward-pass approximation [2] in generating end-to-end adversarial attacks against the entire system, and preventing (un)intentional robustness by gradient obfuscation [2]. The same cannot be said [13] for more capable and modern purification models, such as those based e.g. on diffusive processes, whose robustness assessment is still in the process of being understood [38].

A downside of such choice is represented by the reduced effectiveness of the decoder in the synthesis of complex data, due to well-known model limitations. In fact, we experimentally observe a modest increase in reconstruction cost for non-perturbed inputs, which in turn may limit the clean accuracy of the entire system. Nevertheless, we defend the need for a fair and transparent robustness evaluation, such as the one provided by the use of a V AE-based purifier, in the evaluation of any novel architectureagnostic adversarial defence technique.

A diagram of the whole architecture is shown in Figure 1, and its detailed principles of operation are recapped below.

Training At training time, adversarially-perturbed examples are generated against the classifier , and fed to it. The tensors containing the classifier (pre)activations across the network are then extracted. Finally, the conditional VAE serving as the purifier is trained on perturbation-free input reconstruction, conditional on the corresponding previously extracted internal representations, and using pre-perturbation examples as targets.

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

yi

discard output

f(; 0)

yrob

aggregation

×

C

int.rep.

robust prediction

cat

po(-l; 0)

C

f(c; 0)

～sample

cat

+α

po(-1;0)

Upon completion of the training process, the encoder network may be discarded as it will not be used for inference.

Inference The example requiring classification is fed to the classifier . Its corresponding internal representation is extracted and used to condition the generative process described by the decoder of the VAE . Stochastic latent variables are repeatedly sampled from the original priors, which are given by an i.i.d. multivariate Standard Normal distribution. Each element in the resulting set of reconstructed inputs is classified by the same classifier , and the individually predicted class logits are aggregated. The result of such aggregation constitutes the robust prediction of the input class.

Remarkably, the only link between the initial potentially-perturbed input and the resulting purified reconstructions (and thus the predicted class) is through the internal representation of the classifier, which serves as a featurisation of the original input. The whole process is exactly differentiable endto-end , and the only potential hurdle to the generation of adversarial attacks against the entire system is the stochastic nature of the decoding - which is easily tackled by Expectation over Transformation [3].

4.2 A first-principles justification

If we consider a trained ANN classifier, subject to a successful adversarial attack by means of a slightly perturbed example, we observe that - both in terms of ℓ p magnitude and human perception - a small variation on the input side of the network is amplified to a significant amount on the output side, thanks to the layerwise processing by the model. Given the deterministic nature of such processing at inference time, we speculate that the trace obtained by sequentially collecting the (pre)activation values within the network, along the forward pass, constitutes a richer characterisation of such an amplification process compared to the knowledge of the input alone. Indeed, as we do, it is possible to learn a direct mapping from such featurisation of the input, to a distribution of possible perturbation-free input reconstructions - taking advantage of such characterisation.

4.3 Hierarchical input and internal representation encoding

Training a conditional VAE requires [56] that the conditioning set c is concatenated to the input x before encoding occurs, and to the sample of latent variables z right before decoding. The same is

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

also true, with the suitable adjustments, for any conditional generative approach where the target and the conditioning set must be processed jointly.

In order to ensure the usability and scalability of CARSO across the widest range of input data and classifier models, we propose to perform such processing in a hierarchical and partially disjoint fashion between the input and the conditioning set. In principle, the encoding of x and c can be performed by two different and independent subnetworks, until some form of joint processing must occur. This allows to retain the overall architectural structure of the purifier, while having finer-grained control over the inductive biases [43] deemed the most suitable for the respective variables.

In the experimental phase of our work, we encode the two variables independently. The input is compressed by a multilayer convolutional neural network (CNN). The internal representation which in our case is composed of differently sized multi-channel images - is processed layer by layer by independent multilayer CNNs (responsible for encoding local information), whose flattened outputs are finally concatenated and compressed by a fully-connected layer (modelling inter-layer correlations in the representation). The resulting compressed input and conditioning set are then further concatenated and jointly encoded by a fully-connected network (FCN).

In order to use the V AE decoder at inference time, the entire compression machinery for the conditioning set must be preserved after training, and used to encode the internal representations extracted. The equivalent input encoder may be discarded instead.

4.4 Adversarially-balanced batches

Training the purifier in representation-conditional input reconstruction requires having access to adversarially-perturbed examples generated against the classifier, and to the corresponding clean data. Specifically, we use as input a mixture of clean and adversarially perturbed examples, and the clean input as the target.

Within each epoch, the training set of interest is shuffled [51, 10], and only a fixed fraction of each resulting batch is adversarially perturbed. Calling ϵ the maximum ℓ p perturbation norm bound for the threat model against which the classifier was adversarially pre-trained, the portion of perturbed examples is generated by an even split of FGSM ϵ / 2 , PGD ϵ / 2 , FGSM ϵ , and PGD ϵ attacks.

Any smaller subset of attack types and strengths, or a detailedly unbalanced batch composition, always experimentally results in a worse performing purification model. More details justifying such choice are provided in Appendix C.

4.5 Robust aggregation strategy

At inference time, many different input reconstructions are classified by the classifier , and the respective outputs concur to the settlement of a robust prediction .

Calling l α i the output logit associated with class i ∈ { 1 , . . . , C } in the prediction by the classifier on sample α ∈ { 1 , . . . , N } , we adopt the following aggregation strategy:

with P i being the aggregated probability of membership in class i , Z a normalisation constant such that ∑ C i =1 P i = 1 , and e Euler's number.

Such choice produces a robust prediction much harder to take over in the event that an adversary selectively targets a specific input reconstruction. A heuristic justification for this property is given in Appendix D.

5 Experimental assessment

Experimental evaluation of our method is carried out in terms of robust and clean image classification accuracy within three different scenarios ( a , b and c ), determined by the specific classification task.

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

The white-box threat model with a fixed ℓ ∞ norm bound is assumed throughout, as it generally constitutes the most demanding setup for adversarial defences.

5.1 Setup

Data The CIFAR-10 [33] dataset is used in scenario (a) , the CIFAR-100 [33] dataset is used in scenario (b) , whereas the TINYIMAGENET-200 [14] dataset is used in scenario (c) .

Architectures A WIDERESNET-28-10 model is used as the classifier , adversarially pre-trained on the respective dataset - the only difference between scenarios being the number of output logits: 10 in scenario (a) , 100 in scenario (b) , and 200 in scenario (c) .

The purifier is composed of a conditional VAE, processing inputs and internal representations in a partially disjoint fashion, as explained in subsection 4.3. The input is compressed by a two-layer CNN; the internal representation is instead processed layerwise by independent CNNs (three-layered in scenarios (a) and (b) , four-layered in scenario (c) ) whose outputs are then concatenated and compressed by a fully-connected layer. A final two-layer FCN jointly encodes the compressed input and conditioning set, after the concatenation of the two. A six-layer deconvolutional network is used as the decoder.

More precise details on all architectures are given in Appendix E.

Outer minimisation In scenarios (a) and (b) , the classifier is trained according to [18]; in scenario (c) , according to [62]. Classifiers were always acquired as pre-trained models, using publicly available weights provided by the respective authors.

The purifier is trained on the VAE loss, using summed pixel-wise channel-wise binary cross-entropy as the reconstruction cost. Optimisation is performed by RADAM+LOOKAHEAD [41, 69] with a learning rate schedule that presents a linear warm-up, a plateau phase, and a linear annealing [55]. To promote the learning of meaningful reconstructions during the initial phases of training, the KL divergence term in the VAE loss is suppressed for an initial number of epochs. Afterwards, it is linearly modulated up to its actual value, during a fixed number of epochs ( β increase ) [26]. The initial and final epochs of such modulation are reported in Table 14.

Additional scenario-specific details are provided in Appendix E.

Inner minimisation ϵ ∞ = 8 / 255 is set as the perturbation norm bound.

Adversarial examples against the purifier are obtained, as explained in subsection 4.4, by FGSM ϵ / 2 , PGD ϵ / 2 , FGSM ϵ , and PGD ϵ , in a class-untargeted fashion on the cross-entropy loss. In the case of PGD, gradient ascent with a step size of α = 0 . 01 is used.

The complete details and hyperparameters of the attacks are described in Appendix E.

Evaluation In each scenario, we report the clean and robust test-set accuracy - the latter by means of AUTOATTACK [17] - of the classifier and the corresponding CARSO architecture.

For the classifier alone, the standard version of AUTOATTACK ( AA ) is used: i.e. , the worst-case accuracy on a mixture of AUTOPGD on the cross-entropy loss [17] with 100 steps, AUTOPGD on the difference of logits ratio loss [17] with 100 steps, FAB [16] with 100 steps, and the black-box SQUARE attack [1] with 5000 queries.

In the evaluation of the CARSO architecture, the number of reconstructed samples per input is set to 8 , the logits are aggregated as explained in subsection 4.5, and the output class is finally selected as the arg max of the aggregation. Due to the stochastic nature of the purifier , robust accuracy is assessed by a version of AUTOATTACK suitable for stochastic defences ( randAA ) - composed of AUTOPGD on the cross-entropy and difference of logits ratio losses, across 20 Expectation over Transformation (EOT) [3] iterations with 100 gradient ascent steps each.

Computational infrastructure All experiments were performed on an NVIDIA DGX A100 system. Training in scenarios (a) and (c) was run on 8 NVIDIA A100 GPUs with 40 GB of dedicated memory each; in scenario (b) 4 of such devices were used. Elapsed real training time for the purifier in all scenarios is reported in Table 1.

299

5.2 Results and discussion

An analysis of the experimental results is provided in the subsection that follows, whereas their 300 systematic exposition is given in Table 2. 301

Table 2: Clean (results in italic ) and adversarial (results in upright) accuracy for the different models and datasets used in the respective scenarios. The following abbreviations are used: Scen : scenario considered; AT/Cl : clean accuracy for the adversarially-pretrained model used as the classifier , when considered alone; C/Cl : clean accuracy for the CARSO architecture; AT/AA : robust accuracy (by the means of AUTOATTACK) for the adversarially-pretrained model used as the classifier , when considered alone; C/randAA : robust accuracy for the CARSO architecture, when attacked end-to-end by AUTOATTACK for randomised defences; Best AT/AA : best robust accuracy result for the respective dataset (by the means of AUTOATTACK), obtained by adversarial training alone (any model); Best P/AA : best robust accuracy result for the respective dataset (by the means of AUTOATTACK), obtained by adversarial purification (any model). Robust accuracies in round brackets are obtained using the PGD+EOT [38] pipeline, developed for diffusion-based purifiers. The best clean and robust accuracies per dataset are shown in bold . The clean accuracies for the models referred to in the Best columns are shown in Table 15 (in Appendix F).

Scenario (a) Comparing the robust accuracy of the classifier model used in scenario (a) [18] with that resulting from the inclusion of the same model in the CARSO architecture, we observe a +8 . 4% increase. This is counterbalanced by a -5 . 6% clean accuracy toll. The same version of CARSO further provides a +5 . 03 robustness increase w.r.t. the current best AT-trained model [47] that employs a ∼ 3 × larger RAWIDERESNET-70-16 model.

In addition, our method provides a remarkable +9 . 72% increase in robust accuracy w.r.t. to the best adversarial purification approach [40], a diffusion-based purifier. However, the comparison is not as straightforward. In fact, the paper [40] reports a robust accuracy of 78 . 12% using AUTOATTACK on the gradients obtained via the adjoint method [45]. As noted in [38], such evaluation (which uses the version of AUTOATTACK that is unsuitable for stochastic defences) leads to a large overestimation of the robustness of diffusive purifiers. As suggested in [38], the authors of [40] re-evaluate the robust accuracy according to a more suitable pipeline (PGD+EOT, whose hyperparameters are shown in Table 12), obtaining a much lower robust accuracy of 66 . 41% . Consequently, we repeat the same evaluation for CARSO and compare the worst-case robustness amongst the two. In line with typical AT methods, and unlike diffusive purification, the robustness of CARSO assessed by means of randAA is still lower w.r.t. than achieved by PGD+EOT.

Scenario (b) Moving to scenario (b) , CARSO achieves a robust accuracy increase of +27 . 47% w.r.t. the classifier alone [18], balanced by a 5 . 79% decrease in clean accuracy. Our approach also improves upon the robust accuracy of the best AT-trained model [62] (WIDERESNET-70-16) by 23 . 98% . In the absence of a reliable robustness evaluation by means of PGD+EOT for the best purification-based method [40], we still obtain a +20 . 25% increase in robust accuracy upon its (largely overestimated) AA result.

Scenario (c) In scenario (c) , CARSO improves upon the classifier alone [62] (which is also the best AT-based approach for TINYIMAGENET-200) by +22 . 26% . A significant clean accuracy toll is

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

imposed by the relative complexity of the dataset, i.e. -8 . 87% . In this setting, we lack any additional purification-based methods.

Assessing the impact of gradient obfuscation Although the architecture of CARSO is algorithmically differentiable end-to-end - and the integrated diagnostics of the randAA routines raised no warnings during the assessment - we additionally guard against the eventual gradient obfuscation [2] induced by our method by repeating the evaluation at ϵ ∞ = 0 . 95 , verifying that the resulting robust accuracy stays below random chance [12]. Results are shown in Table 3.

Table 3: Robust classification accuracy against AUTOATTACK, for ϵ ∞ = 0 . 95 , as a way to assess the (lack of) impact of gradient obfuscation on robust accuracy evaluation.

5.3 Limitations and open problems

In line with recent research aiming at the development of robust defences against multiple perturbations [20, 35], our method determines a decrease in clean accuracy w.r.t. the original model on which it is built upon - especially in scenario (c) as the complexity of the dataset increases. This phenomenon is partly dependent on the choice of a VAE as the generative purification model, a requirement for the fairest evaluation possible in terms of robustness.

Yet, the issue remains open: is it possible to devise a CARSO-like architecture capable of the same - if not better - robust behaviour, which is also competitively accurate on clean inputs? Potential avenues for future research may involve the development of CARSO-like architectures in which representation-conditional data generation is obtained by means of diffusion or score-based models. Alternatively, incremental developments aimed at improving the cross-talk between the purifier and the final classifier may be pursued.

Lastly, the scalability of CARSO could be strongly improved by determining whether the internal representation used in conditional data generation may be restricted to a smaller subset of layers, while still maintaining the general robustness of the method.

6 Conclusion

In this work, we presented a novel adversarial defence mechanism tightly integrating input purification, and classification by an adversarially-trained model - in the form of representation-conditional data purification. Our method is able to improve upon the current state-of-the-art in CIFAR-10, CIFAR100, and TINYIMAGENET ℓ ∞ robust classification, w.r.t. both adversarial training and purification approaches alone.

Such results suggest a new synergistic strategy to achieve adversarial robustness in visual tasks and 354 motivate future research on the application of the same design principles to different models and 355 types of data. 356

357

References

Maksym Andriushchenko et al. 'Square Attack: a query-efficient black-box adversarial attack via random 358 search'. In: 16th European Conference on Computer Vision . 2020. 359

Anish Athalye, Nicholas Carlini and David Wagner. 'Obfuscated Gradients Give a False Sense of Security: 360

Circumventing Defenses to Adversarial Examples'. In: Proceedings of the International Conference on 361 Machine Learning . 2018. 362

Anish Athalye et al. 'Synthesizing Robust Adversarial Examples'. In: Proceedings of the International 363 Conference on Machine Learning . 2018. 364

Emanuele Ballarin. ebtorch : Collection of PyTorch additions, extensions, utilities, uses and abuses . 365 2024. URL: https://github.com/emaballarin/ebtorch . 366

Vincent Ballet et al. 'Imperceptible Adversarial Attacks on Tabular Data'. In: Thirty-third Conference on 367 Neural Information Processing Systems, Workshop on Robust AI in Financial Services: Data, Fairness, 368 Explainability, Trustworthiness, and Privacy (Robust AI in FS) . 2019. 369

Atilim Gunes Baydin et al. 'Automatic differentiation in machine learning: a survey'. In: The Journal of 370 Machine Learning Research 18.153 (2018), pp. 1-43. 371

Battista Biggio and Fabio Roli. 'Wild patterns: Ten years after the rise of adversarial machine learning'. 372 In: Pattern Recognition 84 (2018), pp. 317-331. 373

Battista Biggio et al. 'Evasion Attacks against Machine Learning at Test Time'. In: Proceedings of the 374 2013th European Conference on Machine Learning and Knowledge Discovery in Databases - Volume 375 Part III . 2013. 376

Luca Bortolussi and Guido Sanguinetti. Intrinsic Geometric Vulnerability of High-Dimensional Artificial 377 Intelligence . 2018. arXiv: 1811.03571 . 378

Léon Bottou. 'On-Line Algorithms and Stochastic Approximations'. In: On-Line Learning in Neural 379 Networks . Cambridge University Press, 1999. Chap. 2. 380

Ginevra Carbone et al. 'Robustness of Bayesian Neural Networks to Gradient-Based Attacks'. In: 381 Advances in Neural Information Processing Systems . 2020. 382

Nicholas Carlini et al. On Evaluating Adversarial Robustness . 2019. arXiv: 1902.06705 . 383

Huanran Chen et al. Robust Classification via a Single Diffusion Model . 2023. arXiv: 2305.15241 . 384

Patryk Chrabaszcz, Ilya Loshchilov and Frank Hutter. A Downsampled Variant of ImageNet as an 385 Alternative to the CIFAR datasets . 2017. arXiv: 1707.08819 . 386

Moustapha Cisse et al. 'Parseval Networks: Improving Robustness to Adversarial Examples'. In: Pro387 ceedings of the International Conference on Machine Learning . 2017. 388

Francesco Croce and Matthias Hein. Minimally distorted Adversarial Examples with a Fast Adaptive 389 Boundary Attack . 2020. arXiv: 1907.02044 . 390

Francesco Croce and Matthias Hein. 'Reliable Evaluation of Adversarial Robustness with an Ensemble of 391 Diverse Parameter-free Attacks'. In: Proceedings of the International Conference on Machine Learning . 392 2020. 393

Jiequan Cui et al. Decoupled Kullback-Leibler Divergence Loss . 2023. arXiv: 2305.13948 . 394

Gavin Weiguang Ding, Luyu Wang and Xiaomeng Jin. AdverTorch v0.1: An Adversarial Robustness 395 Toolbox based on PyTorch . 2019. arXiv: 1902.07623 . 396

Hadi M. Dolatabadi, Sarah Erfani and Christopher Leckie. ' ℓ ∞ -Robustness and Beyond: Unleashing 397 Efficient Adversarial Training'. In: 18th European Conference on Computer Vision . 2022. 398

Ian Goodfellow, Jonathon Shlens and Christian Szegedy. 'Explaining and Harnessing Adversarial Ex399 amples'. In: International Conference on Learning Representations . 2015. 400

Ian Goodfellow et al. 'Generative Adversarial Nets'. In: Advances in Neural Information Processing 401 Systems . 2014. 402

Sven Gowal et al. 'Improving Robustness using Generated Data'. In: Advances in Neural Information 403 Processing Systems . 2021. 404

Sven Gowal et al. Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial 405 Examples . 2020. arXiv: 2010.03593 . 406

Shixiang Gu and Luca Rigazio. 'Towards Deep Neural Network Architectures Robust to Adversarial 407 Examples'. In: Workshop Track of the International Conference on Learning Representations . 2015. 408

Irina Higgins et al. 'beta-V AE: Learning Basic Visual Concepts with a Constrained Variational Frame409 work'. In: International Conference on Learning Representations . 2017. 410

Mitch Hill, Jonathan Mitchell and Song-Chun Zhu. 'Stochastic Security: Adversarial Defense Using Long411

Run Dynamics of Energy-Based Models'. In: International Conference on Learning Representations . 412 2021. 413

Chin-Wei Huang, Jae Hyun Lim and Aaron C Courville. 'A Variational Perspective on Diffusion-Based 414

Generative Models and Score Matching'. In: Advances in Neural Information Processing Systems . 2021. 415

Uiwon Hwang et al. 'PuVAE: A Variational Autoencoder to Purify Adversarial Examples'. In: IEEE 416 Access . 2019. 417

Andrew Ilyas et al. 'Adversarial Examples Are Not Bugs, They Are Features'. In: Advances in Neural 418 Information Processing Systems . 2019. 419

Xiaojun Jia et al. 'LAS-AT: Adversarial Training With Learnable Attack Strategy'. In: Proceedings of the 420 IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2022. 421

Diederik P. Kingma and Max Welling. 'Auto-Encoding Variational Bayes'. In: International Conference 422 on Learning Representations . 2014. 423

Alex Krizhevsky. 'Learning Multiple Layers of Features from Tiny Images'. In: 2009. 424

Alexey Kurakin, Ian J. Goodfellow and Samy Bengio. 'Adversarial Examples in the Physical World'. In: 425 Artificial Intelligence Safety and Security (2018). 426

Cassidy Laidlaw, Sahil Singla and Soheil Feizi. 'Perceptual Adversarial Robustness: Defense Against 427 Unseen Threat Models'. In: International Conference on Learning Representations . 2021. 428

Yann LeCun and Corinna Cortes. The MNIST handwritten digit database . 2010. 429

Yann LeCun et al. 'A tutorial on energy-based learning'. In: Predicting structured data . MIT Press, 2006. 430 Chap. 1. 431

Minjong Lee and Dongwoo Kim. 'Robust Evaluation of Diffusion-Based Adversarial Purification'. In: 432 International Conference on Computer Vision . 2024. 433

Fangzhou Liao et al. 'Defense Against Adversarial Attacks Using High-Level Representation Guided 434 Denoiser'. In: IEEE Conference on Computer Vision and Pattern Recognition . 2018. 435

Guang Lin et al. Robust Diffusion Models for Adversarial Purification . 2024. arXiv: 2403.16067 . 436

Liyuan Liu et al. 'On the Variance of the Adaptive Learning Rate and Beyond'. In: International 437 Conference on Learning Representations . 2020. 438

Aleksander Madry et al. 'Towards Deep Learning Models Resistant to Adversarial Attacks'. In: Interna439 tional Conference on Learning Representations . 2018. 440

Tom M. Mitchell. The Need for Biases in Learning Generalizations . Tech. rep. New Brunswick, NJ: 441 Rutgers University, 1980. 442

Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi and Pascal Frossard. 'DeepFool: A Simple and 443 Accurate Method to Fool Deep Neural Networks'. In: IEEE Conference on Computer Vision and Pattern 444 Recognition . 2016. 445

Weili Nie et al. 'Diffusion Models for Adversarial Purification'. In: Proceedings of the International 446 Conference on Machine Learning . 2022. 447

Adam Paszke et al. 'PyTorch: An Imperative Style, High-Performance Deep Learning Library'. In: 448 Advances in Neural Information Processing Systems . 2019. 449

ShengYun Peng et al. Robust Principles: Architectural Design Principles for Adversarially Robust CNNs . 450 2023. 451

Yao Qin et al. 'Imperceptible, Robust, and Targeted Adversarial Examples for Automatic Speech Recog452 nition'. In: Proceedings of the International Conference on Machine Learning . 2019. 453

Sylvestre-Alvise Rebuffi et al. 'Data Augmentation Can Improve Robustness'. In: Advances in Neural 454 Information Processing Systems . 2021. 455

Danilo Jimenez Rezende, Shakir Mohamed and Daan Wierstra. 'Stochastic Backpropagation and Ap456 proximate Inference in Deep Generative Models'. In: Proceedings of the International Conference on 457 Machine Learning . 2014. 458

Herbert Robbins and Sutton Monro. 'A Stochastic Approximation Method'. In: The Annals of Mathemat459 ical Statistics 22.3 (1951), pp. 400-407. 460

Pouya Samangouei, Maya Kabkab and Rama Chellappa. 'Defense-GAN: Protecting Classifiers Against 461 Adversarial Attacks Using Generative Models'. In: International Conference on Learning Representations . 462 2018. 463

Changhao Shi, Chester Holtz and Gal Mishne. 'Online Adversarial Purification based on Self-supervised 464 Learning'. In: International Conference on Learning Representations . 2021. 465

Naman D Singh, Francesco Croce and Matthias Hein. Revisiting Adversarial Training for ImageNet: 466 Architectures, Training and Generalization across Threat Models . 2023. arXiv: 2303.01870 . 467

Leslie N. Smith. 'Cyclical Learning Rates for Training Neural Networks'. In: IEEE Winter Conference 468 on Applications of Computer Vision . 2017. 469

Kihyuk Sohn, Honglak Lee and Xinchen Yan. 'Learning Structured Output Representation using Deep 470 Conditional Generative Models'. In: Advances in Neural Information Processing Systems . 2015. 471

Christian Szegedy et al. 'Intriguing properties of neural networks'. In: International Conference on 472 Learning Representations . 2014. 473

Florian Tramèr and Dan Boneh. 'Adversarial Training and Robustness for Multiple Perturbations'. In: 474 Advances in Neural Information Processing Systems . 2019. 475

Florian Tramèr et al. 'Ensemble Adversarial Training: Attacks and Defenses'. In: International Conference 476 on Learning Representations . 2018. 477

Florian Tramèr et al. 'On Adaptive Attacks to Adversarial Example Defenses'. In: Advances in Neural 478 Information Processing Systems . 2020. 479

Pascal Vincent et al. 'Extracting and composing robust features with denoising autoencoders'. In: Inter480 national Conference on Machine Learning . 2008. 481

Zekai Wang et al. Better Diffusion Models Further Improve Adversarial Training . 2023. arXiv: 2303. 482 10130 . 483

Han Xiao, Kashif Rasul and Roland Vollgraf. Fashion-MNIST: a Novel Image Dataset for Benchmarking 484 Machine Learning Algorithms . 2017. arXiv: 1708.07747 . 485

Xinchen Yan et al. 'Attribute2Image: Conditional Image Generation from Visual Attributes'. In: Proceed486 ings of the European Conference on Computer Vision . 2016. 487

Zhaoyuan Yang et al. 'Adversarial Purification with the Manifold Hypothesis'. In: AAAI Conference on 488 Artificial Intelligence . 2024. 489

Jongmin Yoon, Sung Ju Hwang and Juho Lee. 'Adversarial Purification with Score-based Generative 490 Models'. In: Proceedings of the International Conference on Machine Learning . 2021. 491

Hongyang Zhang et al. 'Theoretically Principled Trade-off between Robustness and Accuracy'. In: 492 Proceedings of the International Conference on Machine Learning . 2019. 493

M. Zhang, S. Levine and C. Finn. 'MEMO: Test Time Robustness via Adaptation and Augmentation'. In: 494 Advances in Neural Information Processing Systems . 2022. 495

Michael Zhang et al. 'Lookahead Optimizer: k steps forward, 1 step back'. In: Advances in Neural 496 Information Processing Systems . 2019. 497

Zhaoyu Zhang, Mengyan Li and Jun Yu. 'On the Convergence and Mode Collapse of GAN'. In: SIG498 GRAPH Asia 2018 Technical Briefs . 2018. 499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

A On Projected Gradient Descent adversarial training

The task of determining model parameters θ ⋆ that are robust to adversarial perturbations is cast in [42] as a min-max optimisation problem seeking to minimise adversarial risk , i.e. :

where D is the distribution on the examples x and the corresponding labels y , f ( · ; θ ) is a model with learnable parameters θ , L is a suitable loss function, and S is the set of allowed constrained perturbations. In the case of ℓ p norm-bound perturbations of maximum magnitude ϵ , we can further specify S := { δ | ∥ δ ∥ p ≤ ϵ } .

The inner optimisation problem is solved, in [42], by Projected Gradient Descent (PGD), an iterative algorithm whose goal is the synthesis of an adversarial perturbation ˆ δ = δ ( K ) after K gradient ascent and projection steps defined as:

where δ (0) is randomly sampled within S , α is a hyperparameter ( step size ), L ce is the cross-entropy function, and P A is the Euclidean projection operator onto set A , i.e. :

The outer optimisation is carried out by simply training f ( · ; θ ) on the examples found by PGD against the current model parameters - and their original pre-perturbation labels. The overall procedure just described constitutes PGD adversarial training .

B On the functioning of (conditional) Variational Autoencoders

Variational autoencoders ( VAE s) [32, 50] learn from data a generative distribution of the form p ( x , z ) = p ( x | z ) p ( z ) , where the probability density p ( z ) represents a prior over latent variable z , and p ( x | z ) is the likelihood function, which can be used to sample data of interest x , given z .

Training is carried out by maximising a variational lower bound, -L VAE ( x ) , on the log-likelihood log p ( x ) - which is a proxy for the Evidence Lower Bound ( ELBO ) -i.e. :

where q ( z | x ) ≈ p ( z | x ) is an approximate posterior and KL( ·∥· ) is the Kullback-Leibler divergence.

By parameterising the likelihood with a decoder ANN p θ D ( x | z ; θ D ) ≈ p ( x | z ) , and a possible variational posterior with an encoder ANN q θ E ( z | x ; θ E ) ≈ q ( z | x ) , the parameters θ ⋆ D of the generative model that best reproduces the data can be learnt - jointly with θ ⋆ E - as:

where D is the distribution over the (training) examples x .

From a practical point of view, optimisation is based on the empirical evaluation of L VAE ( x ; θ ) on mini-batches of data, with the term -E z ∼ q θ E ( z | x ; θ E ) [log p θ D ( x | z ; θ D )] replaced by a reconstruction cost

530

i.e. decoding samples from p ( z ) .

The setting is analogous in the case of conditional Variational Autoencoders [56, 64] (see section 3), 531 where conditional sampling is achieved by 532

C Justification of Adversarially-balanced batches

During the incipient phases of experimentation, preliminary tests were performed with the MNIST [36] and Fashion-MNIST [63] datasets - using a conditional VAE as the purifier , and small FCNs or convolutional ANN s as the classifiers . Adversarial examples were generated against the adversarially pre-trained classifier , and tentatively denoised by the purifier with one sample only. The resulting recovered inputs were classified by the classifier and the overall accuracy was recorded.

Importantly, such tests were not meant to assess the end-to-end adversarial robustness of the whole architecture, but only to tune the training protocol of the purifier .

Generating adversarial training examples by means of PGD is considered the gold standard [24] and was first attempted as a natural choice to train the purifier. However, in this case, the following phenomena were observed:

Unsatisfactory clean accuracy was reached upon convergence, speculatively a consequence of the VAE having never been trained on clean -toclean example reconstruction;

Persistent vulnerability to same norm-bound FGSM perturbations was noticed;

Persistent vulnerability to smaller norm-bound FGSM and PGD perturbations was noticed.

In an attempt to mitigate such issues, the composition of adversarial examples was adjusted to specifically counteract each of the issues uncovered. The adoption of any smaller subset of attack types or strength, compared to that described in subsection 4.4, resulted in unsatisfactory mitigation.

At that point, another problem emerged: if such an adversarial training protocol was carried out in homogeneous batches, each containing the same type and strength of attack (or none at all), the resulting robust accuracy was still partially compromised due to the homogeneous ordering of attack types and strengths across batches.

Such observations lead to the final formulation of the training protocol, detailed in subsection 4.4, which mitigates to the best the issues described so far.

D Heuristic justification of the robust aggregation strategy

The rationale leading to the choice of the specific robust aggregation strategy described in subsection 4.5 was an attempt to answer the following question: 'How is it possible to aggregate the results of an ensemble of classifiers in a way such that it is hard to tilt the balance of the ensemble by attacking only a few of its members?'. The same reasoning can be extended to the reciprocal problem we are trying to solve here, where different input reconstructions obtained from the same potentially perturbed input are classified by the same model (the classifier ).

Far from providing a satisfactory answer, we can analyse the behaviour of our aggregation strategy as the logit associated with a given model and class varies across its domain, under the effect of

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

The generation of new data according to the fitted model is achieved by sampling from 529

adversarial intervention. Comparison with existing (and more popular) probability averaging and 566 logit averaging aggregation strategies should provide a heuristic justification of our choice. 567

We recall our aggregation strategy: 568

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

where Q α = ∑ C j =1 e l α j .

Finally, since l α i ∈ R , ∀ l α i , lim x →-∞ e x = 0 and e 0 = 1 , we can observe that e l α i > 0 and e e l α i > 1 , ∀ l α i .

Now, we consider a given class i ⋆ and the classifier prediction on a given input reconstruction α ⋆ , and study the potential effect of an adversary acting on l α ⋆ i ⋆ . This adversarial intervention can be framed in two complementary scenarios: either the class i ⋆ is correct and the adversary aims to decrease its membership probability, or the class i ⋆ is incorrect and the adversary aims to increase its membership probability. In any case, the adversary should comply with the ϵ ∞ -boundedness of its perturbation on the input.

Logit averaging In the former scenario, the product of e l α i terms can be arbitrarily deflated (up to zero) by lowering the l α ⋆ i ⋆ logit only. In the latter scenario, the logit can be arbitrarily inflated, and such effect is only partially suppressed by normalisation by Z (a sum of 1 / N -exponentiated terms).

Probability averaging In the former scenario, although the effect of the deflation of a single logit is bounded by e l α ⋆ i ⋆ > 0 , two attack strategies are possible: either decreasing the value of l α ⋆ i ⋆ or increasing the value of Q α ⋆ , giving rise to complex combined effects. In the latter scenario, the reciprocal is possible, i.e. either inflating l α ⋆ i ⋆ or deflating Q α ⋆ . Normalisation has no effect in both cases.

Ours In the former scenario, the effect of logit deflation on a single product term is bounded by e e l α ⋆ i ⋆ > 1 , thus exerting only a minimal collateral effect on the product, through a decrease of Z . This effectively prevents aggregation takeover by logit deflation. Similarly to logit averaging , in the latter scenario, the logit can be arbitrarily inflated. However, in this case, the effect of normalisation by Z is much stronger, given its increased magnitude.

From such a comparison, our aggregation strategy is the only one that strongly prevents adversarial takeover by logit deflation , while still defending well against perturbations targeting logit inflation .

E Architectural details and hyperparameters

In the following section, we provide more precise details about the architectures (subsection E.1) and hyperparameters (subsection E.2) used in the experimental phase of our work.

Additionally, we recall logit averaging aggregation 569

and probability averaging aggregation

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

E.1 Architectures

In the following subsection, we describe the specific structure of the individual parts composing the purifier - in the three scenarios considered. As far as the classifier architectures are concerned, we redirect the reader to the original articles introducing those models ( i.e. : [18] for scenarios (a) and (b) , [62] for scenario (c) ).

During training, before being processed by the purifier encoder, input examples are standardised according to the statistics of the respective training dataset.

Afterwards, they are fed to the disjoint input encoder (see subsection 4.3), whose architecture is shown in Table 4. The same architecture is used in all scenarios considered.

The original input is also fed to the classifier . The corresponding internal representation is extracted, preserving its layered structure. In order to improve the scalability of the method, only a subset of classifier layers is used instead of the whole internal representation. Specifically, for each block of the WIDERESNET architecture, only the first layers have been considered; two shortcut layers have also been added for good measure. The exact list of those layers is reported in Table 5.

Each extracted layerwise (pre)activation tensor has the shape of a multi-channel image, which is processed - independently for each layer - by a different CNN whose individual architecture is shown in Table 6 ( scenarios (a) and (b) ) and Table 7 ( scenario (c) ).

The resulting tensors (still having the shape of multi-channel images) are then jointly processed by a fully-connected subnetwork whose architecture is shown in Table 8. The value of fcrepr for the different scenarios considered is shown in Table 13.

The compressed input and compressed internal representation so obtained are finally jointly encoded by an additional fully-connected subnetwork whose architecture is shown in Table 9. The output is a tuple of means and standard deviations to be used to sample the stochastic latent code z .

The sampler used for the generation of such latent variables z , during the training of the purifier , is a reparameterised [32] Normal sampler z ∼ N ( µ, σ ) . During inference, z is sampled by reparameterisation from the i.i.d Standard Normal distribution z ∼ N (0 , 1) ( i.e. from its original prior).

The architectures for the decoder of the purifier are shown in Table 10 ( scenarios (a) and (b) ) and Table 11 ( scenario (c) ).

E.2 Hyperparameters

In the following section, we provide the hyperparameters used for adversarial example generation and optimisation during the training of the purifier , and those related to the purifier model architectures. We also provide the hyperparameters for the PGD+EOT attack, which is used as a complementary tool for the evaluation of adversarial robustness.

Table 7: Architecture for the layerwise internal representation encoder of the purifier . The architecture shown in this table is used in scenario (c) . The architecture is represented layer by layer, from input to output, in a PyTorch-like syntax. The following abbreviations are used: Conv2D : 2-dimensional convolutional layer; ch_in : number of input channels; ch_out : number of output channels; ks : kernel size; s : stride; p : padding; b : presence of a learnable bias term; BatchNorm2D : 2-dimensional batch normalisation layer; affine : presence of learnable affine transform coefficients; slope : slope for the activation function in the negative semi-domain. The abbreviation [ci] indicates the number of input channels for the (pre)activation tensor of each extracted layer. The abbreviation ceil indicates the ceiling integer rounding function.

Table 8: Architecture for the fully-connected representation encoder of the purifier . The architecture shown in this table is used in all scenarios considered. The architecture is represented layer by layer, from input to output, in a PyTorch-like syntax. The following abbreviations are used: Concatenate : layer concatenating its input features; flatten_features : whether the input features are to be flattened before concatenation; feats_in , feats_out : number of input and output features of a linear layer; b : presence of a learnable bias term; BatchNorm1D : 1-dimensional batch normalisation layer; affine : presence of learnable affine transform coefficients; slope : slope for the activation function in the negative semi-domain. The abbreviation [computed] indicates that the number of features is computed according to the shape of the concatenated input tensors. The value of fcrepr for the different scenarios considered is shown in Table 13.

Table 9: Architecture for the fully-connected joint encoder of the purifier . The architecture shown in this table is used in all scenarios considered. The architecture is represented layer by layer, from input to output, in a PyTorch-like syntax. The following abbreviations are used: Concatenate : layer concatenating its input features; flatten_features : whether the input features are to be flattened before concatenation; feats_in , feats_out : number of input and output features of a linear layer; b : presence of a learnable bias term; BatchNorm1D : 1-dimensional batch normalisation layer; affine : presence of learnable affine transform coefficients; slope : slope for the activation function in the negative semi-domain. The abbreviation [computed] indicates that the number of features is computed according to the shape of the concatenated input tensors. The value of fjoint for the different scenarios considered is shown in Table 13. The last layer of the network returns a tuple of 2 tensors, each independently processed - from the output of the previous layer - by the two comma-separated sub-layers .

Table 10: Architecture for the decoder of the purifier . The architecture shown in this table is used in scenarios (a) and (b) . The architecture is represented layer by layer, from input to output, in a PyTorch-like syntax. The following abbreviations are used: Concatenate : layer concatenating its input features; flatten_features : whether the input features are to be flattened before concatenation; feats_in , feats_out : number of input and output features of a linear layer; b : presence of a learnable bias term; ConvTranspose2D : 2-dimensional transposed convolutional layer; ch_in : number of input channels; ch_out : number of output channels; ks : kernel size; s : stride; p : padding; op : PyTorch parameter ' output padding ', used to disambiguate the number of spatial dimensions of the resulting output; b : presence of a learnable bias term; BatchNorm2D : 2-dimensional batch normalisation layer; affine : presence of learnable affine transform coefficients; slope : slope for the activation function in the negative semi-domain. The values of fjoint and fcrepr for the different scenarios considered are shown in Table 13.

Attacks The hyperparameters used for the adversarial attacks described in subsection 4.4 are shown 632 in Table 12. The value of ϵ ∞ is fixed to ϵ ∞ = 8 / 255 . With the only exception of ϵ ∞ , AUTOATTACK 633 is to be considered a hyperparameter-free adversarial example generator. 634

Architectures Table 13 contains the hyperparameters that define the model architectures used as 635 part of the purifier , in the different scenarios considered. 636

Training Table 14 collects the hyperparameters governing the training of the purifier in the different 637 scenarios considered. 638

F Additional tables 639

The following section contains additional tabular data that may be of interest to the reader. 640

Table 15 reports the respective clean accuracies for the best models available in terms of AUTOAT641 TACK robust accuracy, in scenarios (a) and (b) . Models are further divided in AT-based and 642 purification-based, so as to match the corresponding columns for robust accuracy shown in Table 2. 643

The best AT-based model for CIFAR-10 is taken from [18], whereas that for CIFAR-100 from [62]. 644 Both best purification-based models are taken from [40]. 645

646

647

648

The clean and robust accuracies for the best AT-based model on TINYIMAGENET-200 ( scenario (c) ) are already part of Table 2 and we redirect the reader there for such information. We are not aware of any published state-of-the-art adversarial purification-based model for TINYIMAGENET-200.

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Claims made in the abstract and introduction of the paper accurately reflect its contributions, and those are directly corroborated by experimental analysis. Results and their discussion is available in subsection 5.2 and subsection 5.3.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: subsection 5.2 and subsection 5.3 also contain the discussion of potential limitations of the method, and open problems it introduces.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not contribute novel theoretical results. Assumptions anyway related to the contribution are clearly stated throughout the paper.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Training and evaluation details required to reproduce the experimental results of the paper are reported in section 5 and Appendix E. Code and data for the reproduction of all experiments are additionally released as part of Supplementary Material.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code and data for the reproduction of all experiments are released to reviewers as part of Supplementary Material. The public version of the paper will include instructions to obtain the same material from a dedicated publicly accessible source.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimiser, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Training and evaluation details, including hyperparameters, required to reproduce the experimental results of the paper are reported in section 5 and Appendix E. Code and data for the reproduction of all experiments are additionally released as part of Supplementary Material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

696

Justification: As doing adversarial training with 40-steps PGD is roughly 40 times more computationally demanding than nominal training, unfortunately, we are unable to show error bars or otherwise quantify statistical errors. In any case, the improvement induced by our method w.r.t. their state-of-the-art counterparts is well clear of the threshold for statistical significance.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Information about computational resources and required time is contained in subsection 5.1 as well as in the supplementary materials.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper does conform, in every respect, with the NeurIPS Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper proposes a novel technique to mitigate the problem of adversarial vulnerability of high-dimensional classifiers. Such vulnerability may pose potential societal impacts, as discussed in section 1.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not plan to release models or data with a high risk for misuse. Models to be released do not reasonably carry risk for misuse.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators and/or owners of assets used in the paper are either credited by reference to their original research work, or directly with a link to the preferred landing page for such assets. The use of licensed material is compliant with the respective licenses.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Trained model weights are provided alongside the paper, and their details are provided as part of supplementary materials. In case of release to the public, such details will be provided contextually to the models.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Peer reviews:
comments: SUMMARY:
To better defend against adversarial attacks, the paper proposes a novel adversarial defense mechanism for image classification – CARSO – blending the paradigms of adversarial training and adversarial purification in a synergistic robustness-enhancing way.

STRENGTHS:
The paper proposes a novel defense mechanism.

The proposed method is validated on multiple datasets.

WEAKNESSES:
1. The presentation of the paper is poor.

    a) In the first half of the paper, the author merely describes some background. There is a lack of analysis of existing methods, such as the shortcomings of the current methods, what problems the proposed method can solve, and why it can solve these problems.

    b) Some descriptions are unclear, such as 'Upon completion of the training process, the encoder network may be discarded as it will not be used for inference.' I think 'may' should be removed here.

2. The current experiments are insufficient to prove the effectiveness of the proposed method.

    a) Table 2 simplifies a lot of information, which reduces clarity; for example, it only records the mean or best results of multiple methods and lacks the clean accuracy of the purification method. I suggest listing all methods according to both clean accuracy and adversarial accuracy. The existing content in Table 2 can be added as additional row information.

    b) Since the paper does not give specific problems, only a general goal, which is to better defend against adversarial attacks, the experiments become relatively limited. I believe the author should re-summarize the shortcomings of existing methods and the advantages of the proposed method and conduct more experimental comparisons.

QUESTIONS:
See Weaknesses.

LIMITATIONS:
The authors have discussed limitations of the work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-06-20

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study proposes a novel adversarial defense method called CARSO. CARSO consists of two models: a classifier and a purifier. The classifier is (pre)trained to correctly classify possibly perturbed data. The encoder of the purifier is trained to generate a latent space from the internal representation of the classifier and the original (possibly perturbed) input. The decoder of the purifier is trained to reconstruct a sample from the latent representation and the internal representation of the classifier. The final prediction is determined by aggregating the outputs of the classifier for reconstructed data.\n\nDetailed procedures are summarized as follows:\n\n- The classifier is always kept frozen. Other parts, including the VAE and small CNNs for compression, are trained on a VAE loss consisting of a reconstruction loss based on a pixel-wise channel-wise binary cross-entropy loss and KL-div.\n- The internal representation and input are compressed by small CNNs before being inputted into the encoder of the purifier.\n- The classifier is pretrained according to [18] or [62].\n- When training the purifier, each batch contains both clean and adversarial samples.\n- The aggregation is represented by a double exponential function.\n- Evaluations are conducted under $L_\\infty$ attacks.\n\nSTRENGTHS:\n- The concept of blending adversarial training and purification is novel and interesting. The proposed method, CARSO, achieves robust accuracy that surpasses the SOTA adversarially trained models and purification methods, including diffusion-based models, despite its relatively simple mechanism.\n- The evaluation was carefully conducted. The authors explicitly address common pitfalls in evaluating robustness. For example, they conducted end-to-end validation (full whitebox setting), addressed concerns about gradient obfuscation, and used PGD+EOT to address the stochasticity of CARSO.\n- CARSO can utilize existing pretrained models, which have already achieved high robust accuracy.\n- A wide variety of datasets (CIFAR-10, CIFAR-100, and TinyImageNet-200) were used for evaluation.\n\nWEAKNESSES:\n**1**. In my opinion, the claim that CARSO surpasses the used adversarially trained model seems questionable. If my understanding is correct, during inference, the decoder takes class information only from the internal representation of the classifier. Thus, I believe the decoder can correctly reconstruct the sample only if the classifier, outputting the internal representation, can correctly extract class information from the original perturbed sample. Could the authors clarify this?\n\nNote: Initially, I doubted whether some experimental or evaluation settings were appropriate. However, as far as I can tell, there are no issues. Just in case, I recommend the authors review their source code again.\n\n**2**. CARSO sacrifices clean accuracy more significantly than existing SOTA methods. Additionally, to compare CARSO and the best AT/purification models in terms of clean accuracy, Table 2 should include the clean accuracy of the best AT/purification models (i.e., the contents in Table 15). The scenario or dataset columns in Table 2 might not be necessary.\n\n**3**. Few ablation studies. The authors should include the case of $L_2$ perturbations and use internal representations from different layers. Particularly, the relationship between the layers used for extracting representation and robust accuracy is of interest.\n\nQUESTIONS:\nMinor comments:\n- The authors should standardize the meaning of each symbol. If I understand correctly, $i$ represents the sample index in Figure 1 but represents the class index in Section 4.5, leading to low readability.\n- In Figure 1, $\\mathcal{L}_{VAE}$ is not defined. I believe it is first explained in Line 270.\n\nLIMITATIONS:\nThe authors explicitly addressed the limitations in Section 5.3.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-20"}

comments: SUMMARY:
This paper integrates adversarial training and adversarial purification to enhance robustness. It specifically maps the internal representation of potentially perturbed inputs onto a distribution of tentative reconstructions. These reconstructions are then aggregated by the adversarially-trained classifier to improve overall performance.

STRENGTHS:
The idea of combining adversarial training and adversarial purification is interesting.

WEAKNESSES:
1, The experiments are too weak. I hope the authors can refer to at least [1][2][3], which are relevant to adversarial purification, to conduct experiments from more dimensions and consider more baselines and fundamental experiments.

2, Could we just combine [1] with an adversarially-trained model to achieve similar performance?

3, Why should the classifier be adversarially trained for better accuracy?

4, Why can't we directly purify the image? Could we use an image-to-image method to purify the input image?





[1] DISCO: Adversarial Defense with Local Implicit Functions.
[2] Diffusion Models for Adversarial Purification
[3] IRAD: Implicit Representation-driven Image Resampling against Adversarial Attacks

QUESTIONS:
The first two points mentioned above are my key concerns.

LIMITATIONS:
The method heavily relies on training a VAE as the generative purification model.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-09