Boosting Adversarial Transferability with Spatial Adversarial Alignment
Authors: Zhaoyu Chen
HaiJing Guo
Kaixun Jiang
Jiyuan Fu
Xinyu Zhou
Dingkang Yang
Hao Tang
Bo Li
Wenqiang Zhang
Venue: NeurIPS 2025 poster ()
Keywords: adversarial transferability
model alignment
Primary area: deep_learning
TL;DR: We propose Spatial Adversarial Alignment (SAA), which leverages a witness model to fine-tune the surrogate model via spatial-aware and adversarial-aware alignment to generate highly transferable adversarial examples.

Abstract: Deep neural networks are vulnerable to adversarial examples that exhibit transferability across various models. Numerous approaches are proposed to enhance the transferability of adversarial examples, including advanced optimization, data augmentation, and model modifications. However, these methods still show limited transferability, partiovovocularly in cross-architecture scenarios, such as from CNN to ViT. To achieve high transferability, we propose a technique termed Spatial Adversarial Alignment (SAA), which employs an alignment loss and leverages a witness model to fine-tune the surrogate model. Specifically, SAA consists of two key parts: spatial-aware alignment and adversarial-aware alignment. First, we minimize the divergences of features between the two models in both global and local regions, facilitating spatial alignment. Second, we introduce a self-adversarial strategy that leverages adversarial examples to impose further constraints, aligning features from an adversarial perspective. Through this alignment, the surrogate model is trained to concentrate on the common features extracted by the witness model. This facilitates adversarial attacks on these shared features, thereby yielding perturbations that exhibit enhanced transferability. Extensive experiments on various architectures on ImageNet show that aligned surrogate models based on SAA can provide higher transferable adversarial examples, especially in cross-architecture attacks.

Boosting Adversarial Transferability with Spatial Adversarial Alignment
Zhaoyu Chen Haijing 2.+ Kaixun Jiang'
Jiyuan Fu² Xinyu Zhou² Dingkang Yang' Hao Tang' Bo Li Wengiang Zhang! 121
College of Intelligent Robotics and Advanced Manufacturing, Fudan University Cooleeee Computer Science and Artificial Intelligence, Fudan University China Southern Power Grid Artificial Intelligence Technology Co., Ltd. School of Computer Science, Peking University 'vivo Mobile Communication Co., Ltd. {zhaoyuchen20, wqzhang) @fudan. edu. en, hjguo220m. fudan, edu. cn
Abstract
Deep neural networks are vulnerable adversarial examples that exhibit trans- ferability across various models. Numerous approaches are proposed to enhance the transferability of adversarial examples, including advanced optimization, data augmentation, and model modifications. However, these methods still show limited transferability, particularly in arosttccchteeetuee scenarios, such as from CNN VÝT. To achieve high transferability, we propose technique termed Spatial Adver- sarial Aligmment (SAA). which employs an alignment loss and leverages a witness model to fine-tur tine-tuneee the surrogate model. Specifically, SAA consists of two key parts: spatial-aware alignment and adversarial-awan alignment. First, we minimize the divergences of features between the two models in both global and local regions, facilitating spatial alignment. Second, we introduce self-adversarial strategy that leverages adversarial examples impose further constraints, aligning features from an adversarial perspective. Through this alignment, the surrogate model is trained to concentrate the common features extracted by the witness model. This facilitates adversarial attacks on these shared features, thereby yielding per- turbations that exhibit enhanced transferability. Extensive experiments various architectures on ImageNet show that aligned surrogate models based on SAA can provide higher transferable adversarial examples, especially in roos-architecture attacks.
1 Introduction
Deep neural networks (DNNs) have been successfully and extensively deployed across security- sensitive applications 332 including autonomous driving 663
68 67], facial verification [70] 69
91 and video surveillance 23
72
24
20 However, DNNs exhibit considerable vulnerability 7
5 30 B3,, where imperceptible perturbations are introduced into natural images, leading models to produce incorrect predictions. In real-world applications, DNNs are typically concealed from user access, necessitating adversaries to generate adversarial examples within a black-box setting, where no knowledge of the target model's parameters plays crucial role in black-bon settings it allows adversaries to effectively compromise target models by employing adversarial examples generated on surrogate models. In black-box settings, adversarial transferability plays crucial role, which enables adversaries to leverage adversarial examples crafted on surrogate models
to adversarial examples 18 39
or architecture is available. Adversarial transferability [33 26 51]
indicates equal contributions.
I indicates corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

to effectively attack target models. Thus, generating highly transferable adversarial examples is instrumental in uncovering and understanding the vulnerabilities within DNNs, drawing substantial attention in recent research.
Cross-model transferability has been extensively studied for CNNs
13
61 4
Highly transferable
and data augmenta- alleviate the overfitting of adversarial examples on surrogate models, determining whether the attack can be successfully transferred to the target models. In addition, some model modification methods [59 21, such as amplifying the gradient on skip connec- tions (the structure in ResNet (221), can also improve transferability. However, few works explore adversarial transferability on Vision Transformer (ViT) [15] and the performance of existing work extending CNN to ViT is poor due to significant structural differences. Specifically, ViT flattens the image into a sequence of patch tokens and employs multi-head self-attention to capture global relationships among the patches. In contrast, CNNs typically consist of stacked convolutional layers that learn feature relationships progressively through downsampling. Therefore, [58] first empirically
adversarial examples are usually based on advanced optimization [3
29
tion [6] 14 37). The principle
analyzes the structure of ViT and propose PNA and PatchOut (58) but there is still much room for improvement in rross-architecture transferability.
In this paper, we argue that unique structural features are critical to cross-architecture adversarial transferability. Given a dataset, various models tend to exhibit analogous decision boundaries [B4] arising from their ability to learn similar features. If we can obtain surrogate model whose features are similar to those of models with different architectures, then the resulting adversarial perturbation can be transferable across different models. A recent technique known as Model Alignment (MA) (38 employs an alignment loss to minimize prediction divergences between surrogate models and witness models, thereby indirectly facilitating the extraction of features that are similarly represented by the witness model. However, directly applying MA to black-box attacks may lead to the degradation of cross-architecture transferability. The main reasons are: (i) Features are not aligned in space (58). MA only uses the final prediction of the model, but in fact, the spatial features of ViT and CNN are different (7). It is difficult to directly constrain the similarity of features only by the final logits. (ii) Features are not aligned from the perspective of adversarial features. In addition to the features of clean images, the features of adversarial examples also have similarities across different models and need to be considered.
To overcome these challenges and enhance transferabilityy we propose a technique called Spatial Adversarial Alignment (SAA), which utilizes an alignment loss from the perspective of spatial and adversarial features and incorporates a witness model to refine the surrogate model. SAA consists of two key parts: spatial-aware alignment and adversarial-awan alignment. In the spatial-aware alignment, in addition to aligning on the final global features, we also focus on the features of local regions. We make local features of CNNs by position to align ViTs' embeddings at the same position. In the adversarial-awes -wware alignment, we introduce self-adversarial strategy, which constructs adversarial examples so that the model can learn the differences between different architectures in adversarial features, thereby enabling the model to further capture more common features. Aligned surrogate models by SAA provide promising adversarial transferability and can be seamlessly integrated with existing transfer attacks. In addition, further summarize the empirical guidance for the selection of surrogate and witness models in SAA. Our contributions can be summarized as follows:
We reveal for the first time the importance of spatial and adversarial features for cross-architecture transferability, which supports alignment with different models.
We propose Spatial Adversarial Alignment (SAA), which leverages witness model to fine-tune the surrogate model via spatial-awar and adversarial awwre alignment to generate highly transferable adversarial examples. addition, we further summarize the empirical guidance for the model selection in SAA.
Experiments 6 CNNs and 4 ViTs show that SAA has state-of-the-ar adversarial transferability, especially in cross-model transferability. Compared with MA, on ResNet50, the transferability from CNN to ViT is improved by 25.5-39.1%
2

---

2 Methodology
2.1 Preliminaries
In this paper, we focus on the image classification task on DNNs. Let fo(-) represent DNN-based classifier with different network parameters We denote the clean image as z and its corresponding ground-truth label as y. Following [3 [1 we evaluate the adversarial transferability under untargeted adversarial attacks with (o norm. Therefore, the goal of transfer attacks is to add an adversarial perturbation to the clean image x based on the information of the surrogate model fe, (-) to obtain the adversarial example Iado [18]. so that fe. (Iade) * y subject to the constraint that Iaade =1/00= I In the black-box setting, information about the target model such as its architecture, weights, or gradients accessible. Therefore, adversarial examples are generated solely by utilizing surrogate model Jo, (+), leveraging their transferability to deceive the target model (.).
2.2 Spatial Adversarial Alignment (SAA)
Spatial Adversarial Alignment (SAA) employs alignment loss tailored both spatial and adver- sarial feature perspectives, incorporating a witness model to fine-tune the surrogate model. SAA aims adjust the surrogate model to extract features closely aligned with those of the witness model, capturing both spatial and adversarial features shared across models. As shown in Figure SAA consists of parts, namely spatial-aware alignment and adversarial-awar alignment.
Spatial-aware Aligmment. The purpose of spatial-aware alignment is make the sur- rogate and witness models more consistent in the feature space. Naturally, the most intuitive approach to aligning the feature distributions of two models is minimize the distance between their final outputs (38). However, when the models exhibit signifi- cant architectural differences, ensuring out- put similarity alone is insufficient achieve alignment in intermediate features. In black- box attacks, where the details of the target model's architecture are unknown, this is- sue becomes more pronounced. Taking the challenging scenario of CNN to ViT as an example, their intermediate layer features differ substantially in semantic levels [71]. This discrepancy arises primarily from differ- ences in receptive fields, stacking methodolo- gies, and normalization techniques between CNNs and ViTs. Therefore, relying solely on output alignment for model fine-tuning indirectly captures
Clean Image
LoCal Back
Spotia--we award
Clean Imsgge
Share
Wines
Advertaiallawor Alignment
Eourple e
Okiee Fackum
Pool
Felaye
Locil Back
Surrogate Model
Spatial Adversarial Alignment (SAA) consists of two parts: spatial-aware alignment and adversarial-awan alignment. Initially, we aim to min- imize the feature divergences between the two models across both global and local regions, thereby promot- ing spatial alignment. Subsequently, we introduce self-adverss veersarial strategy that utilizes adversarial ex- amples to impose additional constraints, aligning the adversarial features.
Figure 1:
common features, but this approach can, in certain cases, re- sult in degraded transferability, observed in methods like Model Alignment (MA) (38).
Therefore, in addition to aligning the final global features, we also need to focus on the features of local regions. For ease of understanding, of the model corresponding to the input æ. For CNNs, is the output of features by the last layer. For ViTs, it refers to the final embedding of the [CLS] token after the MLP block. First, perform alignment at the global feature level by defining an alignment loss between the surrogate model and witness model at the output layer: Cytobal (z: 0.) DKL (fo,fo, fe..
define the global features fo(x)) the logits
(1)
where DKL measures the feature divergence with Ballback-Leible (KL) divergence.
3

---

Next, we align the models at the local feature level. Here, we define (0(x;) as local features. Each spatial position (h, w) within this feature map is treated as a distinct local region and the feature for each local region is 2 (x), where q {1, ..., H x W.. Since every local feature is embedded with the corresponding sub-image and position information, regarding them as spatial dense predictions is [0) reasonable. Let =0, (x)) and _ = (x) denote the local features associated with each local region for the
surrogate model and witness model, respectively. For CNNs, 29 2x(x) IXX represents the logits generated by the final convolutional layer and then pass the final MLP. For ViTs, zg
x = is the embeddings of patch tokens after passing through the last MLP except for the [CLS] token, where each patch token corresponds to a specific spatial region in the input. Next, we argue pseudo- labels better aggregate local information, so we compute pseudo-labela labels of the local region 9 and [ol denote this pseudo-label as Va.. which is obtained by taking the arg max over the logits after the last
BxC"
W
MLP of the witness model: olu! arg max (x)). Then, the learning of the local feature of the surrogate model. To achieve local alignment, we minimize the divergence of corresponding local regions' logits and the local alignment loss is expressed as:
this pseudo-label to supervise
HW
Lroeat (r; 0,) = HW
Σ
(0 (x), (z), Io.)).
(2)
g=1
where DoE is the cross-entropy loss. Therefore, the spatial-aware alignment loss is expressed as: (x 02) Lgiobal(x; (2;0,) + 7
(x;
where y is the spatial factor. By minimizing this spatial- aware aware alignment loss, we encourage the surrogate model to produce features in both global and local regions that are consistent with those of the witness model, even across different architectures.
Avvers Alignment. The relationship between features and adversarial vulnerability is highly significant. Some hypotheses [64 4
propose that adversarial examples possess distinet feature distributions compared to normal examples, which may inherently predispose models to adversarial vulnerability-4 notion supported by several studies (7]. Beyond normal examples, learning adversarial features may offer way to capture shared features between surrogate models and witness models. Furthermore, [4] suggests that models trained with adversarial examples focus on more discriminative regions within images, displaying feature recognition patterns distinct from those of normally trained models. Thus, adversarial examples play a crucial role in achieving model alignment.
In our adversarial alignment, introduce self-adversarial strategy that constructs adversarial examples of the surrogate model to enable the model to discern architectural differences in adversarial features effectively. Specifically, we leverage the gradients iteratively generate adversarial examples
(0) under the supervision of the global features of the witness model. Assuming x adde = x, we define the
adversarial example Fade of the surrogate model as:
(t+1)
T (L+1) IIr, (2)(1) + a sign DvD (fo, for (z))))
(4)
where DKI. denotes the KL divergence, x (() denotes the adversarial example at iteration 6, a is the step size, and II, projects the adversarial example onto an e-bounded neighborhood around the original input x.
Once the adversarial example Tade is generated, we also perform adversarial aawre adversarial examples from global and local features further align the surrogate and witness models. The loss of the adversarial-aware alignment is expressed as:
aware alignment on the
(A 0.) = Lytobal (Fadv; 0z) w Llocat(Fadts Ox),
(5)
where w is the adversarial factor.
Optimization. Combining spatial-aware and adversarial-aware alignment, the final optimization goal of spatial-adversane alignment is:
CSAA(E; (2; 0.) = LS 0,) + K CAs 0,):
(6)
Generallyy VIT's patch embeddings =(x) is (B,N, C²) by default. We first transform it to (B, C² H W"). where N H' x W Then, we perform adaptive pooling operation transform it to (B, C¹ H, W).

---

Inpet
Unaligned
Witness
Aligaed
Unaligned
Witness
Aligued
Witness
Adversatia Example
Clean
Grad-CAM visualizations comparing the feature distribution of unaligned and aligned surrogate models (Res50) on clean inputs and adversarial examples (generated by SSA-DI-TI-MI).
Figure 2:
where k. is the alignment factor to balance the two alignments. If not otherwise stated, we define 7 = 0.2, w 0.02, and x 0.02 in this paper.
Opatial-adversarte alignment facilitates the alignment of the surrogate model with the witness model further improve the adversarial transferability. The parameter update rule for the surrogate model, based on stochastic gradient descent (SGD), can be expressed as follows:
0 ((0) 7
Vogn pin Lsa 0.);
ZEB
where f is the epoch, 17 is the learning rate and B means the mini-batch samples. Please refer to Algorithm| for the detailed loss calculation process.
2.3 A Close Look at SAA
To verify whether SAA significantly improves the spatial and adversarial features after model aligmment, we conduct quantitative and qualitative analyses based on the models before and after aligmment. We randomly sample 100 images from ImageNet val and then compute the cosine similarity between the global features of the surrogate models before and after applying SAA with different witness models. Table [ shows that, whether for clean images adversarial examples (generated by SSA-DI-TI-MII the feature similarity improves after alignment. Notably, when the surrogate model is VIT-B, the improvement in similarity is even more pronounced. In addition, the feature gap gap between Res50 and Swin-B is indeed significant, but even so, SAA achieves a 0.0309
0.0551 -0.0369 49.3% improvement in similarity. This result suggests that, after applying SAA, the aligned surrogate models effectively capture features shared with the witness model, providing strong evidence of the alignment's 's success.
Then, we use Grad-CAM [45)'s heatmaps to simu- late the feature distribution of the model, as shown
Table 1: Cosine similarity of global features of surrogate models.
Figure 2 For the clean inputs (first four cols), the heatmaps generated by the unaligned surrogate model (2-nd col) primarily focus on local regions of the ob- ject. In contrast, the aligned surrogate model (4-th col) heatmaps demonstrate more diffuse attention spread across the entire object, similar to that of the witness model (ViT-B, 3-rd col), which shows aligned surro- gate models learn the common spatial features. For adversarial examples (last four cols), the 5-th and 7-th cols display the heatmaps of adversarial examples generated by the unaligned and aligned surrogate models, respectively. The 6-th and 8-th show the witness model's 's responses these adversarial examples. Notably, the adversarial examples generated by the unaligned surrogate model fail to effectively transfer to the witness model (6-th col) due to still focusing on the target subject, indicating limited cross-model transferability. examples generated by the aligned surrogate model (7-th col) successfully transferred to the witness model (8-th col) as the features are spread out, demonstrating enhanced cross- moosl transferability achieved through SAA.
Clean
Adv
Surrogate Wimness
Unaligned Aligned Unaligned Aligned
Re<50
1,0000
0.9949
1.0000
0.9922
DN121
0.0573
0.1153
0.0700
0.1328
Res50
VIT-B
0.0533
0.1408
0.0452
0.1191
Swin-B
0,0352
0.0448
0.0369
0.0551
Res50
0.0566
0.1323
0.0672
0.1544
DN121
0.4016
0.6278
0.4121
0.6551
VIT-B
VIT-B
1,0000
0.9706
1.0000
0.9728
Swin-B
0.3058
0.5115
0.3169
0.4257
contrast, adversarial
5

---

3 Experiments
3.1 Experimental Setup
Datasets. Our experiments utilize the ImageNet-comppaibble dataset [27], a widely adopted subset containing 1,000 images from the ImageNet validation set [1. This dataset is commonly used in adversarial robustness studies, such as those in [6]1))
Models. To assess the adversarial transferabiliity of different network architectures, we focus on convolutional neural networks (CNNs) and vision transformers (ViTs) as the target models. For CNNs. we select the typically trained ResNet-1e 18 (Res18), ResNet-50 (Res50) and ResNet- 101 (Res101) [22].
VGG-19 [6).. DenseNet- 121 (DN121) [25), and Inception-v3 (Inc-v3) [48] For ViTs, we evaluate
the Vision Transformer (ViT-B) 15) Swin Transformer (Swin-B) 35 Pyramid Vision Transformer
(PVT-v2) [52] and Mobile (MobViT) 44).
Metric. Adversarial transferability is quantified by calculating the average attack success rate (Avg. ASR, %) across target models (excluding the surrogate model), with a higher success rate signifying enhanced transferability. In the paper, 'n/a" is the baseline and defined as the average attack success rate obtained by generating adversarial examples using the surrogate model without any alignment.
Implementation Details. In our experiments, we select the MI [33 attack as the baseline for generating adversarial examples with high transferability, it is widely recognized within the field [29 [56] 59 58). For MI, we set the perturbation magnitude 16 [13 [8] perform 10 iterations, with step size of 16 1.6, and use momentum = 1. During the Spatial Adversarial Alignment, all surrogate models are fine-tuned for 1 epoch using stochastic gradient descent (SGD) with momentum of 0.9, and learning rate adjustments are applied. It is important to note that no additional data is used for fine-tuning, as it relies solely on the same training data used for both the surrogate and witness models. The number of adversarial examples generated by SAA is in a : ratio with the training dataset. The settings for Model Alignment (MA) 38 are consistent with the parameters specified in the original paper.
of adversarial transferability [14 61 [3337] 37 54 51
3.2 Performance Comparison
Performance comparison with alignment methods. We first compare with existing alignment methods (38], where adversarial examples are generated based MI (13). Table illustrates the performance difference between MA and SAA in terms of adversarial transferability, with SAA demonstrating significant advantage over MA. For instance, when the surrogate model is Res50, and the witness model is also Res50, SAA achieves a 16.6% improvement in average ASR over the original surrogate model, compared to a modest 3.6% improvement with MA. This highlights that SAA, even without introducing additional information, enhances adversarial transferability through the alignment of adversarial features. Furthermore, when the witness models are DN121, ViT-B, and Swin-B, SAA outperforms MA by 11.9%. 10.4%. and 13.1%, respectively. In addition to the remarkable adversarial transferability that SAA provides, we make two other key observations: (i) MA only considers global features, which makes it difficult to align features between models with large differences, which may lead to decrease in transferability. When the surrogate model DN121 and the witness model is Swin-B, the ASR of Vit-B, Swin-B, -B, PVT-v2, and Mob VIT is not as good as the origin DN121, which shows that relying solely on global features for alignment not enough, and can only achieve poor transferability. (ii) SAA provides strong cross-architecture transferability. When the surrogate model is Res50, and the witness models are Res50, DN121, ViT-B, and Swin-B, the transferability of SAA on ViT's is improved by 39.06%, 31. 31.29% 25.51 %, and 37.74% respectively compared with MA itself, and it also has high transfcrability between CNNs.
Performance comparison with transfer attacks. Aligned surrogate models by SAA have great potential for adversarial transferability, so existing transfer attacks such as advanced optimization and data augmentation can further improve transferability. Here, we choose Res50 as the surrogate model
and ViT-B as the witness model, and superimpose them with MI [3.. NI 29 GI [5I.. DI [6]], TI [4] and SSA 37 to evaluate the transferability, as shown in Table 3 Taking GI and SSA as examples, the transferability of the model after SAA is improved by 21. 7% and 8.4% respectively, compared with the origin surrogate model, which is very significant improvement. When multiple attacks are integrated, such as SSA-DI-TI-MI, SAA further enhances the transferability by 2.0%. achieving impressive 88.2% ASR, which closes white-box attacks' performance. This indicates

---

Table 2: Comparison of adversarial transfcrability on different alignment methods.
Turget Model
Surrogate
Witness
Attack
CNNs
ViTs
Avg.
(%)
Res18
Res50
Res101
VGG19
DN121
Inc.v3
VIT-B
Swin-B
PVT-22
MosVI
n/a
MI
57.7
99.9
58.1
54.2
55.1
39.0
9.4
33.0
38.0
35.7
42.2
Res50
SAA
MA
77.5
100.0
99.8
71.7
72.0
60.3
77.0
67.3
58.7
19.9
12.1
47.8
51.8
52.8
39.0
58.8
Res50
DN121
SAA
MA
92.4
83.1
96.7
87.1
75.8
90.3
$2.3
94.6
87.1
61.0
30.5
19.8
49.6
54.5
76.5
59.2
75.8
63.9
V
SAA
MA
84.1
74.2
99.2
99.6
63.5
74.7
80.3
69.3
72.8
81.8
51.5
65.7
24.3
18.5
41.5
48.7
42.7
52.9
47.4
62.5
63.9
53.5
MA
64.7
90.4
61.2
61.6
38.8
SAA
79.7
95.7
66.4
74.1
75.9
56.9
19.6
41.9
47.9
55.2
57.5
n/A
MI
84.1
69.6
51.8
76.6
100.0
56.5
16.2
42.1
43.9
52.6
58.6
Res50
SAA
MA
95.7
88.8
65.8
80.3
49.8
70.9
87.7
78.3
100.0
99.6
78.2
31.4
14.9
55.0
56.6
53.0
72.1
71.9
57.6
DN121
DNI21
SAA
MA
90.1
79.3
69.5
54.9
78.8
100.0
55.4
25.3
12.3
39.9
41.7
52.4
70.1
57.2
B
SAA
MA
94.0
89.6
83.6
80.6
75,2
71.1
90.3
83.1
100.0
99.8
81.7
70.0
22.4
27.2
53.9
58.0
59,1
58.4
80.3
68.7
69.1
74,0
Swin-B
SAA
MA
58.3
94.8
63.9
80.0
49.2
69.1
77.8
88.0
100.0
99.7
74.4
26.3
50.2
$1.6
52.2
71.2
69.5
n/a
MI
48.6
41.0
31.9
58.1
50.8
44.4
100.0
58.1
44.7
47.4
53.8
Res50
SAA
MA
65.8
87.3
80.6
51,9
43.0
75.9
86.1
66.9
60.0
87.5
78.0
51.5
99.7
99.9
66.0
91.0
85.4
57.3
57.6
85.2
63.1
86.3
VTT-B
DNI2I
SAA
MA
88.9
73.6
67.6
89.1
87.8
74.9
100.0
99.8
84.9
78.3
82.7
83.8
WT-B
SAA
MA
62.7
54.3
57.2
46.9
29.2
40.9
64.2
56.0
47.99
57.5
42.4
52.1
100.0
100.0
59.5
52.8
42.3
52.6
47.4
58.8
60.9
52.5
Swin-B
SAA
MA
83.0
69.2
38.3
65.6
67.5
81.2
57.6
78.1
74,2
99.7
99.3
$4.3
76.0
58.2
78.3
80.0
n/a
MI
48.2
31.3
20.0
49.3
34.3
29.7
13.9
100.0
45.4
41.4
42.5
Res50
SAA
MA
90.3
54.8
85.3
50.9
77.9
36.5
92.0
61.8
49.3
88.4
76.4
40.1
64.7
32.4
100.0
99.9
92.4
68.5
89.0
55.6
85.7
55.4
Swin-B
DN121
SAA
MA
94.9
77.3
91.7
71.4
60.0
83,9
96.5
94.7
76.4
60.7
74.3
49.0
100.0
85.9
80.9
95.4
91.4
74.9
B
MA
62.6
52.8
40.0
6.2
55.3
45.6
34.0
100.0
70.1
61.8
59.5
SAA
84.7
80.0
74,2
90.5
85.5
75.7
70.3
100.0
91.9
88.6
84.6
Swin-B
SAA
MA
73.5
52.8
58.9
45.7
40.0
79.2
63.4
55.3
50.7
34.0
32.8
100.0
75.8
61.8
71.4
65.8
that SAA substantially narrows the performance gap between black-box and white-box attacks, thereby facilitating a more comprehensive evaluation of the adversarial robustness of existing models.
Performance comparison with attacks on ViTs. To further explore the cross-architecture trans- ferability, we evaluate adversarial attacks on ViTs, including SGM [59], PatchOut (58), PNA (58), and TGR [66). Here, we choose the surrogate model as ViT-B and the witness model as Res50. In PatchOut, SAA first improves the transferability between ViTs, for example, the ASR from ViT-B to Swin-B B improved by 42.6%. Secondly, SAA greatly improves the transferability from ViT to CNNs, for example, the ASR is improved by 45.0% and 42.0% on Res50 and DN121 respectively. On SGM, PNA, and TGR, SAA also achieves stronger cross-architecture transferability without modifying the forward and backward propagation of the model.
Summary. Although we conduct 16 combinations and evaluate on CNNs and ViTs, showing that SAA's improvement is not limited to specific choices of witness models. Based on the above experiments, we further summarize the empirical guidance for the selection of surrogate and witness models: i) Self-alignment (surrogate and witness models are consistent) usually only provides minor improvements (it is difficult learn unique structural features), and alignment between different models improves more significantly. ii) When the surrogate model is a ViT-like model and the witness model a CNN-like model, better cross-architecture transferability can be obtained. This is mainly because VIT has stronger performance, while CNN further provides unique structural features. It should noted that the target models are unknown in the transfer attack, so no information of the target models can be used for alignment.
7

---

Table 3: SAA has stronger adversarial transferability with existing transfer attacks.
Target Model
Attack
CNNs
ViTs
Avg.
ASR (%)
Res18
Res50
Res101
VGG19
DN121
Inc-v3
VIT-B
Swin
PVTv2
MobViT
MI
$7.7
99.9
58.1
54.2
55.1
39.0
9.4
33.0
38.0
35.7
42.2
MI-SAA
84.1
99.6
74.7
80.3
81.8
65.7
24.3
48.7
52,9
62.5
63.9
NI
58.9
100.0
63.2
59.3
61.4
40.0
9.6
37.4
41.8
38.1
45.5
NI-SAA
86.1
99.9
76.3
82.2
83.7
67.6
24.0
50.6
55.7
64.8
65.7
GI
100.0
40.7
12.5
36.8
40.7
GI-SAA
86.5
99.7
78.8
83,9
84.8
70.4
27.6
52.7
55.8
66.3
67.4
DI
44.1
95.8
41.7
56.1
44.2
26.1
5.6
30.9
36.7
35.1
35.6
DI-SAA
94.4
53.1
10.6
44.0
50.8
63.6
56.9
TI
38.5
99.9
37.8
33.9
36.1
24.2
5.4
21.0
29.0
21.1
27.4
TI-SAA
59.7
94.9
35.2
50.6
54.5
40.6
22.8
29.3
33.6
37.3
99.9
57.0
48.5
55.0
59.5
SSA-SAA
91.5
99.5
77.8
85.7
88.4
74.9
23.4
46.7
57.1
66.0
67.9
DI-MI
65.5
97.0
65.0
74.7
65.7
49.1
16.4
49.0
54,9
$7.9
55.4
DI-M-SSAA
78.3
TI-MI
61.4
99,9
60.5
60.9
60.9
44.3
15.2
37.4
42.3
41.8
47.2
TI-MI-SA
84.8
99.3
71.9
79.0
81.8
69.1
27.0
45.0
52.8
62.4
63.8
SSA-MI
89.6
99.9
92.2
91.0
77.6
39.2
74.4
76.4
76.3
78.5
SSA-MI-SAA
96.3
99.8
95.6
96.5
97.2
91.5
46.3
74.1
80.4
88.4
85.1
SI--DLTI-MI
93.5
98.5
92.3
95.0
93.7
85.5
55.9
83.2
87.1
89.8
86.2
SSA-DI-TI-MI-SA
98.8
97.6
96.7
94.3
53.6
81.6
84.4
94.7
88.2
Table 4: SAA further improves the adversarial transferability of adversarial attacks on ViTs.
Target Model
Avg. ASR (%)
CNNs
Res18
Res50
Res101
VGG19
DN121
Inc-v3
VTTB
Swin-B
PVT-22
MobVIT
59.4
75.4
71.3
99.7
83.3
SGM-SAA
91.1
79.8
73.3
87.5
87.3
80.9
99.5
90.5
82.6
86.3
86.6
PatchOut
45.6
27.4
20.3
36.1
33.9
41.0
PatchOurSSAA
76.5
72.4
70.3
79.4
78.1
71.3
94.7
83.6
77.2
76.8
78.7
PNA
61.2
45.0
38.1
5488
49.0
99.6
66.3
55.8
56.8
60.3
PNA-SAA
82.7
78.1
73.4
84.0
75.1
97.4
89.3
82.1
83.3
TGR
74.0
55.6
48.4
73.2
66.6
59.0
99.3
74.5
61.6
69.6
69.6
85.6
79.6
3.3 Ensemble Attacks
The ensemble of multiple surrogate models can further improve adversarial transfcrability [33, so we want explore whether SAA further improve transferability. In the setting of ensemble attacks, we follow the setting of MI [13 and use the logits of multiple surrogate models for integration. Here choose multiple groups of settings, such as Res50 (VTT-B) means that the surrogate model is Res50 and the witness model is ViT-B. Res50+ViT-B means the logits of these two models are integrated, and on. As shown Table 5 SAA further improves the performance of ensemble attacks. First, the alignment-based method is not as good as the direct integration strategy in white-box performance, but it has significant improvement in black-box transferability. For example, on (Res50+ViT-B), MA and SAA are 9.1% and 20.8% higher than the direct ensemble ASR, respectively. Then, MA directly constrains global features and does not take into account the huge differences in features between different structures, which may lead to degradation of transfcrability, such as on Swin-B. Finally, SAA has achieved sfate-of-the-ee transferability and further enhance ensemble attacks because considers spatial and adversarial features from structural perspective.
3.4 Adversarial Defenses
To illustrate the effect of SAA in the face of adversarial defenses, we choose Inc-v3 as the target model and evaluate its performance 12 adversarial defenses, including HGD [28] R&P (60).

---

Table 5: SAA improves the adversarial transfcrability of ensemble attacks.
Target Model
Surrogate
Attack
CNNs
ViIs
Arg. ASR (%)
Res.8
Res50
Res101
VGG19
DN121
Inc-x3
VT-B
Swin-B
PVT-22
MobNTT
Res50
VIT.B
41.0
58.1
w/a
61.3
100.0
63.0
60.8
61.2
45.0
97.6
50.4
51.3
48.1
55.1
Resh0ORRehh0) Res50t VIT-B)
SAA
MA
86.8
79.3
100.0
100.0
85.9
74.3
83.9
72.3
86.7
80.0
70,0
56.3
20.3
30.7
60.4
47.4
64.1
51.7
52.3
64.2
75,9
RestD-DNI2I- -IT-DINNII-ITT--SSSE NiT-R-Swin-f
65.4
100.0
77.4
71.8
RectN
87.0
75.5
Res5Dr - Res50, B)
SAA
93.5
100.0
92.9
92.1
95.1
82.3
39.6
71.8
80.4
81.7
87.2
Table 6: SAA improves adversarial transferability against adversarial defenses.
Attack
HGD
R&P
NIPS-r3
JPEG
Bit-Red
NRP
Diffpare
Inc-v3
Inc-x3.nst
IncRes-v2... 10.4
Avg. ASR
MI
42.5
21.9
25.3
33.9
42.4
23.6
29.3
6.7
13.8
33.3
31.3
23,3
27.3
ME-SAA
73,3
57.8
60.4
39.6
42.3
12.0
22.4
68.9
65.5
57.9
53.0
SSA-DI-TI-MI-SA SAA
96.0
932
94.8
95.1
94.9
89.8
85.5
20.2
78.8
95.7
94.5
93.3
$5.9
Table 7: Ablation study on alignment modules.
Module
Target Model
Spatial
Adversurial
Global
Local
Res18
Res50
Res101
VGG19
DN121
Inc-v3
VIT-B
Swin-B
PVT-22
MobVIT
60.4
99.8
56.4
60.3
67.3
44,3
12.1
35.6
37.2
39.0
45.8
67.5
98.0
58.1
69.1
70.1
37.7
38.9
47.7
50.1
81.6
97.9
68.1
79.8
63.6
25.5
47.9
48.7
51.3
84.1
99.6
74.7
80..3
81.8
65.7
24.3
48.7
52.9
62.5
63.9
N JPEG 19
FD 36
RS 10 Bit-Red [2], NRP 41. Diffpure 43
InC-v3enss [50]
Inc-v3ensd 50 and IncRes-v2ens [0) Here we use the ensemble attack setting, and the surrogate ViT-B models are (esRReeSSO=-SSSDDN221-11111 + Reeee500SSwnnn
As shown in Table 6 although adversarial defenses weaken transferability, SAA still achieves a significant improvement in adversarial transferability compared to the origin surrogate models.
3.5 Ablation Studies
We select ResNet-50 as the surrogate model and ViT-B as the witness model for ablation studies (see Appendix
C for more details, including training epochs, distance metric, GPU memory and computing cost, self-adversarial strategy).
Alignment Module. In spatial-aware alignment, 'global' represents Lygobblt (Equation Ty while 'local' represents Liocal (Equation 2} Similarly, 'adversarial" represents LAS (Eqution 5 of adversarial- aware alignment. As shown in Table when global features are introduced into the alignment, the transferability of the aligned surrogate model will increase by 3.6%. Based on Lotobal, when only local features are introduced, the overall transferability is improved by 4.3% due to better alignment of features of different architectures in local regions, especially by 8.7% on MobViT. Since intermediate layer features between models are diverse and different, we choose last layer features that summarize the spatial information, which simple and effective strategy. Based on Lioobel. when only adversarial features are introduced, the transferability is greatly improved, reaching 12.6% ASR, and the improvement significant on ViTs. Finally, by integrating all features, the aligned surrogate model achieves state-of-the-an transferability.
4 Conclusions
In this study, we introduce a novel technique called Spatial Adversarial Alignment (SAA), which incorporates alignment loss function and utilizes witness model to fine-tune surrogate model by focusing on both spatial-aware and a adversarial-aware alignments. Through experimental analysis, we

---

show that leveraging these spatial and adversarial features for model aligmment significantly enhances the adversarial transferability of surrogate models, with particularly pronounced improvement in their cross-architecture capabilities. SAA not only integrates seamlessly with existing transfer attack strategies but also further amplifies adversarial transferabilitty thereby contributing more complete evaluation of the adversarial robustness of DNNs.
Boarder Impacts. Adversarial examples by SAA exhibit enhanced adversarial transferability, especially in cross-architecture capabilities. This poses huge threat to the deployment of real-world applications. Simultaneously, it is also conducive to better evaluating their adversarial robustness.
Limitations. The huge gaps between network architectures limit transferability. While SAA aligns unique features under both spatial and adversarial conditions to mitigate these gaps, it does not fully resolve them. Then, SAA also introduces some computing costs during the alignment but no extra costs during attacking. In addition, theoretical analysis is lacking. We have made significant progress, but there is still much to be done in addressing this issue.
Acknowledgments. This work was supported by National Natural Science Foundation of China (No.62576109, 620
10

---

References
[1] Junyoung Byun, Myung-Joon Kwon, Seungju Cho, Yoonji Kim, and Changick Kim. Introducing competi- tion to boost the transferability targeted adversarial examples through clean feature mixup. In IEEE/CVF Conference on Computer Vixion and Pattern Recognition, CVPR 2023, Vancouver, BC. Canada, June 17-24, 2023, pages 24648-24657 IEEE, 2023.
[2] Yanbei Chen, Yongqin Xian, A. Sophia Koepke, Ying Shan, and Zeynep Akata. Distilling audio-visual knowledge by compositional contrastive learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 7016-7025 Computer Vision Foundation IEEE, 2021.
[3] Zhaoyu Chen, Bo Li, Kaixun Jiang, Shuang Wu, Shouhong Ding, and Wenqiang Zhang. Exploring decision-based black-box attacks on face forgery detection. arXiv preprint arXiv:2310.12017. 2023.
[4] Zhaoyu Chen, Bo Li. Shuang Wu, Shouhong Ding, and Wengiang Zhang. Query-efficient decision-basee black-box patch attack. IEEE Transactions on Information Forensics and Security, 18:5522 5536, 2023,
[5] Zhaoyu Chen, Bo Li. Shuang Wu, Kaixun Jiang. Shouhong Ding, and Wenqiang Zhang. Content-based unrestricted adversarial attack. In Advances in Neural Information Processing Systems, volume 36, pages 51719-51720 2023.
[6] Zhaoyu Chen, Bo Li, Shuang Wu, Jianghe Xu, Shouhong Ding, and Wenqiang Zhang. Shape matters: deformable patch attack. In European Conference on Computer Vision, pages 529-548. Springer, 2022.
Zhaoyu Chen, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Wenqiang Zhang. Towards practical certifiable patch defense with vision transformer. In Proceedings of the IEFF//CF Conference on Computer Vision and Pattern Recognition, pages 15148-15158, 2022.
[8] Zhengsu Chen, Lingxi Xie, Jianwei Niu, Xuefeng Liu, Longhui Wei, and Qi Tian. Visformer: The vision-friendly transformer. In 2021 IEEE/CVF International Conference Computer Vision, 2021, Montreal, QC, Canada, October 10-17, 2021. pages 569-578. IEEE, 2021.
[9] Hai Ci, Pei Yang, Yiren Song, and Mike Zheng Shou. Ringid: Rethinking tree-ring watermarking for enhanced multi-key identification. In European Conference on Computer Vision, pages 338-354. Springer. 2024.
[10] Jeremy M. Cohen, Elan Rosenfeld, and J. Zico Kolter. Certified adversarial robustness via randomized smoothing. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th Inter- national Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 1310-1320. PMLR, 2019.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei L.. Imagenet: A large-scale hicrarchical image datahase. In CVPR, pages 248-255, 2009.
[12] Terrance DeVries. Improved regularization of convolutional neural networks with cutout, arXiv preprint arXiv:1708. 04552, 2017.
[13] Yinpeng Dong. Fangzhou Liao, Tianyu Pang. Hang Su, Jun Zhu, Xiaolin Hu, Jianguo Li. Boosting adversarial attacks with momentum. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 9185 9193, Computer Vision Foundation/ / IEFE Computer Society, 2018
[14] Yinpeng Dong. Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversarial examples by vannslatoon-nvariiant attacks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4312-4321, 2019.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakoh Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021.
[16] F. Croce and M. Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter- free attacks. volume 1 119, pages 2206-2216, 2020.
[17] Aditya Ganeshan, Vivek BS, and R Venkatesh Babu. Fda: Feature disruptive attack. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 006998879, 2019.
11

---

[18] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
[19] Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens Van Der Maaten. Countering adversarial images using imput transformations, arXiv preprint arXiv: 2017.
[20] Pinxue Guo. Wanyun Li, Hao Huang. Lingyi Hong, Xinyu Zhou, Zhaoyu Chen, Jinglun Li, Kaixun Jiang, Wei Zhang, and Wenqiang Zhang. X-prompt: Multi-modal visual prompt for video object segmentation. In Proceedings of the 32nd ACM International Conference on Multinedia, pages 5151-5160, 2024.
[21] Yiwen Guo, Qizhang Li, and Hao Chen. Backpropagating linearly improves transferabilitty of adversarial examples. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[22] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[23] Lingyi Hong, Wenchao Chen, Zhongying Liu, Wei Zhang, Pinxue Guo, Zhaoyu Chen, and Wenqiang Zhang. Lvos: A benchmark for long-term video object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13480- 13492, 2023.
[24] Lingyi Hong. Shilin Yan, Renrui Zhang, Wanyun Li, Xinyu Zhou, Pinxue Guo, Kaixun Jiang, Yiting Chen, Jinglun Li, Zhaoyu Chen, et al. Onetracker: Unifying visual object tracking with foundation models and efficient tuning. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 19079-19091, 2024.
[25] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, pages 2261-2269, 2017.
[26] Kaixun Jiang, I Lingyi Hong, Zhaoyu Chen, Pinxue Guo, Zeng Tao, Yan Wang, and Wenqiang Zhang. Exploring the adversarial robustness of video object segmentation via one-shot adversarial attacks. In Proceedings of the 31st ACM International Conference on Multimedia, pages 8598-8607, 2023.
[27] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial intelligence safety and security, pages 99 112. Chapman and Hall/CRC, 2018.
[28] Fangzhou I Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks using high-level representation guided denoiser. In 2018 IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 1778-1787. Computer Vision Foundation/ IEEE Computer Society, 2018
[29] Jiadong Lin, Chuanbiao Song, Kun He, Liwei Wang, and John E. Hopcroft. Nesterov accelerated gradient and scale invariance for adversarial attacks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababo, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[30] Daizong Liu and Wei Hu. Imperceptible transfer attack and defense on 3d point cloud classification. IEEE transactions on pattern analysis and machine intelligence, 45(4):4727-4746, 2022.
[31] Daizong Liu, Wei Hu, and Xin Li. Point cloud attacks in graph spectral domain: When 3d geometry meets graph signal processing. IEFE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[32] Daizong Liu, Mingyu Yang, Xiaoye Qu, Pan Zhou, Yu Cheng, and Wei Hu. A survey of attacks on large vision-language models: Resources, advances, and future trends. arXiv preprint (tti::200.0740000 2024.
[33] Siao Liu, Zhanyu Chen, Wei Li, Jiwei Zhu, Jiafeng Wang, Wenqiang Zhang, and Zhongxue Gan. Efficient universal shuffle attack for visual object tracking. In ICASSP 2022-2022 IEFF International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2739-2743. IEEF, 2022.
[34] Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black- box attacks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24 26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[35] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 9992-10002, 2021.
[36] Zihao Liu, Qi Liu, Tao Liu, Nuo Xu, Xue Lin, Yanzhi Wang, and Wujie Wen. Feature distillation: Dnn- oriented jpeg compression against adversarial examples. In 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 60-8868. IEEE, 2019.
12

---

[37] Yuyang Long, Qilong Zhang, Boheng Zeng, Lianli Gao, Xianglong Liu, Jian Zhang, and Jingkuan Song. Frequency domain model augmentation for adversarial attack. In European Conference on Computer Vision, 2022.
[38] Avery Ma, maii-m Farahmand, Yangchen Pan, Philip Tor, and Jindong Gu. Improving adversarial transferability via model alignment. In European Conference on Computer Vision, pages 74 92, Springer, 2024.
[39] Aleksander Madry, Alcksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
[40] Sachin Mehta and Mohammad Rastegari. Mobilevit: Light- general-purpose and mobile-friendly vision transformer. In The Tenth International Conference on Learning Representations, ICLR 2022, Viriual Event, April 25-29, 2022, 2022.
[41] Muzammal Naseer, Salman H. Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli. A self- supervised approach for adversarial robustness. In 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 259-268 Computer Vision Foundation IEEE, 2020.
[42] Muzammal Nascer, Kanchana Ranasinghe, Salman Khan, Fahad Shahbaz Khan, and Fatih Porikli. On improving adversarial transferability of vision transformers. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[43] Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. In International Conference on Machine Learning (ICML), 2022.
[44] Zeyu Qin, Yanho Fan, Yi Liu, Li Shen, Yong Zhang, Jue Wang, and Baoyuan Wu. Boosting the transfer- ability of adversarial attacks with reverse adversarial perturbation. In Advances in Neural Information Prucessing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA. USA, November 28- December 9, 2022, 2022.
[45] Ramprasaath R. Selvaraju. Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based localization. Int. J. Comput. Vis., 128(2)33363355 2020.
[46] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In Yoshua Bengio and Yann LeCun, editors, ICI.R, 2015.
[47] Zhimin Sun, Shen Chen, Taiping Yao, Ran Yi, Shouhong Ding, and Lizhuang Ma. Rethinking open-world deepfake attribution with multi-perspoctive sensory learning. International Journal of Computer Vision, 2024.
[48] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV. USA, June 27-30, 2016. pages 2818 2826, 2016.
[49] Hugo Touvron, Matthiuu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 32-42. JEEE, 2021.
[50] Florian Tramèr, Alexey Kurakin, Nicolas Paperoo,, Ian J. Goodfellow, Dan Bonch, and Patrick D. Mc- daniel. Ensemble adversarial training: Attacks and defenses. In International Conference on Learning Representations, 2018
[51] Jiafeng Wang, Zhaoyu Chen, Kaixun Jiang, Dingkang Yang. Lingyi Hong. Pinxue Guo, Haijing Guo, and Wenqiang Zhang. Boosting the transferability of adversarial attacks with global momentum initialization. Expert Systems with Applications, 255:174757, 2024.
[52] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. PVT v2: Improved baselines with pyramid vision transformer. Comput. Vis. Media, 8(3)-115-424 2022.
[53] Xiaosen Wang and Kun He. Enhancing the transferability of adversarial attacks through variance tuning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 1924 1933, Computer Vision Foundation IEEE, 2021.
13

---

[54] Xiaosen Wang, Xuanran He, Jingdong Wang. and Kun He. Admix: Enhancing the transferability of adversarial attacks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16158-16167, 2021.
[55] Xiaosen Wang, Kangheng Tong, and Kun He. Rethinking the backward propagation for adversarial transferability. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023. 2023.
[56] Xiaosen Wang, Zeliang Zhang, and lianping Zhang. Structure invariant transformation for better adversarial transferability. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4607-4619. October 2023.
[57] Zeyu Wang. Xianhang Li, Hongru Zhu, and Cihang Xie. Revisiting adversarial training scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24675-24685, June 2024.
[58] Zhipeng Wei, Jingjing Chen, Micah Goldblum, Zuxuan Wu, Tom Goldstein, and Yu-Gang Jiang. Towards transferable adversarial attacks on vision transformers. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intellignce, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22- March 1, 2022. pages 2668 166882666..26 AAAI Press, 2022.
[59] Dongxian Wu, Yisen Wang, Shu-Tao Xia, James Bailey, and Xingjun Ma. Skip connections matter: On the transferability of adversarial examples generated with resnets. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[60] Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. arXiv preprint 1eXiv:1111001900 2017.
[61] Cihang Xie, Zhishuai Zhang. Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan Yuille. Improving transfcrability of adversarial examples with input diversity. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 2730-2739, 2019.
[62] Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. In 25th Annual Network and Distributed System Security Symposium, NDSS 2018, San Diego, California, USA, February 18-21, 2018. The Internet Society, 2018.
[63] Dingkang Yang. Shuai Huang. Zhi Xu, Zhenpeng Li, Shunli Wang. Mingcheng Li, Yuzheng Wang. Yang Liu, Kun Yang. Zhaoyu Chen, et al. Aide: A vision-driven muet, multi-modal, multi-tasking dataset for assistive driving perception. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 20459-20470, 2023.
[64] Xiao Yang, Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Boosting transferabiliit of targeted adversarial examples via hierarchical generative networks. In Computer Vision ECCV 2022 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IV, volume 13664 of Lecture Notes in Computer Science, pages 725-742. Springer, 2022.
[65] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
[66] Jianping Zhang, Yizhan Huang, Weibin Wu, and Michael R. Lyu. Transferable adversarial attacks on vision transformers with token gradient regularization. IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 16415-16424, IEEE, 2023.
[67] Jingyu Zhang, Kun Yang. Hanqi Wang, Peng Sun, and Liang Song. Efficient vehicular collaborative perception based on saptial-temporal feature compression. IEEE Transactions on Vehicular Technology, 2024.
[68] Jingyu Zhang, Kun Yang, Yilei Wang. Hanqi Wang, Peng Sun, and Liang Song. Ermvp: Communication- efficient and collaboration the /REE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12575-12584, June 2024.
perception in challenging euvironments. In Proceedings of
[69] Tianyi Zheng, Bo Li. Shuang Wu, Ben Wan, Guodong Mu, Shice Liu, Shouhong Ding, and Jia Wang. MFAE: masked frequency autoencoders for domain generalization face anti- aati-spooiing IEEE Trans. Inf. Forensics Secur., 19:4058-4066 2024.
14

---

[70] Tianyi Zheng, Qinji Yu, Zhaoyu Chen, and Jia Wang. Famim: A novel frequency-domatt augmentation masked image model framework for domain generalizable face anti-spoofing. ICASSP 2024 2024 IEEE International Conference on Acousticx, Speech and Signal Prucessing (ICASSP), pages 4470-4474, 2024.
[71] Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Animashree Anandkumar. Jiashi Feng, and M Alvarez. Understanding the robustness in vision transformers. In International conference on machine learning, pages 27378-2739) PMLR, 2022.
[72] Xinyu Zhou, Pinxue Guo, Lingyi Hong. Jinglun Li, Wei Zhang, Weifeng Ge, and Wenqiang Zhang Reading relevant feature from global representation memory for visual object tracking. In Advances in Neural Information Processing Systems, volume 36, pages 10814-10822 2023.
15

---

This document provides more details of Spatial Adversarial Alignment (SAA), organized as follows:
In Section A we describe related work to this paper, including transfer attacks on CNNs
and VIT.
Section we describe the detail loss calculation, corresponding to Section 2.2 of the main body.
In Section C we supplement the more ablation studies, corresponding to Section 2.4.3 of the main body.
In Section D we supplement feature-based transfer attacks with SAA, corresponding to Section 2.4 of the main body.
In Section B present additional Grad-CAM visualizations target models analyze the impact of SAA.
We provide the implementation our Spatial Adversarial Alignment (SAA) in the Core Codes of Supplementary Material.
A Related Work
A.1 Transfer attacks CNNs
Early transfer attacks are mainly conducted between CNNs, and the most popular methods were
advanced optimization 13 29 data augmentation 61
14 37 and model modification (59] 21
58).
Advanced Optimization. methods can obtain models with better generalization, and therefore also generate adversarial ex- amples with higher transferabiliity. FGSM [18] is the earliest gradient-based based transfer attack, which was then extended to I-FGSM (27). The subsequent advance optimization further improves the
29 compare adversarial attacks model training: better optimization
transferability by introducing momentum [13
29 53 51
and smoothness
44
Data Augmentation. Data augmentation serves as an effective strategy to prevent model overfitting, achieving state-of-the-art performance in model generalization [65] 12]. Building this principle, numerous adversarial attacks incorporate various transformations to enhance adversarial transfer- ability, including modifications in size [61], scale (29). mixup (54), and frequency domain (37) adjustments. This integration aims to mitigate the overfitting of adversarial examples to the surrogate model, thereby increasing their effectiveness across different models.
Model Modification. According to certain characteristies of the model, modifying the parameters of the surrogate model or changing the forward or backward propagation can also improve the transfer- ability. Skip Gradient Method (SGM) [59] using more gradients from the skip connections rather than the residual modules, allows one craft adversarial examples with high transferability. Similarly, Linear Backpropagation BPi 21 and Backward Propagation Attack (BPA) (55) concentrate on non-linear activations by modifying the ReLU derivatives to enhance attack transferability. Model Alignment (MA) (38) promotes alignment of model predictions through an alignment loss relative to witness model, with the aim of capturing shared features across models. However, MA overlooks spatial and adversarial feature alignment across architectures, limiting its effectiveness. Unlike these methods, our SAA requires modifications to the forward or backpropagation processes, enabling the efficient generation of highly transferable adversarial examples with minimal training overhead. In contrast, LinBP and BPA, involve altering backpropagation significantly higher computational costs.
full model retraining, incurring
A.2 Transfer attacks on ViTs
Current transfer attacks for ViTs largely adapt methods developed for CNNs. Pay No Attention (PNA) [58] method extends Skip Gradient Method (SGM) to ViTs by omitting the gradient com- putation of attention blocks during pro--propagation. thereby enhancing adversarial transferability. PatchOut (58) strategy selects random subset of image patches to compute the gradient at each iteration, functioning as an image transformation technique to increase transferability. Then, Self- Ensemble (SE) [2 approach employs the class token at each layer with shared classification
16

---

head to create an ensemble model, facilitating optimized perturbation; however, many ViTs, such as Visformer 8 and CaiT 49 lack sufficient class tokens build this ensemble. Additionally, Token Refinement (TR) 42 module fine-tunes class tokens to further boost transferabilityy Recently, Token Gradient Regularization (TGR) [66] works from the perspective variance reduction, stabilizing the gradient direction to prevent adversarial examples from getting stuck in poor local optima. Distinet from these approaches, SAA is the first method to specifically analyze architectural differences across models. By leveraging shared features between different architectures, SAA enables the creation of more generalized surrogate models that integrate seamlessly with optimization and data augmentation methods, ultimately achieving state-of-te cae--of-thaette art transferability.
Algorithm Loss LSAA
Require: Input x, surrogate model fe., witness model fe...
Ensure: CSAA
#Preliminaries
2: if fo is CNN then
3: Compute global logits:
[a( Classifier (wwoeealloollaoooonnnneeeeeeeeee
4:
Obtain local regions collection z:
z0(x)) Classifier( BaakkeoeeeB
5:
else if f ViT then
6:
z' TransformerEncoderr(t)
7:
Compute global logits:
fz) Classitier(="
8: Obtain local regions collection 2:
20(x)) Classifier :1)
9: end if
#Step 1: Compute spatial-aware alignment loss
11: Obtain global logits fa, (x) and Sa., (x)
12: Compute Liglbbal (a:; 0g) DRL.(Jo (x), for, (x:)
13: for each local region do
14: Obtain local logits =/0 and [o]
15: Derive pseudo-label 0%0 arg max 20
(x:)
Compute local alignment loss for region q: DCE(2 (2
(z), )lt)
16:
17:
end for Calculate total local alignment
18:
loss:
Liocal 0s)
#W E Dc (x),
1
(x)., arg m
[9] (x)))
19: Calculate spatial-aware alignment loss: (2, = Eytobal(1 0.) + ? Llocal 0s)
#Step 2: Compute adversarial-aware alignment loss
21: Initialize adversarial example I =
22: for each iteration do
23: Fade (1+1) 1T.,, Fale + a sign D.. (z for (x))))
24: end for
25: Obtain final adversarial example Tadv = x (T)
adv
26: Calculate adversarial-awar alignment loss:
Lad(Fadui 0x) Lglobal 0x) +
Lroeat (aad; 0.)
27: #Step 3: Calculate spatial adversarial alignment loss
28: LSAA(E; (x; 0.) = LSA (z; 0g) + R. LAA (Radv; 0s)
Return CSAA
B Loss Calculation
We introduce the calculation of loss in detail, as shown in Algorithm T tion, please refer to the code provided in Supplementary Material.
For the specific implementa-
17

---

Table 8: Ablation study on distance metrics of global features.
Target Model
Loss
Avg. ASR
CNNs
ViTs
Res18
Res50
Res101
VGG19
DN121
Inc-v3
ViT-B
Swin-B
PVT-v2
MobVIT
n/a
57.7
99.9
58.1
54.2
55.1
33.0
38.0
35.7
42.2
KL
84.1
99.6
74.7
80.3
81.8
65.7
24.3
48.7
52.9
62.5
63.9
TV
62.1
97.8
56.2
67.5
63.9
47.6
9,4
32.2
37.2
44,1
46.7
J5
83.3
99.1
70.0
78.2
80.9
63.5
22.8
46.4
49.3
58.5
61.4
NCE
99.3
69.1
77.0
Table 9: Ablation study on different supervision in self-adversarial strategy.
Source Witness
Attack
Res18
Res50
Res101
VGG19
DN121
IncV3
VIT-B Swin-B PVT-v2 MobVIT Avg. ASR (%)
MA
74,2
99.2
63.5
69.3
72.8
51.5
18.5
41.5
42.7
47.4
53.5
Res50
VIT-B SAA (a))
74.7
80.3
81.8
65.7
24.3
48.7
52.9
62.5
63.9
SAA (fo (r))
81.1
96.6
68.7
77.5
79.5
60.6
19.4
45.9
46.2
57.6
59.6
C Ablation Study
We select ResNet, DenseNet, and Swin, all of which have won Best Paper Awards, and ViT, which pioneering work. These models are representative are all from timm, ensuring the generalization of our conclusions. We conduct 16 combinations of experiments and evaluate on CNNs and 4 ViTs, showing that SAA's improvement not limited to specific choices of witness models.
both CNNs and ViTs and their weights
Training Epochs. In Section 3.2 we reveal the powerful potential of SAA for adversar- ial transferability after training for only one epoch. Furthermore, we explore the perfor- mance difference after training for multiple epochs, as shown in Figure 3 We calcu- late the average attack success rate except for the Res50 surrogate model itself and find
50
75
SAA
70
65
ASR
60
55
that with the increase of epochs, the adver- sarial transferability of the aligned surrogate model further improved, reaching conver- gence around the 9-th epoch. Compared with MA, SAA can achieve higher transferability in small epochs, and after multiple rounds of training. the transfcrability has a higher upper limit, which shows the importance of using spatial and adversarial features for model alignment.
50
45
40
Epochs
Figure 3: Ablation study on training epochs.
Distance Metric. There are many ways to align the output of global features. To better align the global features, we select Kullback-Leible Divergence (KL), Total Variation (TV), Divergence (JS), and NCE 2 loss for evaluation. We select ResNet- as the surrogate model and ViT-B as the witness model for ablation studies. As shown in Table 8 KL exceeds the next best JS by 2.5%, so we choose KL as the metric.
GPU Memory and Computing Cost. SAA's GPU memory is split into alignment and attack. In alignment, we infer the witness model and train the surrogate model for one epoch, the memory usage is the sum of both. When the surrogate is Res50 and the witness is ViT-B, SAA's GPU memory only 2794MB with a batch size of Once alignment is complete, attack costs only are an aligned surrogate model, making SAA's memory the same as SOTA attacks without extra memory. Thus, its GPU memory is acceptable. For computing cost, SAA needs about 10 hours' training time on ImageNet with batch size of 64 under Nvidia RTX 3090. The total computational cost of MA is hours with the same setting. Since adversarial-aware alignment introduces the solution of adversarial examples, in the practice of SAA, model training takes about 2 hours, and solving adversarial examples takes about hours, for total of 10 hours. Note that the main computational bottleneck is actually obtaining adversarial examples, not model training. Some newer adversarial
18

---

Table 10: Ablation study on label supervision in local alignment.
Source Witness
Attack
Res18
Res50
Res101
VGG19
DN121
In:
VIT-B Swin-B PVT_y2 v2_2 MobViT Avg. ASR (9)
SAA-MI (Ours)
84.1
99.6
74.7
80.3
81.8
65.7
24.3
48.7
52,9
62.5
63,9
VIT-B
SAA-MI (soft)
81.9
96.4
65.9
74.5
77.7
60.8
18.8
44.4
43.1
53.0
57.8
Table 11: SAA improves the adversarial transferability of feature-based based transfer attacks.
Target Model
Attack
CNNs
ViT'
Avg. ASR (%)
Res18
Res50
Res101
VGG19
DN121
Inc-v3
VIT-B
Swin-B
PVT-v2
MobVIT
FDA
55.2
31.6
36.8
52.0
43.6
27.1
2.7
21.0
24.5
27.8
32.3
FDA-SAA
82.9
79.0
72.5
79.1
77.1
60.1
14.4
39.5
43.4
58.4
58.6
FDA-MI
75.0
46.1
60.6
74.0
65.3
46.0
8.4
27.3
29.4
47.0
48.1
FDA-MI-SAA
89.7
88.2
81.7
86.9
85.3
73.1
21.9
41.7
44.7
68.9
66.0
attack techniques [57 may be able to accelerate this process and reduce the time to 1/3 of the original, which is auspicious research direction.
Differnet supervision in self-adversariall strategy. In SAA, adversarial-aware alignment is per- formed using the velf-aaversarial strategy, where the supervision comes from global features of the witness model. Here, we investigate whether supervision from global features of the surrogate model or global features of the witness model is more effective, and compare both approaches with the MA 38) baseline. The results in Table 9 show that SAA (Jo.. (z)) outperforms SAA (fo. (x)) by 4.3% in ASR, indicating that fea (z:) provides stronger supervision for adversarial- -ware alignment, leading to highly transferable adversarial examples. This highlights the effectiveness of alignment using Ja_ (ar) and its superiority over fa, (zr), demonstrating that leveraging witness model features enhances adversarial robustness.
Label supervision in local aligmment. We add experiments using soft labels (logits) and the KL divergence loss function for local alignment, as shown in Table [0] The surrogate model is Res50 and the witness model is ViT-B. In this scenario, using hard labels (local pscudo-labels) achieves higher adversarial transfcrability (improving ASR by 6.1%).
D Feature-based Transfer Attacks
Here we combine SAA with feature-based transfer attacks for experiments, as shown in Table Here select ResNet-50 as the surrogate model and VTT-B the witness model. Since the aligned surrogate model after SAA can learn more common features, SAA further improves attack performance of FDA [17]. On FDA and FDA-MI, SAA improves the black-box attack success rate by 25.7% and 17.1%, respectively.
E Grad-CAM Visualization
We present Grad-CAM visualizations on two target models: ResNet-101 and ViT-S, as shown in Figure Each set consists of three columns: the first column shows the Grad-CAM heatmap of the target model on clean input; the second column displays the heatmap for adversarial examples generated by the unaligned model; the third column corresponds to adversarial examples generated by SAA aligned model. The visualizations indicate that adversarial examples from the unaligned model produce heatmap distributions more similar to those of clean inputs, whereas adversarial examples from the aligned model effectively disrupt the target model's attention distribution, leading to a more thorough attack.
F Security Implications
Potential misuse. SAA improves the cross-architecture adversarial transferability of surrogate models, which means attacks against unknown systems pose a higher threat. For example, for face forgery detection system [3] open-source models typically CNNs, while the most advanced
19

---

Clean
Unaligned-Ady
Aligned-Ady
Clean
Unaligned-Adv
Aligned-Ady
ResNet-101
ViT-S
Figure 4: Grad-CAM on target models.
commercial applications may already be ViT-based models. In this case, using existing CNN detectors as surrogate models, combined with SAA, can lead to successful attack on commercial applications.
Mitigation strategies. We think that potential defense strategies primarily involve introducing adversarial training and increasing model diversity. The former recommends introducing small number of adversarial examples during model training to improve robustness against adversarial examples. The latter recommends introducing multi-model ensemble training during training to increase the model's 's feature complexity and enhance deployment robustness.
20

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction has accurately reflected the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: This paper has discussed the limitiations of this work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
This paper doesn't 't provide the theoretical analysis.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the detailed parameters and release the codes to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer:
[Yes]
Justification: The data is publicty available and we release the codes to reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
wtt.:g/dddeeeeeSSuuurrtr ssicnlolicy) for more details. hetpoolliigguddde/ooooooo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the detailed parameters and release the codes to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Because of the huge computing cost.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
23

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe the the computer resources Apppendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wwaarpccccppb/a/ccccccaa
Answer:
[Yes]
Justification: Everything preserves anonymity.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss this in last section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original paper that produced the code package dataset.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
26

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
27


Reviews:
[{"review_id": "TYomXDlGvr", "reviewer": "Reviewer_uYz7", "summary": "This paper proposes Spatial Adversarial Alignment (SAA), a method for enhancing the transferability of adversarial examples across architectures, especially from CNNs to ViTs. The method leverages a witness model to fine-tune a surrogate model by aligning both global and local features (spatial-aware alignment) and adversarial representations (adversarial-aware alignment). Experiments show that SAA improves adversarial transferability significantly over prior model alignment (MA) methods and can enhance the performance of multiple transfer attacks under black-box and ensemble settings.", "questions": "How robust is SAA to witness model choice? If the witness model is very weak (e.g., shallow CNN), does alignment still help? \n\nWhat is the compute cost of SAA training? How many GPU hours are needed per model alignment? This is especially relevant for large ViTs. \n\nWhat happens if the surrogate and witness have no architectural overlap (e.g., MobileNet and Swin-B)? Does SAA still improve transferability? \n\nHow often do pseudo-labels differ from the true labels? Have you analyzed the reliability of local feature alignment via argmax over logits? \n\nWhy use KL divergence for local logits? Did the authors try feature-matching (e.g., MSE between embeddings), or attention-head alignment as in recent interpretability work? \n\nIn transfer settings where the target model is known (but frozen), wouldn’t it make more sense to align to it directly rather than via a witness?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\nThe paper presents an array of novel contributions and new ideas. SAA integrates spatial-level and adversarial-level feature alignment, going beyond prior work that focused only on logits or global outputs. \n\nThere are strong empirical results supported by extensive experiments across 6 CNNs and 4 ViTs show substantial gains over existing methods, especially under cross-architecture black-box settings. \n\nSAA supports multiple attack types. Unlike many prior works that improve a single attack (e.g., I-FGSM), SAA shows gains when applied to multiple families: MI, DI, TI, PatchOut, GI, etc. \n\nSAA significantly boosts attack transferability between very different model families (e.g., CNN → ViT), which is a difficult and under-addressed problem. \n\nThe framework is designed as a training-time plug-in that does not change the attack itself — this modularity is appealing to practitioners. \n\nWeakenesses\n\nThe compute budget is not reported. Surrogate model retraining with adversarial alignment is costly. The paper lacks reporting on total training time, epochs, GPU hours, or number of adversarial examples generated. \n\nThe security implications are unexplored. By boosting black-box adversarial transferability, SAA may exacerbate real-world model vulnerabilities, however the paper doesn’t reflect on the responsible disclosure or ethical implications of this. \n\nThere appears to be an overreliance on cosine similarity as a metric: Feature similarity is evaluated only via cosine similarity, which may not correlate strongly with transferability. A deeper metric (e.g., centered kernel alignment or probing transfer success vs. feature similarity) would add insight. \n\nLabel correctness of adversarial examples is not verified. The method assumes that adversarial examples retain their semantic label under perturbation and alignment, but this is not validated with human or automatic checks. \n\nThe \"pseudo-labeling\" step for local features may introduce noise. Aligning local logits using pseudo-labels from a potentially miscalibrated witness model could misguide learning, but no analysis or ablation is presented. \n\nTraining the surrogate to mimic the witness may lead to overfitting, reducing diversity in learned features. There’s no metric to evaluate how independent or useful the aligned features are post-attack.", "was_revised": true, "final_justification": "In my opinion, the paper does contribute to the idea of Spatial Adversarial Alignment. The key work that needs to be address is the foundational differences between architectures like CNNs and ViTs and the issue around SAA. This would allow the paper to be a notch up.", "rebuttal": "**Weakness 1 & Question 2:** The compute budget is not reported (total training time, epochs, GPU hours, or number of adversarial examples). \n\n We would like to clarify that the relevant information is already included in the paper, and we will consolidate it into the \"Implementation Details\" section in the final version for better accessibility. Specifically:  \n\n1. In \"Training Epochs\" of Appendix C, we note that all experiments in this paper use 1 training epoch. As described in Algorithm 1, the number of adversarial examples generated by SAA is in a 1:1 ratio with the training dataset. Additionally, we explore performance differences across multiple training epochs and find that compared to MA, SAA achieves higher transferability even with fewer epochs, and its transferability reaches a higher upper bound after multiple training rounds.  \n\n2. In \"GPU Memory and Computing Cost\" of Appendix C, we report the training time (GPU hours). For computing cost, training SAA for one epoch on ImageNet (with a batch size of 64) requires approximately 10 hours using an Nvidia RTX 3090.  \n\n---\n\n**Weakness 2:** The security implications are unexplored.\n\nWe would like to clarify that the relevant information is already included in Lines 333-335 of the main body: Adversarial examples by SAA exhibit enhanced adversarial transferability, especially in cross-architecture capabilities. This poses a huge threat to the deployment of real-world applications. Simultaneously, it is also conducive to better evaluating their adversarial robustness.\n\n\n---\n\n**Weakness 3:** Overreliance on cosine similarity.\n\nIn Section 2.3, our core focus is to verify whether SAA improves the **spatial and adversarial features** of surrogate models after alignment. Regarding the use of cosine similarity: since spatial features here refer to representations of entire images (not localized regions), cosine similarity serves as an intuitive and widely adopted metric to quantify global feature similarity between models. As shown in Table 1 of the main body, two key observations emerge: \n1. there is a significant inherent feature gap between diverse architectures (e.g., Res50 and Swin-B); \n2. SAA consistently reduces this gap, with meaningful improvements (e.g., the 49.3% similarity gain for Res50-Swin-B on adversarial examples). These results quantitatively confirm that SAA effectively identifies shared features between distinct models, which in turn enhances adversarial transferability. \n\nCosine similarity quantitatively demonstrates that SAA can improve the feature similarity between surrogate and witness models, proving the validity of this paper's hypothesis. Regarding adversarial transferability, a more direct comparison can be found in Tables 2 and 3, where ASR (Adversarial Success Rate) better demonstrates attack performance. Given the diversity of CNN and ViT models, the relationship between feature similarity and attack performance has been underexplored. SAA opens this door, making it a promising future research topic.\n\n\n---\n\n**Weakness 4:** Label correctness.\n\nBy definition, adversarial examples are crafted with human-imperceptible perturbations (under $l_ₚ$ norm constraints) that mislead model predictions while preserving the original semantic content. In the field of adversarial machine learning, this property is foundational: the perturbations are strictly bounded to ensure they do not alter the image’s meaning as perceived by humans. This is a well-established convention, as $l_ₚ$-constrained perturbations (e.g., $ε = 16/255$ for $l_∞$ setting) are designed to be visually negligible and thus do not interfere with human recognition of the semantic label.  \n\nGiven this standard, verifying individual adversarial examples via manual or automatic checks for semantic label preservation is generally unnecessary in the literature, as the perturbation bounds inherently guarantee that the original label remains semantically valid. Our work adheres to this convention, with perturbations constrained to lₚ norms that ensure such preservation.  \n\n---\n\n**Weakness 5 & Question 4 & Question 5:** Pseudo-labels' noises.\n\nWe should note that local pseudo-labels are not directly related to true labels because we learn local features of the surrogate and witness models, not global features.\n\nFurthermore, we add experiments using soft labels (logits) and the KL divergence loss function for local alignment, as shown below. The surrogate model is Res50 and the witness model is ViT-B. In this scenario, using hard labels (local pseudo-labels) achieves higher adversarial transferability (improving ASR by 6.1%).\n\n\n|          Attack         | Res18 | Res50 | Res101 | VGG19 | DN121 | IncV3 | ViT-B |  SWIN | PVT_v2 | MobViT | Avg. ASR (%) |\n|:-----------------------:|:-----:|:-----:|:------:|:-----:|:-----:|:-----:|:-----:|:-----:|:------:|:------:|:------------:|\n|      SAA-MI (ours)      | 84.1  | 99.6  |  74.7  | 80.3  | 81.8  | 65.7  | 24.3  | 48.7  |  52.9  |  62.5  |     63.9     |\n| SAA-MI (soft label, KL) | 81.9  | 96.4  |  65.9  | 74.5  | 77.7  | 60.8  | 18.8  | 44.4  |  43.1  |  53.0  |     57.8     |\n\nFinally, we include experiments with different distance metrics (which can be understood as embedding metrics) in the \"Distance Metric\" section of Appendix C. Here, we select Kullback–Leibler Divergence (KL), Total Variation (TV), Jensen-Shannon Divergence (JS), and NCE loss for evaluation. We select ResNet-50 as the surrogate model and ViT-B as the witness model for ablation studies. As shown in Table 8, KL exceeds the next best JS by 2.5%, so we choose KL as the metric.\n\n---\n\n**Weakness6:** Overfitting the witness.\n\nThe ultimate metric for evaluating the effectiveness of aligned features is the average attack success rate on the black-box model, not the accuracy of the surrogate model itself on ImageNet. In this paper, our alignment was trained for only one epoch, making it less susceptible to overfitting. Secondly, in \"Training Epochs\" of Appendix C, we further explore performance differences across multiple training epochs. While the transferability of MA converges rapidly with increasing epochs, SAA continues to improve until the ninth epoch. This indicates that MA easily overfits to the witness model, limiting its adversarial transferability. SAA, on the other hand, is less susceptible to overfitting, and its transferability reaches a higher upper bound after multiple training rounds, demonstrating the importance of using spatial and adversarial features for model alignment.\n\n\n---\n\n**Question 1 & Question 3:** Witness model choice.\n\nIn Section 3.2 of the main body, we have conducted a large number of experiments to demonstrate that SAA is robust to model selection. Although we conduct 16 combinations and evaluate on 6 CNNs and 4 ViTs, showing that SAA’s improvement is not limited to specific choices of witness models. Based on the above experiments, we further summarize the empirical guidance for the selection of surrogate and witness models:\n1. Self-alignment (surrogate and witness models are consistent) usually only provides minor improvements (it is difficult to learn unique structural features), and alignment between different models improves more significantly.\n2. When the surrogate model is a ViT-like model and the witness model is a CNN-like model, better cross-architecture transferability can be obtained. This is mainly because ViT has stronger performance, while CNN further provides unique structural features.\n\nIf the shallow CNN is a well-trained model (for example, with normal performance on ImageNet), then there can be an improvement. Even when the surrogate model and witness model have no architectural overlap, SAA still improves adversarial transferability. As shown in Table 2 of the main body, when the surrogate model is DN121 and the witness model is Swin-B, SAA improves the average ASR by 10.9% and 12.6% compared to MI and MA, respectively.\n\n\n\n---\n\n**Question 6:** Known target models.\n\nWe should clarify that in a transfer-based black-box setting, no details of the target model are known. We chaim this in Lines 284-286: It should be noted that the target models are unknown in the transfer attack, so no information about the target models can be used for alignment. Therefore, your assumption is invalid. If the target model is known (but frozen), then the target model can be directly selected as the surrogate model, which is equivalent to a white-box attack."}, {"review_id": "0wbl2jp82X", "reviewer": "Reviewer_ZtAM", "summary": "This paper proposes Spatial Adversarial Alignment (SAA), a method that improves adversarial transferability by improving upon the model alignment method. Model alignment increases the transferability of adversarial examples by fine-tuning a source model to align its predictions with those of a fixed witness model, using an alignment loss that captures prediction divergence. SAA introduces two improvement: a spatial-aware alignment that encourages alignment across both global and local regions, and an adversarial-aware alignment that focuses the alignment on adversarially perturbed inputs. Experiments on ImageNet show that SAA achieves improved transferability than the original model alignment approach.", "questions": "See weakness.", "limitations": "Although the paper briefly mentions computational complexity, it represents a major trade-off compared to the original MA method and deserves a more detailed discussion.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. Improving the transferability of adversarial perturbations is a practically relevant goal, and this paper builds on the recent model alignment framework. The proposed SAA method introduces two enhancements—spatial-aware and adversarial-aware alignment—that further improve adversarial transferability.\n\n2. The main experiment in this paper is extensive, including comparison with model alignment on the Imagenet dataset. The ablation study demonstrate the compatibility with various existing transfer-improving methods. Table 7 also helps understand how each component of the SAA method contribute to the overall improvement.\n\n**Weaknesses**\n\n1. Spatial-alignment component of SAA requires clearer motivation and explanation of its formulation.\n    1. Several claims in lines 48–61 require either citations or supporting evidence to justify the motivation. For instance, the paper mentions that cross-architecture transferability is limited because \"features are not aligned in space\" and that \"the spatial features of ViTs and CNNs are different.\" Without references or empirical evidence, these statements feel speculative and make it difficult to understand the spatial alignment component.\n    2. The claim that “MA only uses the final prediction of the model” is misleading. The alignment loss in Eq. (1) of Ma et al. can be applied to intermediate representations, and Table 4 in their paper explicitly explores alignment in the embedding space.\n    3. Section 2.2, particularly the spatial-aware alignment portion, lacks clarity. Several parts (e.g., lines 104–105, 108–112, and 115–122) should include citations.\n    4.  The use of the term “global features” to refer to logits is unconventional and potentially confusing. It is unclear why logits are considered “global” and how that definition helps in the context of alignment.\n    5.  The explanation in lines 136–146 is difficult to follow due to unspecified dimensions. For example, what is the dimensionality of $z$? Assuming $H$ and $W$ are image dimensions, does having $H\\times W$ local regions imply that each pixel is treated as a separate region? Clarifying these points would significantly improve the readability of the spatial alignment method.\n    6. It remains unclear how the proposed local/global feature perspective addresses the cross-architecture misalignment problem raised earlier. A more explicit connection between motivation and method would strengthen the argument. \n\n2. Section 2.3: It seems that “feature similarity” refers to logit similarity, but this should be clarified. Moreover, wouldn’t an increase in cosine similarity naturally be expected from any source model that has improved transferability? (i.e., unaligned model A vs unaligned model B, where perturbations from B has better transferablity.) Also, a comparison with MA could potentially better understand the improvement from SAA.\n\n3. Computational overhead: The paper does not discuss the computational cost introduced by the two new alignment components. A quantitative analysis (e.g., number of additional forward/backward passes or runtime comparison) would be helpful for understanding the trade-offs compared to the original MA method.", "was_revised": true, "final_justification": "My main concern has been addressed by the authors during rebuttal, thus the score increase. However, I remain concerned about the quality of the revised paper given the numerous changes, clarifications, and additional details the authors have committed to include.", "rebuttal": "Thanks for your thoughtful suggestions. During the rebuttal period, **we have made careful efforts to address your concerns**. Additionally, we have provided detailed responses to your other questions. **We earnestly hope that you will reconsider our paper.**\n\n---\n\n**Weakness 1.1:** Supports on SA.\n\nThanks for your valuable feedback on the motivation in lines 48–61. The analysis in lines 48–61 focuses on potential reasons for the limited cross-architecture transferability of MA, and this is directly validated by the qualitative and quantitative experiments in Section 2.3—our core effort to verify whether SAA improves spatial and adversarial feature alignment.  \n\n- For **quantitative evidence**: As detailed in Section 2.3, we use cosine similarity to measure global feature similarity (since spatial features here refer to entire image representations). Table 1 clearly shows two key results: \n  1. there is a significant inherent feature gap between diverse architectures (e.g., Res50 and Swin-B), confirming that spatial features of CNNs and ViTs are distinct;\n  2. SAA consistently reduces this gap (e.g., a 49.3% improvement for Res50-Swin-B on adversarial examples), demonstrating that unaligned spatial features hinder transferability—directly supporting the claim that \"features are not aligned in space.\" These results empirically validate the existence of architectural feature differences and misalignment.  \n\n- For **qualitative evidence**: Grad-CAM visualizations in Figure 2 illustrate that unaligned surrogates exhibit feature distributions distinct from witness models, while aligned surrogates (via SAA) learn more consistent patterns—confirming improved spatial alignment. Additionally, Appendix E provides further visualizations: adversarial examples from unaligned surrogates retain heatmap distributions similar to clean inputs (indicating poor transfer due to misaligned features), whereas aligned surrogates produce adversarial examples that disrupt witness model attention, validating the impact of spatial feature alignment on transferability.  \n\n\nTo address the need for citations, we will explicitly reference relevant literature in the revised version: \"features are not aligned in space\" (supported by PNA [51]), and \"the spatial features of ViTs and CNNs are different\" (supported by [1], [2]).  \n\n\n\n---\n\n**Weakness 1.2:** Final prediction.\n\nThank you for pointing out this concern—we appreciate the opportunity to clarify our statement about Model Alignment (MA, ECCV 2024).  \n\nOur claim that \"MA only uses the final prediction of the model\" is grounded in both MA’s official paper and its implementation, and we aim to clarify this alignment:  \n\n1. First, while MA’s Section 3.1 presents formalizations for alignment across different layers, their Section 3.2 explicitly states: \"Specifically, when $q = l$, the alignment loss measures the divergence between the probability distributions generated by the two models. In this scenario, the Kullback-Leibler (KL) divergence is particularly suitable, due to its effectiveness in measuring distribution differences and its relative ease of implementation in practice\", which aligns with our description of \"using the final prediction-related representations\" of the model.  \n\n2. Second, MA’s publicly available code (model-alignment/src/align.py, lines 35–37) explicitly uses the model’s final output as the basis for computing the alignment loss. This implementation directly confirms that MA focuses on the final prediction-related representations rather than intermediate layers in practice.  \n\nThus, our statement is consistent with MA’s own paper and official code. We welcome reviewers to verify this via their publicly accessible repository.  \n\n---\n\n**Weakness 1.3:** References on SA.\n\nWe will add citations to MA [32] in lines 104–105 and 108–112, and to [2] in lines 115–122. In addition, we will further check the rigor of the description, add the corresponding citations, and update them in the final version.\n\n---\n\n**Weakness 1.4:** Global features. \n\nThank you for pointing this out. We would like to clarify that the term \"global feature\" is primarily used to distinguish it from local features in spatial-aware alignment. This term is used because, in the vision backbone, the final output features reflect global image information.\n\nWe will further refine the description in the main body. First, we will highlight the definition and description of global and local features in lines 130-144. Then, we will further deepen the intuitive understanding of global and local features by combining them with Figure 1. We have provided code in the supplementary materials, which is consistent with the description in the paper and hopes to further bridge any gaps in understanding.\n\n---\n\n**Weakness 1.5:** Dimension explanation.\n\nThank you for pointing out this issue. The $H$ and $W$ here refer to the dimensions of the local features, not the image dimensions. In image classification models, the downsampling factor is typically 32. For a $224 \\times 224$ image, downsampling results in local features of $(224/32)\\times(224/32)=7\\times 7$. In this case, $H=W=7$ so that the dimension of $z$ is $7 \\times 7=49$. We will update this description in the final version.\n\n---\n\n**Weakness 1.6:** Feature misalignment.\n\nThank you for this insightful question, which prompts us to clarify the explicit connection between our motivation (addressing cross-architecture misalignment) and the proposed local/global feature perspective.  \n\nAs summarized in lines 48–61, the limited cross-architecture transferability of MA stems from its exclusive reliance on aligning surrogate and witness models via their final global outputs. While MA assumes feature similarity correlates strongly with transferability, its poor performance across architectures reveals a critical gap: relying solely on final global features fails to capture meaningful similarity in cross-architecture settings. This gap arises because CNNs and ViTs inherently differ in structural designs—including receptive fields (local vs. global), layer stacking mechanisms, and normalization techniques—leading to misaligned spatial and structural features that hinder transferability.  \n\nOur proposed spatial-aware alignment directly addresses this by explicitly bridging local and global features, creating a clear link between motivation and method:  \n- Beyond aligning final global features (as MA does), we additionally focus on learning local regional features. This dual focus targets the structural differences between CNNs and ViTs: local features capture fine-grained spatial patterns critical for CNNs, while global features retain the holistic token-based representations central to ViTs.  \n- By integrating both perspectives, surrogate models learn not just high-level global similarity but also low-level structural nuances unique to each architecture. This enables them to acquire \"unique structural features\" that transcend architectural differences—features that are aligned both spatially (locally) and semantically (globally), thereby mitigating cross-architecture misalignment.  \n\nIn essence, the local/global feature perspective directly responds to the root cause of misalignment (structural differences between CNNs and ViTs) by ensuring alignment at both granularities, strengthening transferability where MA falls short.  \n\n\n---\n\n**Weakness 2:** Feature similarity.\n\nYes, we have explained in line 188 that feature similarity is calculated using cosine similarity of global features. In Section 2.3, we focus on **verifying whether SAA improves the spatial and adversarial features of surrogate models after alignment**. Therefore, we need to verify the feature similarity before and after alignment and the corresponding visualization. Table 2 of the main body already fully demonstrates that the adversarial transferability brought by SAA significantly exceeds that of MA. Therefore, we focus more on the nature of SAA itself in this section. For more qualitative and quantitative analysis, please refer to **Weakness 1**.\n\nWhile it might be obvious that a model with better transferability can improve a surrogate model with poor transferability when used as a witness model, in SAA, a model with average transferability can further improve a surrogate model with better transferability. For example, when Res50 is aligned with ViT-B, Res50 learns unique structural features, allowing the generated adversarial examples to still improve transferability. SAA reveals this phenomenon and demonstrates that there are still many areas of urgent research in feature similarity and adversarial transferability.\n\n---\n\n**Weakness 3 & Question 1:** Computational overhead.\n\nIn \"GPU Memory and Computing Cost\" of Appendix C, we report the training time (GPU hours). For computing cost, training SAA for one epoch on ImageNet (with a batch size of 64) requires about 10 hours using an Nvidia RTX 3090.\n\nHere, we analyze two alignments: spatial-aware alignment supports parallelization, so only one forward and backward propagation is required for loss calculation; for adversarial-aware alignment, since adversarial examples need to be generated, $t$ forward and backward propagations are required. Therefore, the time complexity of training one epoch is $O(1+t)$. In the above practice, spatial-aware alignment and adversarial-aware alignment take about 2 and 8 hours respectively.\n\nNote that the main computational bottleneck is actually obtaining adversarial examples, not model training. Some newer adversarial example techniques [3] may be able to accelerate this process, which is a very promising research direction.\n\n[1] Do vision transformers see like convolutional neural networks?, NeurIPS 2022\n\n[2] Understanding the robustness in vision transformers, ICML 2022\n\n[3] Revisiting Adversarial Training at Scale, CVPR 2024"}, {"review_id": "DhjBV9QLhZ", "reviewer": "Reviewer_Qa5A", "summary": "The authors tackle the problem of cross-architecture adversarial transferability of adversarial attacks. As different architectures have significantly different structures, existing methods are suboptimal when transfering from different model families (e.g., CNN to ViT). Thus, they propose a technique termed Spatial Adversarial Alignment (SAA) consisted of spatial-aware alignment and adversarial-aware alignment. SAA aims to adjust the surrogate model to extract features closely aligned with those of the witness model by aligning both the local and global features between a surrogate and a witness model. SSA performs such alignment both in the clean and adversarial setting. SSA demonstrates significant improvements compared to MA across a wide range of models and demonstrate improvements in transfer attacks and ensemble attacks.", "questions": "- What can we learn from the fact that some architectures of ViT transfers better to some CNN architectures? e.g., Swin-B to VGG?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- Elegant yet practical idea – SAA keeps the training recipe minimal while still delivering substantial transfer-rate gains; the large, multi-model benchmark makes the effect hard to dismiss.\n- Clear exposition – the narrative flows logically from motivation through method to experiments, making the paper easy to read even for newcomers.\n- Comprehensive empirical evidence – the authors supply full ablations (loss terms, distance metrics, training length) and report results on six CNNs, four ViTs, ten attack variants, and a dozen defences, lending strong credibility to the claims.\n\n\n**Weaknesses**\n- Narrow comparative baseline – Table 2 pits SAA only against Model-Alignment (MA), an older 2022. Comparison to more modern methods is missing. \n\n- Limited insight in section 2.3 – the section merely restates that the loss drives global-feature similarity and then confirms that the learned features are, in fact, similar.\n\n- Weak qualitative evidence – the Grad-CAM comparison is based on just two images; furthermore, the surrogate heat-maps appear more diffused rather than genuinely mirroring the witness, so the visual argument feels anecdotal. A systematic, quantitative localisation metric would strengthen the case.", "was_revised": true, "final_justification": "Based on the authors' rebuttal, I maintain my positive score", "rebuttal": "**Weakness 1:** More modern methods is missing.\n\nIn fact, we compared many transfer attacks from the past three years. Among aligned-based methods, **Model Alignment (MA) is quite new, having been accepted by ECCV 2024, not 2022.** The fact that SAA outperforms MA clearly demonstrates its state-of-the-art performance. Furthermore, experiments combining it with other transfer attacks also incorporate a lot of work from the past three years, such as GI (accepted by Expert Systems with Applications 2024).\n\n---\n\n**Weakness 2 & Weakness 3:** Limited insight and Weak qualitative evidence in section 2.3.\n\nThank you for your insightful comments, which help us clarify the focus and limitations of our analysis in Section 2.3. We appreciate the opportunity to address your concerns.  \n\n\nFirst, we acknowledge that qualitatively and quantitatively dissecting differences between model features is inherently challenging. In Section 2.3, our core focus is to verify whether SAA improves the **spatial and adversarial features** of surrogate models after alignment, rather than analyzing local vs. global features in spatially-aware alignment. This distinction is important: the link between such feature improvements and practical transferability has already been validated through quantitative transferability analysis in Table 7, which demonstrates that enhanced feature alignment directly translates to stronger attack performance across architectures.  \n\n\nSecond, regarding the use of cosine similarity: since spatial features here refer to representations of entire images (not localized regions), cosine similarity serves as an intuitive and widely adopted metric to quantify global feature similarity between models. As shown in Table 1, two key observations emerge: \n1. there is a significant inherent feature gap between diverse architectures (e.g., Res50 and Swin-B); \n2. SAA consistently reduces this gap, with meaningful improvements (e.g., the 49.3% gain for Res50-Swin-B on adversarial examples). These results quantitatively confirm that SAA effectively identifies shared features between distinct models, which in turn enhances adversarial transferability.  \n\n\nFinally, we recognize the limitation of the Grad-CAM examples provided in the main body, which are constrained by space. However, these visualizations do illustrate that aligned surrogates learn features more consistent with the witness model—for instance, in Figure 2, aligned surrogates on clean inputs exhibit feature distributions closer to the witness model’s patterns. To address the need for more evidence, we have included additional visualizations and analyses in Appendix E: these show that adversarial examples from unaligned surrogates retain heatmap distributions similar to clean inputs (indicating poor transfer), while those from aligned surrogates effectively disrupt the target model’s attention patterns, leading to more thorough attacks. This strengthens the case that SAA-driven alignment modifies features in a functionally meaningful way.  \n\n\nWe hope these clarifications address your concerns. Thank you again for pushing us to refine our analysis.\n\n---\n\n**Question 1:** Learning from the fact that some architectures of ViT transfers better to some CNN architectures? e.g., Swin-B to VGG?\n\nThank you for this insightful question, which prompts us to delve deeper into the transferability patterns between ViT and CNN architectures.  \n\nAs highlighted in prior work [1], ViTs and CNNs learn distinct feature representations, with notable differences particularly in their early-layer computations. Specifically, [1] demonstrated that aligning ResNet (a CNN) layers to match the hidden representations of a ViT requires modifying more layers than the reverse process (aligning ViT layers to match ResNet). This observation reveals an inherent asymmetry in cross-architecture alignment: ViTs are more adaptable to aligning with CNNs than CNNs are to aligning with ViTs.  \n\nThis asymmetry sheds light on why certain ViT architectures (e.g., Swin-B) transfer better to specific CNNs (e.g., VGG). ViTs, by nature, exhibit a stronger capacity to capture unique structural features—features that are generalizable enough to align with the representational spaces of CNNs. In contrast, CNNs, with their inductive biases toward local spatial hierarchies, are less flexible in adapting to the global, token-based features learned by ViTs. Consequently, when generating adversarial examples, ViTs can encode features that resonate more effectively with the internal representations of CNNs, leading to stronger transferability from ViT to CNN architectures.  \n\nWe appreciate this opportunity to clarify this mechanism, and we will emphasize this insight in the revised manuscript. Thank you again for your valuable feedback.  \n\n[1] Do vision transformers see like convolutional neural networks? NeurIPS 2021"}, {"review_id": "k6EZKqIFbb", "reviewer": "Reviewer_hgUT", "summary": "This paper explores the transferability of adversarial examples across architectures. The core insight is that previous work doesn't perturb the common features between different models. Therefore, the authors propose a technique termed Spatial Adversarial Alignment (SAA), which employs an alignment loss and leverages a witness model to fine-tune the surrogate model. Through spatial-aware and adversarial-aware alignment, the surrogate model is trained to concentrate on the common features extracted by the witness model. This facilitates adversarial attacks on these shared features, thereby yielding perturbations that exhibit enhanced transferability. Experiments on 6 CNNs and 4 ViTs show that SAA has state-of-the-art adversarial transferability, especially in cross-model transferability. In addition, the authors also provide empirical guidance for model selection and release code in the supplementary material.", "questions": "See Weaknesses.", "limitations": "Yes, this paper has chaimed its limitations and I think it is enough.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThis paper is of high value because exploring the transferability of adversarial examples across architectures has important security implications for the deployment of practical models. The authors further summarize the empirical guidance for the model selection in SAA, which is beneficial for practice.\nThe proposed aligned-based method is novel, focusing on the common features in space and adversarial space that were ignored in previous work, and can be seamlessly integrated with existing optimization and data augmentation-based methods to achieve stronger performance.\nExtensive experiments in the main body and appendix verify the effectiveness of SAA. The code released in the supplementary material provides reproducibility.\nWeaknesses:\n\nThere is a lack of description on the difference between alignment and direct ensemble model attack. From the perspective of model configuration, the two are very similar.\nWhy does the self-aligned strategy also achieve certain improvements?\nThe differences between SAA and other attacks on ViTs such as SGM, PNA, and TGR were not discussed.", "was_revised": true, "final_justification": "The review meets my requirements. I recommend: 5: Accept!", "rebuttal": "Thank you for your recognition and support of our work. We hope you can be our champion in future discussions.\n\n---\n\n**Weakness 1:** Difference between alignment and direct ensemble model attack.\n\nWe sincerely appreciate the reviewer's insightful question. To clarify the difference between alignment-based methods and direct ensemble model attacks: While both aim to enhance adversarial transferability, their approaches are distinct. Alignment-based methods, such as ours, focus on **improving surrogate models themselves** by leveraging witness models to align structural features, thus generating more transferable adversarial examples. complementary strengths. These two strategies are complementary rather than conflicting.\n\nNotably, our alignment-based method (SAA) can **further enhance ensemble attacks**, as demonstrated in Table 5 of the main body. For example, when combined with an ensemble of Res50+ViT-B, SAA improves the attack success rate (ASR) by **20.8%** compared to the direct ensemble baseline, outperforming even the state-of-the-art MA method (+9.1%). This confirms that Alignment-based refinement of surrogate models provides orthogonal benefits to ensemble strategies, strengthening their transferability in black-box settings. We hope this clarifies their distinct contributions.\n\n---\n\n**Weakness 2:** The improvement of the self-aligned strategy.\n\nWhile the unique spatial features in the self-aligned strategy are relatively limited, the surrogate models can still learn features related to adversarial patterns and category relationships. In terms of adversarial features, the adversarial-aware alignment enables dynamic learning of adversarial characteristics. Since the witness model itself is frozen, the surrogate models can align with adversarial features without deviating significantly from their own inherent properties.\nAdditionally, the surrogate models can still acquire high confidence in the correct categories under soft labels, which also includes the \"ambiguity\" towards similar categories. Such soft labels guide the surrogate models to learn more robust feature representations, thereby enhancing the transferability of the generated adversarial examples.\nWe believe these factors collectively contribute to the improvements brought by the self-aligned strategy. Thank you again for your valuable feedback.\n\n---\n\n\n**Weakness 3:** The differences between SAA and other attacks on ViTs such as SGM, PNA, and TGR.\n\nWe appreciate your attention to the differences between SAA and other attacks on ViTs, such as SGM, PNA, and TGR. As shown in Table 4 of our paper, SAA demonstrates stronger cross-architecture transferability compared to these methods, which preliminarily reflects the superiority of SAA in common settings.​\n\nFurthermore, we have provided a detailed discussion on SGM, PNA, and TGR in Appendix A.2. Specifically, the Pay No Attention (PNA) method extends the Skip Gradient Method (SGM) to ViTs by omitting the gradient computation of attention blocks during back-propagation, thereby enhancing adversarial transferability. Token Gradient Regularization (TGR) operates from the perspective of variance reduction, stabilizing the gradient direction to prevent adversarial examples from getting stuck in poor local optima.​\n\nDistinct from these methods, SAA is the first method to specifically analyze architectural differences across models. It enables the creation of more generalized surrogate models that integrate seamlessly with optimization and data augmentation methods without modifying the forward and backward propagation of the model. This unique design is what ultimately allows SAA to achieve state-of-the-art transferability."}]

Author rebuttal:

Meta-review: This work proposes Spatial Adversarial Alignment (SAA) as a means to boost transferability of adversarial examples across different DNN models and architectures. First, using a witness model and surrogate model, SAA promotes spatial alignment by minimizing divergence of features between the two models. Second, SAA uses a “self-adversarial” strategy that then realigns features to adversarial examples. Intuitively, this promotes attacks that focus on shared features of the models, that is, attack transfer.

Several weaknesses were highlighted, particularly: request for reporting of compute budget (authors clarified this was included), ethical impact of the work (while the attacks improve transferability, I don’t believe the work recognizes new attack surfaces or fundamentally new vulnerabilities; although the authors’ response doesn’t point to lines in the paper addressing this, and I’d urge the authors to include a clearer impact statement on the importance of highlighting security vulnerabilities in AI, and that while the paper improves existing approaches, the vulnerabilities highlighted are not fundamentally new to service providers), a range of clarifying remarks by ZtAM (I believe adequately responded to by the authors, in responses that will be quite feasible to roll in to the final version of the paper), a concern that the final paper will require a range of revisions (though the corresponding reviewer also indicated that the rebuttal and discussion adequately addressed concerns).

Further helpful notes were raised, that I didn’t consider to be significant weaknesses: reliance on cosine similarity (I didn’t view a strong dominating alternative), verification of labels (this isn’t common practice/a requirement for the work to have impact though nice to have), noise introduced by pseudo-labelling (the empirical evaluation in a sense circumvents this risk as the performance is viewed end-to-end), a desire for more qualitative evidence (while this was acknowledged as a limitation by the authors, I don’t think this is a key negative of the work nor community expectation in the area), and discussion of alignment vs direct ensemble model attacks (clarified by authors as different in terms of the approach taken though having similar objectives).

The reviewers observed the following main initial strengths: the integration of two approaches to feature alignment, comprehensive evaluation including 10x models of various kinds and different threat model settings including support for multiple attacks, ablations. While the paper was perhaps marginal in terms of ratings, I believe overall there are good merits for the paper supporting acceptance. There is conceptual and technical novelty (even if not technical depth which is not a requirement of impact) that is well aligned with the task of transferability. The results speak for themselves, it is apparent that the SAA approach has merit and is likely to have good impact to progress attack research where fundamentally new ideas are relatively rare now. It is particularly promising that this can support attacks generally, and overall the approach is quite practical/versatile.
Decision: Accept (poster)