Transferable Adversarial Attacks on SAM and Its Downstream Models
Authors: Song Xia
Wenhan Yang
Yi Yu
Xun Lin
Henghui Ding
LINGYU DUAN
Xudong Jiang
Venue: NeurIPS 2024 poster (2024)
Keywords: Segment anything
Security of large fundation model
Transfer-based adversarial attack
Fine-tuning
Primary area: safety_in_machine_learning

Abstract: The utilization of large foundational models has a dilemma: while fine-tuning downstream tasks from them holds promise for making use of the well-generalized knowledge in practical applications, their open accessibility also poses threats of adverse usage.
This paper, for the first time, explores the feasibility of adversarial attacking various downstream models fine-tuned from the segment anything model (SAM), by solely utilizing the information from the open-sourced SAM. 
In contrast to prevailing transfer-based adversarial attacks, we demonstrate the existence of adversarial dangers even without accessing the downstream task and dataset to train a similar surrogate model.
To enhance the effectiveness of the adversarial attack towards models fine-tuned on unknown datasets, we propose a universal meta-initialization (UMI) algorithm to extract the intrinsic vulnerability inherent in the foundation model, which is then utilized as the prior knowledge to guide the generation of adversarial perturbations.
Moreover, by formulating the gradient difference in the attacking process between the open-sourced SAM and its fine-tuned downstream models, we theoretically demonstrate that a deviation occurs in the adversarial update direction by directly maximizing the distance of encoded feature embeddings in the open-sourced SAM.
Consequently, we propose a gradient robust loss that simulates the associated uncertainty with gradient-based noise augmentation to enhance the robustness of generated adversarial examples (AEs) towards this deviation, thus improving the transferability.
Extensive experiments demonstrate the effectiveness of the proposed universal meta-initialized and gradient robust adversarial attack (UMI-GRAT) toward SAMs and their downstream models.
Code is available at https://github.com/xiasong0501/GRAT.

Transferable Adversarial Attacks on SAM and Its Downstream Models
Song Xia', Wenhan Yang?, Yi Yu', Xun Lin Henghui Ding", Lingyu Duan Xudong Jiang"
Nanyang Technological University, "Pengcheng Laboratory, BBihhnn University, "Fudan University, "ekknng University {xias0002, fxias0002.yyy10001, yuy10010 exdjiang) @ntu. edu. sg, yangwh@pcl ac. en, linxun@buaa, edu. en, hhdingefudan, edu. cn, lingyupku@edu.cm en
Abstract
The utilization of large foundational models has a dilemma: while fine-tuning downstream tasks from them holds promise for making use of the well-generalized knowledge in practical applications, their open accessibility also poses threats of adverse usage. This paper, for the first time, explores the feasibility of adversarial attacking various downstream models fine-tuned from the segment anything model (SAM), by solely utilizing the information from the to prevailing transfer-based adversarial attacks, we demonstrate the existence of adversarial dangers even without accessing the downstream task and dataset train similar surrogate model. To enhance the effectiveness of the adversarial attack towards models fine-tuned on unknown datasets, we propose a universal inttainiiiialzztionn (UMI) algorithm to extract the intrinsic vulnerability inherent in the foundation model, which then utilized as the prior knowledge to guide the generation of adversarial perturbations. Moreover, by formulating the gradient dif- ference the attacking process between the open-sourced SAM and its tine-tuned downstream models, we theoretically demonstrate that deviation occurs in the adversarial update direction by directly maximizing the distance of encoded feature embeddings in the open-sourced SAM. Consequently, propose gradient robust loss that simulates the associated uncertainty with gradient-basee noise augmenta- tion enhance the robustness of generated adversarial examples (AEs) towards this deviation, thus improving the transferability. Extensive experiments demonstrate the effectiveness of the proposed universal meta-ini vntta-iitiialized and gradient robust ad adversarial attack (UMI-GRAT) toward SAMs and their downstream models. Code is available https//ggghbbbboooomauamm//////7
SAM. In contrast
Introduction
Large foundation models that are trained a broad scale of data have gained massive success in
various applications [6), such as vision-language chatbot [ text-image generation [42 44 46),
gongg--roundedde text generation 2 (1). and anything segmentation [26]. The segment anything model (SAM) (26), trained on vast amounts of data from the SA-IB dataset, is capable of handling diverse and complex visual tasks. The open accessibility of SAM makes it promising foundation model, serving as the starting point for finettuning analytics models certain domains and downstream applications, é.g., medical segmentation [6]
(54 [39] 3D object segmentation (9, camouflaged object segmentation [II, overhead image segmentation [45], and high-quality segmentation (25). However, many studies 5 [8 57 38 27 55 588 66 65 59 48 have highlighted the secure issues of deep
learning models towards adversarial attacks. By corrupting the clean input with a finely crafted and
"Corresponinng author (exdjiang @ @ntu.edu.sg)
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Open-sourced SAM based MUI-GRAT
Task-specific SAMs after fine-tuning
Shadow
Shacow Datasets
SAM
Camouffage
Camooulagge Datasels
SAM
+0.04
Medical Dalasets
Medical
SAM
Clean inputs
Adversarial noise
Prediction
GT mask
Figure 1: An illustration of UMI-GRAT towards SAM and its downstream tasks. The UMI-GRAT can mislead various downstream models by solely utilizing information from the open-sourced SAM.
nearly imperceptible adversarial perturbation, the attacker can mislead the well-trained model at high success rate, with limited information available (e.g.. the surrogate model or limited queries). Consequently, significant concerns arise regarding fine-tuning open-sourced models, as it inevitably leaks critical information on downstream models, increasing their vulnerability adversarial attacks.
Existing adversarial attacks can be roughly categorized into white- box attacks 3 18] and black- boox
attacks [56] 23 based on whether the attacker fully access the victim model. The pre-requisite
of fully accessing the victim model complicates the practical deployment of white-box attacks. Conversely, transfer-based black-box attacks that require less information pose substantial threat to weal-world applications. The prevalent transfer-based black-box ad adversarial approaches [8] [12
17
[5 60) typically suppose strong prior knowledge the victim model's 's task and training data, such as 1000-class classification task in ImageNet, thereby facilitating the training of similar surrogate model to generate potent adversarial examples (AEs). However, few studies 64 (3 consider more practical and challenging scenario wherein the attacker is unaware of the victim model's tasks and the associated training data, due stringent privacy and security protection policies (e.g., datasets containing medical or human facial information). Moreover, the increasing size of large foundation models significantly amplifies the costs of training effective task-specific surrogate models. Thus, more general and practical security concern is to explore the capability of attackers to mount adversarial attacks on any victim model even without the need of accessing to downstream tasks-specific datasets to train closely aligned surrogate model.
22
32
34
36,
Given the practical security concerns of utilizing large foundation models, this paper investigates the potential risks associated with fine-tuning the open-sourced SAM on private and encrypted dataset. We introduce a strong transfer-based adversarial attack called universal veta-initialized and gradient robust adversarial attack (UMI-GRAT), which effectively mislead SAMs and their various fine-tuned models without accessing the downstream task and training dataset, illustrated in Figure
The contributions of our paper are summarized as follows:
We begin investigation into attacking various SAMs' downstream models by solely utilizing the information from the open-sourced SAMs. We provide the theoretical insights and build the experimental setting and benchmark, aiming to serve as a preliminary exploration for future research.
practical while challenging adversarial attack problem:
We propose an offline universal meta-initialization (UMI) algorithm extract the intrinsic vulnerability inherent in the foundation model, which is utilized prior knowledge to enhance the effectiveness of adversarial attacks through vettainitialization.
We theoretically formulate that, when using the open-sourced SAM the surrogate model, a deviation occurs inevitably from the optimal direction of updating the adversary. Corre- spondingly, propose a gradient robust loss mitigate this deviation.
Extensive experiments demonstrate the effectiveness of the proposed UMI-GRAT toward SAMs and downstream models. Moreover, UMI-GRAT can serve as plug-and-play strategy to significantly improve current state-of-the-e cansfer-based adversarial attacks.

---

2
Related Work
The adversarial attacks aim to mislead the target (or victim) model by adding a small adversarial perturbation in the clean input. Existing black-box attacks can be broadly categorized into query- basedd and transfer- based adversarial attacks.
2.1 Query-based black box attacks
The query-based attacks consider the scemarios where the attacker does not have enough information to train a satisfying surrogate model, thus generating adversarial examples by interacting and analyzing the outputs from the victim model. This kind of attack can be divided into score- based query attacks (SQAs) 12 24 47 that update the AEs by observing the change of the model's prediction (e.g., the logits or softmax probability) and decision-basee query attacks (DQAs) [0] that only rely on the model's 's top-1 prediction to update the AEs. However, this black-box search is naturally the NP-hard problem, and solving the optimal update strategy is non-differentiable. This makes query-baser attack requires thousands of interactions with the victim model, making query-based attacks characterized by low throughput, high latency, and marked conspicuousness to attack real-world deployed systems.
2.2 Transfer-based black box attacks
The transfer-based adversarial attacks generate the AEs to mislead the victim model based on a similar surrogate model. Existing work mainly focuses improving the transferability of AEs, which can be categorized into four groups: input-angmentation- based attacks [8 [6 52 that enhances the effectiveness of generated AEs by augmenting the clean input (e.g.. using crop rotation),
optimization-basse attacks 11 35
51 62) that utilizes a better optimization strategy to guide the
update of ABs, model modification-bases attacks (53 or ensemble-based attacks 19
43
37
that enhances the AEs by utilizing a more powerful surrogate model, and feature- based attacks 32 34
that attack the extracted feature in the intermediate layer. However, most of the work makes a strong assumption that the surrogate and victim models are optimized for the same task with identical data distribution, for example, both surrogate and victim models are optimized on the ImageNet dataset to complete the classification task. In real-world deployed systems, due to privacy and security concerns, attackers typically cannot access the training data (e.g., datasets containing private information) or obtain the optimization objectives of the victim model, making training a similar surrogate model exceedingly difficult and unfeasible.
To address the challenge of deploying wansfer-based black-box attacks without knowing the victim model's i task and training dataset, this paper investigates the feasibility of attacking any victim model optimized for unknown tasks and distributions that significantly differ from the open-sourced surrogate model. We provide both theoretical and analytical evidence demonstrating that our proposed method can enhance the transferability and effectiveness of the generated AEs. Moreover, the proposed MUI-GRAT can serve plug-and-play adversarial generation strategy to enhance most existing transfer-based adversarial attacks for this challenging task.
3 Preliminaries
3.1 Adversarial attacks
Let f be any deep learning model and L be the loss function (e.g., the cross-entropy loss) that evaluates the quality of the model's prediction. Let (x(2) {x': ||> =||p S c) be an fo-norm
ball centered at the input where is pre-defined perturbation bound. For each input æ, the untargeted adversarial attacks aim to find an adversarial perturbation by solving:
max (+))
Lf (z) . (x 0))
(1)
An effective solution to Equation 0 is iteratively updating the adversarial perturbation 8 based on the gradient of the loss function, for example, the iterative fast sign gradient method (I-FGSM) 28). which iteratively updates o by:
6 clips. (ō, + a sign Co (a 5.)1))
(2)
3

---

where V calculates the gradient and sign returns the sign (i.e.-1 or +1). a is pre-defined step size update the adversarial perturbation. clip constrains the magnitude of the perturbation by projecting 6 into the boundary of the E--norm ball B,
3.2 Segment anything model
The SAM consists of three parts: an image encoder for.. a lightweight prompt encoder fer.. and lightweight mask decoder [4m.. SAM gives the mask prediction based on the image input æ and prompt input p. which is expressed as:
-) = farm (foc.. (x) S0x (p)).
(3)
where fe.... is the image encoder that provides the fundamental understanding by converting natural images into feature embeddings and fø.. extracts prompt embeddings. fø.. the mask decoder that gives the mask prediction by fusing the information from both feature and prompt embeddings.
4 Methodology
4.1 Problem formulation
Let fæ, denote the foundation model trained on general dataset D, and fö. denote the victim model fine-tuned on any downstream dataset D,, the parameters those two models typically satisfy that:
Φ arg min
E dr (..))~
[C (So. (a), v)] Φr arg min d-
E Mr
[CT (Jor (.+). Φr
igétio Φ. (4)
Definition 1 (Transferable ad vdrrsaaii attack via open-sourced SAM). For any SAM's downstream model for and the clean input Iy, without any further information on the downstream task and dataset, the attacker aims find the adversarial perturbation ö, such that:
max L.r (fe- ; (x, + +8)) s.t. {6, (f.. Private (D+)): 6,EB,
(5)
where AT is the adversarial attack strategy and fab, is the open-sourced SAM. A solution to that is fine-tuning an optimal surrogate model før that closely aligns with the victim model. However, this approach becomes extremely challenging, when the downstream dataset is inaccessible the attacker. Alternatively, an effective solution is to design an optimal attack strategy AT" such that:
AT
max AT
E [Cr (for (xx), fo. (xx, + AT (fo. (0,,,,,))) }3]
(6)
(3..,, D. D.D,
Notably, fo. and for and L,, leading to a significant input-output mapping gap and gradient disparity, such as Cosine Cosine_simillee similarity( V fe. (z,). fee (x+)) < 1. This misalignment critically undermines the effectiveness of current gradient- based adversarial attack strategies.
optimized two distinctive distributions D and D, with losses
Further analysis attacking SAMs. The standard operation to deploy SAM on downstream tasks 7 involves fine-tuning the image encoder f inherit some well-generalized knowledge. Concurrently, the lightweight prompt encoder för, and the mask decoder fø... are trained from scratch to better accommodate the task. Considering the pivotal importance of feature embeddings and the substantial variation caused by full retraining, intuitive approach to generate effective adversarial perturbation 8, is utilizing the common information in fer.. to achieve:
max L (Jor... (u+) Sor.. (æ, + 8.1) s.t. 8, AT (f ). 6,88.
(7)
where & denotes the updated parameters for the downstream model's image encoder after fine- tuning. Unless otherwise specified, we denote Φim as Φ in our subsequent content and take the general SAM image encoder fo as the surrogate model fø.. We aim to generate the transfer- based AES to attack any fine-tuned image encoder fø, on task т, thereby misleading the entire prediction.
4.2 Extract the intrinsic vulnerability via universal meta initialization
Considering the great variation brought by fine-tuning the model on new task we aim to extract the intrinsic vulnerability of the foundational model that remains invariant after fine-tuning, Subsequently, this extracted vulncrability is leveraged as prior knowledge to initialize and enhance the adversarial

---

attack AT". Inspired by the universal adversarial perturbation [40] that maintains effectiveness across various inputs, we propose the universal meta initialization (UMI) algorithm. which optimizes the initialization of adversarial perturbation to ensure both effectiveness and fast adaptability by
meta-learning [41 [6]. We define the universal and meta-initialized perturbation d as follows.
Definition 2 (Universal and meta-initialized perturbation 8). Given the foundation model Ja and its fine-tuned models So. on downstream tasks т, the universal and meta-initialized perturbation that extracts the intrinsic vulnerability ensures both effectiveness and fast adaptability, which are:
1. Effectiveness (universal adversarial perturbation): 8 extracts the intrinsic vulnerability in the foundation model, which can mislead the fo successfully on most natural inputs æ, which is:
max 8EB, 2- D
E
[ (a), fo (x + 0]) > >
(8)
where X is a pre-defined threshold for one successful attack, and 1 {-} is the indicator function that returns 1 if the inside condition is satisfied, else O.
2. Fast adaptability ineta-iniitaaization:: for any downstream task with the corresponding private downstream dataset D, and model fø., the attackers can maximize the loss on downstream model fø. by updating the initialization 8 via the surrogate model fo in t steps, which is:
max TE BEB, 2. ~D
[c. (fo. (xx)). (x,, + U² +U*(())) (61))]
(9)
Ut is the operation to update 8 for steps based on input Ir, which is defined as:
U (8) clipa.
8 Σ Adj
(10)
where 16 sign (VL (Jo (2x,) So (I., + US (5))) if we attack the surrogate model fo and update the adversarial perturbation based the first-order gradient.
Generally, Equation 8 aims to extract the intrinsic vulnerability inherent in the model, which remains effective towards the input variation. Equation guarantees that utilizing the perturbation 8 as the initialization can rapidly threaten strong adversarial attacks for any downstream model. However, in Equation 9 D, and fe, are unknown if the attacker is precluded from the downstream dataset. approximated solution to that involves using the dataset D that covers the distribution of most natural inputs and a general model f that can approximately represent the expectation of fe., which is:
max R [ (Ja (a). fo (æ + U (8)))] BEB, 2m
(11)
To optimize the above two objectives simultaneously, our learning aims to move towards the direction that maximizes the inner product of the gradients computed on both objectives. We utilize first-order meta-learning algorithm called Reptile (11,, which defines the noise ó update in each round as:
8 = 8 +
(5.. -))
(12)
where 77 is the update step size, and 6,,, Ut (8) is the updated perturbation objective μ, after optimizing iterations. Here we set n 2. corresponding to the two objectives in Equation 8 and I1
For μn that aims to optimize Equation 11 we set t 5 and U (d) the same as U defined in
Equation 10 For #2 that aims to optimize Equation g we set Ut. p2 (8) as:
Up2 (6) < arg min ||,e, s.t.C (fo (x) 5 A0) A. A6 Σ A5, 6428
(13)
Equation| 13 aims to find a minimal update Ad nearby to mislead the model fs. This can be achieved by using enough iterations t and small but gradually increased norm-ball boundary While finding an effective UMI requires a substantial number of inputs and iterations, this process can be conducted fully offline, thus not hindering real-time adversarial attacks.
4.3 Enhance the transferability via gradient robust loss
Besides the utilization of intrinsic weakness inherent in the foundation model to enhance the ad- versarial attack AT*, another method involves generating the adversarial perturbation that sustains robustness against the deviation arising from updates through surrogate model that exhibits signifi- cant gradient disparity compared to the fine-tuned downstream model.
5

---

Perturbatton
Feulark embeokngg
Fooward Process Gradient Reckward
Difforent Lagenng
SUN Imsage Encodor
SAM irugee
Pertarbebo pplii
UM
Tatislia
Adrersntal Pertrzazion
degiion
Upskit
Linvenol
*
Nera Ioloaliattoo
Figure 2: The data flow of UMI-GRAT, consisting of an offline learning process of UMI and real-time gradient robust adversarial attack.
Let us first assume that the surrogate model fe consists of m sequentially connected modules, denoted as f.fz.... f The outputs of those modules are denoted {y ,..., y²",, with y¹ fa () For the victim model fø. with updated parameter Aø,, the modules are denoted as (S'+ +sp }. The output y- of each module with the input a, is denoted as:
y = feit (y--1)= = fåz (y--1) + hiss (y--))
(14)
where y = x. and Aes is hypothetical function that characterizes the update brought by A
Proposition (Deviation in updating adversarial perturbation). Let f be the victim model fine- tuned on any unknown task T, the deviation in the direction of updating the adversarial perturbation by maximizing predefined loss f. in the surrogate model Jop can be formulated as:
A - 2b. = vL(U"))
II
(V (v. + Vh (v. 1))
V (u) (v)
(15)
In Equation 15 28+ + VC(F)) II
( (yt Vi (v= the update of adversarial
perturbation white-box attack the victim model and A6. + VC("") II Vf. (v--1) the update
of adversarial perturbation by maximizing the pre-defined on feature embeddings of the surrogate model. Proposition esttablishe by simultaneously considering the white-box scenarios for both surrogate and victim models to derive the gradient using the chain rule. It claims that hae; leads to great deviation in updating the adversarial perturbation ô, towards the optimal solution in attacking victim model if directly maximizing the feature embedding distance of the surrogate model, thus degrading the effectiveness of the generated AEs.
Mitigate the deviation caused by gradient disparity. enhance effectiveness of the generated adversarial perturbation ó, under the hypothetical update Vhag.. we propose gradient robust loss LGR, that aims to mitigate the deviation Equation b5]by gradient-based noise augmentation. Denote (e; μu, o² I) the isotropic Gaussian noise with mean μ and variance o², which has the same dimension as Vhaø. The robust update of adversarial perturbation Ad; on the surrogate model based on the noised augmented gradient is:
A + VL
(VI (Di + V (y. "). 1m1
II
(16)
By ignoring higher-order uncertain terms Equation 16 we can simplify it as:
A8; < vl(TT)
V (v:
IV (u--1)
(17)
Following the adversarial perturbation update guidance in Equation the corresponding gradient robust loss LGR is defined as:
m-
81 (F3. (g2tdd) S/s. (wtt)
Lor
fgm (v (ofd fm (V.")
Σ
(18)
m-1
1=1

---

Algorithm 1 Generating adversarial examples by UMI-GRAT
1: Input: open-sourced foundation model fe. natural image dataset D, task-specific image ær number of meta-iterations Tm. universal step size
attack iterations Ta, attack step size a.
2: # Off-line learning UMI
12:
Real-time attack by UMI-GRAT
3: function UNI_ UNI_METE ____A_MTTANIfF INIiA__ D, Tm. 7)
13:
function GR_ ET_AATAAK( Z,,, T, 7,9, 6)
4: Initialize 6, 6 0/z with
adapt the universal perturbation ò
5: for t < 1 to Tm do
15:
Sndp = aadp -sign (V Ladp (So (x-) 9))
6: assign 8,n 8,12 6
16:
Initialize 8 clipn, [ō Sadpl
7: for each in D do
17:
threaten GRAT
8: update 8,1 by Un (6 in Equation 10
18:
for to To do
9:
update 6,2 by H2 (6
in Equation 13
19:
g. + VoR (fo (x, + f (a+))
20:
8 = cips. 16; + a (g.)
10: update 5 = +1 Σ (6p. 6)
21:
x = x. + 8
11:
return 5
22:
return x
where youd" is the extracted adversarial feature by layer i and I|-11, is a predefined -orm- measure that is decided by the (e.g., p = 1 for L1 loss).
16: GGFGSN-GR 18-FGSM
Consine similartt
0.45
LPD
opt
Discussion with the intermediate-level attacks. The intermediate-level attacks (ILAs) 34 22 [2) also aim to max- imize the dissimilarity of feature embeddings between the clean and adversarial inputs. However, the main concern in ILAs is how to find a directional vector v to guide the update direction of fø (z) - f (z"d"), thus assuring that this feature- wised dissimilarity can maximally mislead the final prediction. Different from that, our LGR considers the problem one step further: given an optimal directional vector v, how to generate the adversarial perturbation that is roust towards the potential gradient variation in the victim model, thus maximally misleading the victim model along the direction of v. Our core idea is hence in parallel with ILAs and can be well combined with them to enhance the attack's effectiveness. We experimentally analyze and visualize the cosine similarity of the generated perturbations on CT- scan images by white-box attacking open-sourced SAM and medical SAM using MI-FGSM (I3), ILPD [34], and our gradient robust attack in Figure 3 illustrating that the proposed LGR effectively reduce the deviation caused by gradient variation and achieves a much transfcrability.
0.35
Figure 3: The cosine similarity of white-box generated perturbations on surrogate and victim models.
Implementation of the proposed MUI-GRAT
The detailed implementation of our proposed MUI-GRAT is illustrated in Algorithm aan Figure 2 Our UMI-GRAT method consists of two stages. The initial stage involves the offline learning of a universal meta-initialization (UMI) which aims to find the intrinsic vulnerability inherent in the foundation model. In the subsequent stage, we utilize the learned UMI as the prior knowledge to enhance the gradient-variation robust adversarial attack.
We utilize the image encoder from Vit-H-based SAM as the general foundation model fo for generating the UMI. The natural image dataset D consists of a total of 20,000 images, with 10,000 from ImageNet and 10,000 from the SA-1B dataset. We set the meta iterations Tm 7 and the universal step size as 1. The function Uni Uni_Meta Ini returns the learned UMI ō that can be used to enhance the generation of subsequent input-specific adversarial perturbation.
In Attack, we first adapt the calculated UMI g with the task-specific image x, by one-step update using FGSM [18] with the step size Cadp = 4. Assume that represents the mean of the feature embedding of natural images calculated by fØ, our Ladp is defined as:
Ladp (f
ōl1
(19)
Equation 19 aims to minimize the domain difference between ur,, and the natural images by specific generated perturbation. The UMI is then added by dadp and utilized to initialize 65. We set Ta to 10, and update the adversarial perturbation 8, by maximizing the gradient robust loss CGR.
7

---

Table Comparison results of transfer-base adversarial attacks on different models. The surrogate
model is the open-sourced SAM.
Model
Medical SAM 61
Shadow-SAM
11
Camoullnged-SAM
11
Dataset
CT-Scans
ISTD
CODIOK
CAMO
CHAME
Metries
mDSCI
mHD
BER
S. L
MAE1
Sa
MAE 1
Sa L
MAE
Without attacks
81.88
20.64
1.43
0.883
0.025
0.847
0.070
0.896
0.033
MI-FGSM [3]
40.83
64.42
4.31
0.372
0.214
0.331
0.286
0.352
0.250
DML-FGSM 56
34.51
74.20
4.39
0.435
0.134
0.395
0.210
0.416
0,164
PGN
T7
43.15
58.03
5.16
0.368
0.230
0.336
0.318
0,340
0.275
BSR
5.20
012226
0.401
ILPD 34
33,65
65,98
4.40
0.366
0.245
0,310
0.316
0.327
0.287
MUI-GRAT
5.22
111.87
12.46
0,360
0.248
0.329
0.308
0.332
0.293
MU-GRAAT-DMI-FON
5,28
114.68
5.48
0.409
0.198
0.417
0.267
0.406
0.228
MUI-GRAT+PGN
9.62
115.87
33,98
0.358
0.262
0.353
0.306
0.332
0.296
MUI-GRAT+
3.61
105.31
7.00
0.385
0.219
0.398
0.277
0.387
0.245
MUI-GRAT-ILPD
3.52
121.89
15.56
0.349
0.263
0.321
0.311
0.315
0.317
Experimental Results
6.1 Experiment setup
Evaluation details: we conduct experiments on SAMs' downstream models including, medical image
segmentation SAM [61, shadow segmentation SAM [ and camouflaged object segmentation
SAM [11]. The datasets include: the synapse multi-organ segmentation dataset [29 that contains
3779 abdominal CT scans with 13 types of organs annotated, the ISTD dataset [49 that contains 1 1870 image triplets of shadow images, the COD10K dataset [14] that contains 5066 camouflaged object images, the CHAMELEON dataset that contains 76 camouflaged images, and the CAMO dataset [30,, that contains 1500 camouflaged object images. We report the mean dice similarity score (mDSC) and mean Hausdorff distance (mHD) for evaluating medical segmentation, mean absolute error (MAE) and structural similarity (S.) for camouflaged object segmentation, and the bit error rate (BER) for shadow segmentation. In medical SAM, the image encoder is based on SAM-Vit-B and fine-tuned with LoRA (1 In shadow segmentation and camouflaged object segmentation SAM, the image encoders are based on SAM--VitHH and fine-tuned with the adapter [20] The decoders in those models are all fully retrained.
Compared methods:
We mainly compare and evaluate our method with current transfer-based
adversarial attacks including gradient-based attacks called MI-FGSM [13 and PGN [17, input- augmentation based attacks called DMI-FGSM (56) and BSR [50,, and intermediate-level feature
based attack called ILPD [34].
Implementation details: we use the MI-FGSM [3] as our basic attack method. For all methods reported, we set the attack update iterations T, as 10, with the I bound 10 and the step size a = 2. For our UMI, we set the meta iterations Tm, = 7, universal step size 17 = For PGN and BSR, we set the number of examples as 8 for efficiency.
6.2 Main results
We report our main results in Table 1 The first row of the data presents the model performance with clean inputs. The second part of the data shows the model performance under different adversarial attacks, where the data with the strongest attack is bold. The results demonstrate that the adversarial examples generated by our proposed MUI-GRAT are more effective and generalizable than others. consistently posing significant adversarial threats across various downstream models. In medical seg- mentation and shadow segmentation tasks that share great difference with the natural segmentation tasks, our proposed AUT-GRAT greatly surpasses others (e.g., MUI-GRAT reduces the mDSC from 81.88 5.22 while the previous best is 25.73.). This demonstrates the exceptional effectiveness our proposed MUI-GRAT when the attacker lacks information about the victim model, thereby generating ABS using a surrogate model distinct from the victim model. In the camouflaged object segmentation task where the data closely resembles natural images, all methods exhibit strong transferability. Our MUI-GRAT achieves the best performance on CODIOK and CHAME datasets and performs comparably to the SOTA method the CAMO dataset.

---

The distance of festure embedding under attacks
distance feature embadding under attacks
distance feature ombedding under attacks
1.000
Avg diebance gMo
In AurT-ga
0.039
Agg distmce gap
0.009
Aug stance
Feature distancc
3.055
2.00
rogule
ReStL discarr
5.000
4.05
2.005
Featura
3.00
co15
2.020
2.05
2.025
2000
3.005
Lio
Image index
Image index
Image index
(a) Medical SAM
(b) Shadow SAM
(c) Camouflaged SAM
Figure 4: The la distance of feature embedding from clean inputs and adversarial examples. The small distance gap between the surrogate and victim models indicates better transferability.
In the last part of Table I we analyze the performance of our proposed MUI-GRAT when combined with other SOTA transfer-based attacks. Notably, all methods achieve an overall performance gain after combining with ours. Though a slight performance drop occurs when attacking the CAMO dataset, combining the MUI-GRAT brings great performance gain on attacking other tasks. This demonstrates the versatility of our MUI-GRAT, which can be seamlessly applied in a plug-and-pin manner to bolster existing transfer-based attacks. We report the experiment results for attacking open-sourced SAMs in Appendix A.2 and analyze the real-time attack efficiency for each method in Appendix A.3
6.3 Analysis of the transferability
The transferability property across diverse models ensures the effectiveness of the adversarial ex- ample to mislead an unknown victim model. As discussed in Section 4.1, an intuitive objective for attacking SAM's downstream models is to maximize the dissimilarity of feature embedding extracted from the clean input x and adversarial input ad Based on this, to numerically evaluate the improvement of transferability brought by MUI-GRAT, we present the l2 distance of clean and adversarial feature embeddings attacked by MI-FGSM and ours and then analyze the distance gap between the surrogate and victim models. We propose that a viable transferable attack methodology should induce a substantial feature distance Af f in the victim model, while simultaneously ensuring minimal performance degradation c during the transition from the surrogate to the victim model.
We show this comparison result in Figure 4 where we randomly pick subset of inputs and show the distance of feature embedding for each clean-an aaveessravaaa input pair. The average distance gap between the surrogate and victim models indicates the overall transferability of the attack method. In the medical and shadow segmentation SAM, where the data and task are distinct from the original SAM, we find great performance drop for MI-FGSM when transferred from the surrogate model to the victim model. Though the adversarial examples generated by MI-FGSM induce a large feature distance in the surrogate model, their effect on the victim model relatively minor. Conversely, the adversaries generated by MUI-GRAT maintain much better transferability, suffering from a small performance drop when transferred from the surrogate model to the victim model. In the camouflage object segmentation task, where the data are natural images and the segmentation objective is similar to the original SAM, both attack algorithms show good transferability (nearly no performance drop when transferred from the surrogate model to the victim models). Our MUI-GRAT shows better transferability with nearly no performance drop.
6.4 Ablation study
In this section, we explore the contribution of the proposed MUI and GRAT by integrating them with MI-FGSM and PGN attacks. The ablation results are shown in Table 2 In scenarios where the task and dataset distributions of surrogate and victim models differ markedly (e.g.. medical image segmentation and natural image segmentation), we observe that the GR loss significantly enhances effectiveness. Meanwhile, across all scenarios, the proposed MUI consistently contributes to enhancing the adversarial attacks. Particularly in camouflaged object segmentation when the surrogate and victim models exhibit close similarities, the MUI yields substantial benefits.

---

Table 2: The performance methods combined by MUI and GR.
Medical SAM [6
Shadow-SAM [
CA III
Basic strategy
MUI
GR
mDSC|
mHD 1
BER f
S.
MAE
x 40.83
64.42
4.31
37.17
21.41
MI-FGSM 13
x 37.54
70.13
4.72
36.11
24.74
6.34
100.52
8.07
36.90
21.78
x
X
43.15
58.03
5.16
36.84
23.04
PGN Fi
/
X
41.13
66.
6.46
35.23
27.68
X
15.51
93.98
10.36
36.25
24.10
This observation aligns with our analysis Section By assuming hypothetical update hav, in the victim model, the proposed gradient robust loss greatly enhances the effectiveness of the generated AEs towards the gradient variation, thus benefiting more for medical and shadow segmentation tasks. Moreover, the MUI aims to find the intrinsic vulnerability inherent in the basic foundation model through broad general dataset, which then provided as the prior knowledge for generating more effective ad adversary. Therefore, in scenarios where the victim model inherits substantial information from the surrogate model, this prior knowledge becomes increasingly reliable and effective.
Conclusion
The security of utilizing large foundation models is a critical issue for deploying them in world applications. This paper, for the first time, considers a more challenging and practical attack scenario where the attacker executes potent adversarial attack on 1AM-based downstream models without prior knowledge of the task and data distribution. To achieve that, we propose a universal meta- initialization (UMI) algorithm to uncover the intrinsic vulnerabilities inherent in the foundation model. Moreover, by theoretically formulating the adversarial update deviation during the attacking process between the open-sourced SAM and fine-tuned downstream models, we propose gradient robust loss that simulates the corresponding uncertainty with gradient-based noise augmentation and analytically demonstrates that the proposed method effectively enhances the transfcrability. Extensive experiments validate the effectiveness of the proposed UMI-GRAT toward SAM and its downstream tasks, highlighting the vulnerabilities and potential security risks of the direct utilization and fine-tuning of open-soeneee sourcedd large foundation models.
Acknowledgment
This research is supported by the National Research Foundation, Singapore, and Infocomm Me- dia Development Authority under its Trust Tech Funding Initiative, and by donation from the Ng Teng Fong Charitable Foundation. Any opinions, findings and conclusions recommenda- tions expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore, and Infocomm Media Development Authority. This research is partly supported by the Program of Beijing Municipal Science and Technology Commission Foun- dation (No.Z241 100003524010), and partly supported by Guangdong Basic and Applied Basic Research Foundation 12024A151500045 1515000444). The research was carried out at the ROSE Lab, Nanyang Technological University, Singapore.
References
[1] Achiam, Adler, S. Agarwal, L. Ahmad, Akkaya, Aleman, Almeida, J. Altenschmidr, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08574.22000 2023.
[2] J.BB. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lene, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Proc. Annual Conf. Neural Information Processing Systems, 2022.
[3] A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok. Synthesizing robust adversarial examples. In Proc. Int't l Conf. Machine Learning, 2018.
10

---

[4] P. Benz, C. Zhang, and I. S. Kweon. Batch normalization increases adversarial vulnerability and decreases adversarial transferability: non-robust feature perspective. In Proc. IEEE Int' Conf. Computer Vision, 2021.
[5] B. Biggio, I. Corona, D. Maiorea, B. Nelson, N. Smdić Laskov, G. Giacinto, and F. Roli. Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery databases, 2013.
[6] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 2021.
[7] W. Brendel, J. Rauber, and M. Bethge. Decision-based adversarial attacks: Reliable attacks against black-box machine learning models. In Proc. Int'l Conf. Learning Representations, 2018.
[8] J. Byun, S. Cho, M.-J. Kwon, H.-S. Kim, and C. Kim. Improving the transferability of targeted adversarial examples through object-based diverse input. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2022.
[9] J. Cen, Z. Zhou, J. Fang, W. Shen, L. Xie, D. Jiang, X. Zhang, Q. Tian, et al. Segment anything in 3d with nerfs. Proc. Annual Conf. Neural Information Processing Systems, 2023.
[10] J. Chen, M. I. Jordan, and M. J. Wainwright. Hopskipjumpattack: A query-efficient decision- based attack. In IEEE symposium on security and privacy, 2020.
[11] T. Chen, L. Zhu, C. Deng, R. Cao, Y. Wang. S. Zhang, Z. Li, L. Sun, Y. Zang, and P. Mao. Sam-adapter: Adapting segment anything in underperformed scenes. In Proc. IEEE Ini' Conf. Computer Vision, 2023.
[12] Croce, M. Andriushchenko, N. D. Singh, N. Flammarion, and M. Hein. Sparse-rs: a versatile framework for query-efficient sparse black-box adversarial attacks. In Proc. AAAI Conf. on Artificial Intelligence, pages 6437-6445, 2022.
[13] Y. Dong, F. Liao, Pang, H. Su, Zhu, X. Hu, and Li. Boosting adversarial attacks with momentum. In Pruc. IEEE Int'l Conf. Computer Vision and Puttern Recognition, pages 9185-9193, 2018.
[14] D.-P. Fan, G.-P. Ji, G. Sun, M.-M. Cheng, Shen, and L. Shao. Camouflaged object detection. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, pages 2777-2787, 2020.
[15] Z. Fang, R. Wang, T. Huang, and L. Jing. Strong transferable adversarial attacks via ensembled asymptotically normal distribution learning. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2024.
[16] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proc. Int'l Conf. Machine Learning, 2017.
[17] Z. Ge, H. Liu, W. Xiaosen, F. Shang, and Y. Liu. Boosting adversarial transferability by achieving flat local maxima. Proc. Annual Conf. Neural Information Processing Systems, 2023.
[18] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 1412.61111.662. 2014.
[19] M. Gubri, M. Cordy, M. Papadakis, Y. L. Traon, and K. Sen. Lgv: Boosting adversarial example transferability from large geometric vicinity. In Proc. IEEE European Conf. Computer Vision, 2022.
[20] N. Houlsby, A. Giurgiu, Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo. M. At- tariyan, and S. Gelly. Parameter-efficient transfer learning for nlp. In Proc. Int'l Conf. Machine Learning, 2019.
[21] F. J. Hu, P. Wallis, Z. Allen-Zhu, Li, S. Wang, L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models. In Proc. Int'l Conf. Learning Representations, 2021.
11

---

[22] Q. Huang. I. Katsman, H. He. Z. Gu, S. Belongie, and S.-N. Lim. Enhancing adversarial example transferability with an intermediate level attack. In Proc. IEEE Int'l Conf. Computer Vision, 2019.
[23] A. Dyyss, L. Engstrom, A. Athalye, and Lin. Black-box adversarial attacks with limited queries and information. In Proc. Int'l Conf. Machine Learning, 2018.
[24] A. Iyas, L. Engstrom, and A. Madry. Prior convictions: Black-box adversarial attacks with bandits and priors. In Proc. Inl'l Conf. Learning Representations, 2019.
[25] L. Ke, M. Ye, M. Danelljan, Y.-W. Tai, C.-K. Tang, F. Yu, et al. Segment anything in high quality. Proc. Annual Conf. Neural Information Processing Systems, 2024.
[26] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. Segment anything. In Proc. IEEE Int'l Conf. Computer Vision, 2023.
[27] C. Kong, A. Luo, S. Wang, H. Li, A. Rocha, and A. C. Kot. Pixel-inconsistency modeling for image manipulation localization. arXiv preprint urXiv:2310.00234, 2023.
[28] A. Kurakin, I. J. Goodfellow, and S. Bengio. Adversarial examples in the physical world. In Artificial intelligence safety and security. 2018.
[29] B. Landman, Z. Xu, J. Igelsias, M. Styner, T. Langerak, and A. Klein. Miccai multi-atlas labeling beyond the cranial works-wo and challenge. In Proc. MICCAI Multi-Atlar Labeling Beyond Cranial Vault- worktthp Challenge, 2015.
[30] T.-N. Le, T. V. Nguyen, Z. Nie, M.-T. Tran, and A. Sugimoto. Anabranch network for camou- flaged object segmentation. Computer vision and image understanding, 2019.
[31] Li, D. Li, Savarese, and S. Hoi. Blip-2: frozen image encoders and large language models. In Proc. Int'l Conf. Machine Learning, 2023.
Bootstrapping language-image pre-training with
[32] Q. Li, Y. Guo, and H. Chen. Yet another intermediate-level attack. In Proc. IEEE European Conf. Computer Vision, 2020.
[33] Q. Li, Y. Guo, W. Zuo, and H. Chen. Making substitute models more bayesian enhance transferability of adversarial examples. In Proc. Int'l Conf. Learning Representations, 2022.
[34] Q. Li, Y. Guo, W. Zuo, and H. Chen. Improving adversarial transferability via intermediate-level perturbation decay. Proc. Annual Conf. Neural Information Processing Systems, 2023.
[35] J. Lin, C. Song, K. He. L. Wang, and J. E. Hoperoft. Nesterov accelerated gradient and scale invariance for adversarial attacks. arXiv preprint arXiv: X /908 06281, 2019.
[36] Q. Lin, C. Luo, Z. Niu. X. He, W. Xie, Y. Hou, L. Shen, and S. Song. Boosting adversarial transferability across model genus by deformatiion-consrriine warping. In Proc. AAAI Conf. on Artificial Intelligence, 2024.
[37] Y. Liu, X. Chen, C. Liu, and D. Song. Delving into transferable adversarial examples and black-bo box attacks. In Proc. Int'l I Conf. Learning Representations, 2016.
[38] W. Ma, Y. Li, X. Jia, and W. Xu. Transferable adversarial attack for both vision transformers and convolutional networks via momentum integrated gradients. In Proc. IEEE Int'l I Conf. Computer Vision, 2023.
[39] M. A. Mazurowski, H. Dong, H. Gu, J. Yang, N. Konz, and Y. Zhang. Segment anything model for medical image analysis: experimental study. Medical Image Analysis, 2023.
[40] S.-M. Moosavi-Dezfooll A. Fawzi, O. Fawzi, and P. Frossard. Universal adversarial perturba- tions. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2017.
[41] A. Nichol, J. Achiam, and J. Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999 2018.
12

---

[42] A. Q. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. Mcgrew, I. Sutskever, and M. Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In Proc. Int'l Conf. Machine Learning, 2022.
[43] Y. Qian, S. He, C. Zhao, J. Sha, W. Wang, and B. Wang. Lea2: A lightweight ensemble adversarial attack via non-overlapping vulnerable frequency regions. In Proc. IEEE Int') I Conf. Computer Vision, 2023.
[44] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 22Xiv:2204.06125, 2022.
[45] S. Ren, F. Luzi, S. Lahrichi, K. Kassaw, L. M. Collins, K. Bradbury, and J. M. Malof. Segment anything, from space? Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024.
[46] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proc. IEEE Int'l I Conf. Computer Vision and Pattern Recognition, 2022.
[47] V. Q. Vo, E. Abbasnejad, and D. C. Ranasinghe. Brusleattack: A query-efficient acore-based Mlx sparse adversarial attack. In Proc. Int'l Conf. Learning Representations, 2024.
[48] C. Wang, Y. Yu, L. Guo, and B. Wen. Benchmarking adversarial robustness of image shadow removal with hhadow-adaptin attacks. In Proc. IEEE Int'l Conf. Acoustics, Speech, and Signal Processing, pages 13126-13 13130. IEEE, 2024.
[49] J. Wang, X. Li, and J. Yang. Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. Proc. IEEE Imlll Conf. Computer Vision and Puttern Recognition, 2018. 8.
[50] K. Wang, X. He, W. Wang, and X. Wang. Boosting adversarial transferability by block shuffle and rotation. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2024.
[51] X. Wang and K. He. Enhancing the transferability of adversarial attacks through variance tuning. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2021.
[52] X. Wang, X. He, J. Wang, and K. He. Admix: Enhancing the transferability of adversarial attacks. In Proc. IEEE Int'l Conf. Computer Vision, 2021.
[53] D. Wu, Y. Wang, S.-T. Xia, J. Bailey, and X. Ma. Skip connections matter: On the transferability of adversarial examples generated with resnets. In Proc. Int'l Conf. Learning Representations, 2019.
[54] J. Wu, R. Fu, H. Fang, Y. Liu, Z. Wang, Y. Xu, Jin, and T. Arbel. Medical sam adapter: Adapt- ing segment anything model for medical image segmentation. arXiv preprint arXiv:2304. 2023.
[55] S. Xia, Y. Yu, X. Jiang, and H. Ding. Mitigating the curse of dimensionality for certified robustness via dual randomized smoothing. In Proc. Int'l I Conf. Learning Representations, 2024.
[56] C. Xie, Z. Zhang, Y. Zhou, S. Bai, J. Wang, Z. Ren, and A. L. Yuille. Improving transferability of adversarial examples with input diversity. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2019.
[57] X. Xu, K. Kong, N. Liu, L. Cui, D. Wang, J. Zhang, and M. Kankanhalli. An Ilm can fool itself: A prompt- based adversarial attack. In The Twelfth International Conference on Learning Representations, 2023.
[58] Y. Yu, Y. Wang, Xia, W. Yang, S. Lu, Y.-p. Tan, and A. Kot. Purify unlearnable examples via rate-constrained variational autoencoders. In Proc. Int'l Conf. Machine Learning, 2024.
[59] Y. Yu, Y. Wang, W. Yang, S. Lu, Y.P. Tan, and A. C. Kot. Backdoor attacks against deep image compression via adaptive frequency trigger. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, pages 12250-12259, June 2023,
13

---

[60] Y. Yu, W. Yang, Y.-P. Tan, and A. C. Kot. Towards robust rain removal against adversarial attacks: A comprehensive benchmark analysis and beyond. Proc. IEEE Int'l Conf. Computer Vision and Puttern Recognition, 2022.
[61] K. Zhang and D. Liu. Customized segment anything model for medical image segmentation. arxiv preprint arXiv:2304. 13785, 2023.
[62] Z. Zhao, Z. Liu, and M. Larson. On success and simplicity: A second look at transferable targeted attacks. Proc. Anmual Conf. Neural Information Processing Systems, 2021.
[63] Z. Zhou, Hu, M. Li, H. Zhang, Y. Zhang, and H. Jin. Advclip: Downstream-agnosti adversarial examples in multimodal contrastive learning. In Proceedings of the 31st ACM International Conference on Multimedia, 2023.
[64] 7. Zhou, S. Hu, R. Zhao, Q. Wang, L. Zhang, J. Hou, and H. Jin. Downstream-agnostic adversarial examples. In Proc. IEEE Int'l Conf. Computer Vision, 2023.
[65] Z. Zhou, M. Li, W. Liu, S. Hu, Y. Zhang, W. Wan, L. Xue, L. Y. Zhang, D. Yao, and H. Jin. Securely fine-tuning pre-trained encoders against adversarial examples. In Proceedings of the 2024 IEEE Symposium Security and Privacy (SP'24), 2024.
[66] Z. Zhou, Y. Song, M. Li, S. Hu, X. Wang. L. Y. Zhang, D. Yao, and H. Jin. Darksam: Fooling segment anything model to segment nothing. In Proc. Annual Conf. Neural Information Processing Systems, 2024.
14

---

A Appendix
A.1 Randomness test
Table A.3: Experimental randomness of transfer-basee adversarial attacks on SAMs (subset).
Model
Medical SAM
Shadow-SAM
Camouflaged-SAN
Dataset
CT Scans
ISTD
CODIOK
CAMO
CHAME
Metries
mDSC|
mHD
BER
S,
MAE
S,
MAE ^
4
MAE
MI-FGSM
31.35
73.26
3.05
(.37
(.23
0.33
0.27vat--1
0.37:ate-1
(.24
DML-FGSM
27.19
87.54(16385)
2.53,1800
(.45
(.11
0.39:
0.20
(.43:10--)
(-14
PGN
40.64:a1.21
40.76
3 a.05
(.36
0.25
a.32 us
0.33
0.36
(.23)
BSR
28 41
93.41
2.58
(.45
(_11
0.40
0.20
0.43
(.14
ILPD
28. 28.
63,
2 2.74
0.35
0.25
0.33 a203
0.28(
(.37
(.24(c4e-))
MUI-GRAT
2.01
148.18
0.29
a.40.a
(.31
0.33
0.28
MUI-GRAT-DMI
2.23
123.76
4.52 (ascon)
0.40
0.20xa4e35
(.37
0.27
0.38
0.23
AUI-GRAT+PGN
15. 15.64
16.41
(.33
0.33
0.31
0.34:a
0,27
MUI BUI-GRATTBSR
(.25
82.50
41675)
3.87
Q.20
0.38: (G)-3)
(.27
4.40
0.22(10o0)
MUI-GRAT-ILPR
0.95
165.30
10.67:4088
0. (228:)
0.40:
0.30
0.33ca
0.34
0.28:
We evaluated 10 attack methods presented in our paper over 5 random seed runs on the subset of SAM's downstream tasks and reported the mean performance with its standard deviation. We use the same experimental setting provided in Section 6.1. The results indicate that the randomness is small and similar among all attacking methods. The uncertainty level of UMI-GRAT in mean Hausdorff Distance (mHD) is marginally higher compared to the other methods. This can account for the higher mHD value achieved by the UMI-GRAT.
A.2 Attacking open-sourced SAMs
Table A.4: The performance of open-sourced SAMs under attacks.
Si--Vit-B
SAM-Vit-L
SAM-Vit-H
Surrogate model
Attack
mAP/
miou4
mAP
m1Ou4
mAP/.
mIOu!.
MI-FGSM
1.24
5.23
14.17
24.12
24.20
34.96
SAM-Vit-B
MUI-GRAT
0.65
2.21
3.67
9.27
8.43
15.41
MI-FGSM
14.18
21.11
1.11
4.03
16.42
25.10
SAM-Vit-L
MUI-GRAT
16.41
23.05
1.39
2.40
18.12
28.67
MI-FGSM
21.74
29.35
15.58
24.71
1.46
6.51
SAM-Vit-H
MUI-GRAT
18.50
25.26
10.00
18.12
0.58
2.25
We report the mean average precision (mAP) and mean intersection-over-unioo (mIOU) metrics to evaluate the performance of attacking open open-sourced SAMs. We compure the performance of MI- FGSM and ours MUI-GRAT. The implementation of these attacks adheres to the details described in Section 6.1 We randomly select 500 images from the SA- dataset and evaluate the performance of SAMs under the AutommttiMMaskGenentt and SAM- models. Meanwhile, we also discuss the black-box transferability of generated AEs across different models. The results are shown in Table[A.4
mode. We white-box attacks SAM--ittB, SAM-ViL Li
Our findings reveal that both MI-FGSM and MUI-GRAT achieve comparably strong attack perfor- mance in the white-box scenario. In the black-box scenario, where AES generated one surrogate model are transferred to a different victim model, our results indicate that the surrogate model critical for the transferability of generated AFs. The AEs with the strongest transferability are generated by our MUI-GRAT on attacking SAM-Vit-B, which exhibits a great performance gain compared with others. However, when employing SAM-ViT-L as the surrogate model, there is slight reduction in the transfcrability of AEs produced by our MUI-GRAT. Conversely, when the surrogate model is SAM-Vit-HE the transferability of AEs generated by our MUI-GRAT surpasses MI-FGSM by a large margin.
15

---

A.3 Analysis of the attack efficiency
We analyze the real-time attack efficiency of the meth- ods mentioned above. We report the average time re- quired for generating one AE when the input resolution is 512 x 512 on the SAM-Vit-B model using one RTX 4090 GPU. The results are shown Table A.5
Table A.5: Analysis of the attack efficiency
Num. samples needed iteration
Method
Avg. time (s)
MI-FGSM DMI-FGSM
0.32
0.69
PGN
3.96
We find that our proposed MUI-GRAT achieves second- high efficiency when compared with others. The ILPD requires extra attack iterations (e.g., 10-step MI- FGSM) find directional guide vector v. The input augmentation- based attack methods, such as BSR and DMI-FGSM, require multiple samples in each iteration to do the augmentation and the PGN also requires multiple samples to obtain stable gradi- ent direction. However, our method utilizes an offline generated MUI and only conducts one- me-time gradient augmentation in each iteration, thus achieving much higher efficiency than others.
BSR
2.91
MUI-GRAT
0.36
A.4 Hardware Setup
We run our experiments for attacking the medical segmentation model using one RTX 4090 GPU with 24 GB memory. We the rest of the experiments using RTX A6000 GPU with 48 GB memory.
16

---

A.5 Visualization of the adversarial examples and the prediction
We visualize adversarial examples generated by MUI-GRAT utilizing solely the open-sourced SAM and the corresponding adversarial predictions in Figure A.3 and Figure The images are randomly selected. This visualization provides more straightiorward demonstration of the impact of the adversarial attack threatened by MUI-GRAT, showing how it significantly compromises the reliability of the large foundation model's 's predictions with just single independent input image.
COD-10K
CHAME
CAMO
Clean Image
Adversarial example
Ground truth
Adversarial prediction
Figure A.5: The visualized adversarial attack results in camouflaged object segmentation task.
S4-1B
multi-oraa dataset
ISTD
Clean image
Adversarial example
Ground truth
Adversarial prediction
Figure A.6: The visualized adversarial attack results in natural, medical, and shadow image segmen-
tation tasks.
17

---

A.6 Impact Statement
The increasing reliance on large foundation models in various wal-world upplications amplifies the critical importance of ensuring their secure utilization. This paper mainly discusses the potential risks of adversarial threats associated with the direct utilization and fine-tuning of the open-sourced model even on private and encrypted datasets. To accomplish that, we hegin an investigation into more practical while challenging adversarial attack problem: attacking various SAM's 's downstream models by solely utilizing the information from the open-sourced SAM. We then provide the theo- retical insights and build the experimental setting and benchmark, aiming to serve as a preliminary exploration for future research in this area. Experimentally, and its downstream models under the proposed MUI-GRAT, indicating the security risk inherent in the direct utilization and fine-tuning of open-sourced large foundation models, thus highlighting the urgent need for robust defense mechanisms to protect these models from adversarial threats.
validate the vulnerability of SAM
A.7 Limitations
The limitations of our paper are:
The proposed UMI-GRAT is not contingent upon a prior regarding the model's architecture, suggesting its potential applicability across various model paradigms. However, the experi- ments only tested MUI-GRAT on the prevalent SAMs and their downstream models. The capability of UMI-GRAT to pose threat to other large foundation models remains topic for further exploration.
While this paper highlights the risk of direct utilization of SAM and fine-tuning on the downstream task, this paper does not provide and validate an effective solution for this secure concern.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
We demonstrate the contributions and the scope of our paper in the abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in Appendix A.7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

For each theoretical result in Section we give the detailed proof and illustration.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We present the details of the implementation of our method in Section 6.1
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [No]
Justification: We intend to release the code associated with this work subsequent to the paper's acceptance. This will allow confirm that the code is stable and thoroughly tested prior to its public dissemination. Our goal in releasing the code is to enable replication of our findings, foster collaboration with fellow researchers, and uphold the principles of open science.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https / //nips httpps.ccppbblicc/cc gwtcc//gugeeeeee / ssiicyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We give the implementation details and explain every result we got in the experiment help readers understand.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive. But we keep the random seed fixed for all competing methods.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We list the equipment needed for running and reproduce our experiment in Appendix A.4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics /tctps://neurips... http///wwww.p..eppoboaacaecceaee
Answer:[Yes]
Justification:
We have reviewed and met the code of ethics for our research.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the social impact of our research in Section A.6
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This research does not have this kind of risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
23

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moddl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not describe potential risks incurred by study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.


Reviews:
[{"review_id": "jeQw521yMz", "reviewer": "Reviewer_pDnZ", "summary": "In this paper, the authors propose an adversarial attack method that can contaminate downstream tasks from the perspective of adversarial transferability. They address the problem that SAM models do not have similar optimisation routes after fine-tuning for different downstream tasks by designing universal meta initialization. In this paper, the authors address the problem that SAM models do not have similar optimisation routes after fine-tuning for different downstream tasks by designing UMI noise. The authors introduce the idea of meta-learning to allow their algorithm to quickly adapt to different situations, i.e., downstream tasks.", "questions": "Two questions listed, see Weaknesses for details. Note that if the authors can demonstrate the randomness of their algorithms, that will help to get a higher rating.", "limitations": "The authors correctly list the implications of their work for the use of large models such as SAM in downstream tasks.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The theoretical part of this paper is detailed, the experiments are sufficient. The comparison with other methods shows the sophistication of their approach.\n\n\n\n2. The attacks proposed in this paper are novel. It contributes to the topic of attacking downstream tasks of large models. A discussion on adversarial transferability is introduced under this topic.", "weaknesses": "1. The readability of the Methodology section of this article is somewhat poor. The authors define the problem to be solved through the form of propositions. Similarly, if the authors could summarise the formulas as Theorem and put the proof process (both formulas and reasoning) specifically in the supplementary material, it would make the article more coherent.\n\n\n\n2. The randomness of the experimental results is unknown. I understand that due to the larger computational effort, it is not practical to report error lines on all major experiments. But it would be better for the authors to report a set of randomness on a smaller dataset and simpler settings, which will influence the reviewers' opinion of the results of this method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer pDnz,\n\nThank you so much for taking the time to read this paper and giving constructive feedback. Please find our response below.\n\n> 1.The readability of the Methodology section of this article is somewhat poor. The authors define the problem to be solved through the form of propositions. Similarly, if the authors could summarise the formulas as Theorem and put the proof process (both formulas and reasoning) specifically in the supplementary material, it would make the article more coherent.\n\nThanks for your constructive feedback. We will consolidate the Proposition and Equations (15) through (18) into a Theorem and put the remaining formulas and explanatory text in the appendix.\n\n\n\n> The randomness of the experimental results is unknown. It would be better to report a set of randomness experiments on a smaller dataset and simpler settings.\n\nFollowing your good suggestion, we evaluated **10** attack methods presented in our paper over **5** random seed runs on the subset of SAM's downstream tasks and reported the mean performance with its standard deviation. We use the same experimental setting provided in Section 6.1 of our submitted paper.\n\nThe details of each subset are: \n\n1. For the Medical SAM, we selected 'case0008' (comprising 148 images with a resolution of 512x512) from the validation set of the CT-Scan dataset. \n2. For the Shadow SAM, we randomly selected 100 images with a resolution of 1024x1024 from the test set of the ISTD dataset. \n3. For the Camouflaged SAM that is evaluated across three different datasets, we randomly selected 40 images with a resolution of 1024x1024 from each dataset (COD10k, CAMO, CHAME), with a total of 120 images.\n\nThe results are shown in **Table R1** of our submitted **PDF file**,  which demonstrates that the randomness is small and similar among all attacking methods. The uncertainty level of UMI-GRAT in mean Hausdorff Distance (mHD) is marginally higher compared to the other methods. This can account for the higher mHD value achieved by the UMI-GRAT.\n\nWe greatly appreciate your constructive feedback. We will conduct the randomness evaluation over the entire dataset and update the results in Table 1 and Table 2 of our paper in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9wtoTQ5pyF", "reviewer": "Reviewer_Qxcd", "summary": "This paper investigates the vulnerability of Segment Anything Model (SAM) and its downstream models to transferable adversarial attacks. The authors propose a novel attack method called Universal Meta-Initialized and Gradient Robust Adversarial attack (UMI-GRAT) that leverages the open-sourced SAM to generate adversarial examples effective against fine-tuned downstream models, even without access to the downstream task or dataset.", "questions": "1. How does the performance of UMI-GRAT vary with different choices of hyperparameters, such as the perturbation bound ε and the number of iterations in UMI and LGR. Especially, line 278 mentions that the perturbation bound is 10, which is a bit too large.\n2. According to the latest benchmark **[R1]**, baselines used in the paper are not SOTA methods. How does it compare with NCS **[R2]**, ANDA **[R3]**, DeCowA **[R4]** and L2T **[R5]**?\n3. Is the proposed method a universal transfer attack method? Although this question is mentioned on line 548, can the performance of UMI-GRAT and SOTA be compared under a general transfer attack test setting?\n4. UMI-GRAT is a gradient-based attack method. How does the proposed method perform when the model has a certain robustness (such as adversarial training)?\n\n---\n**[R1]** Devling into Adversarial Transferability on Image Classification: A Review, Benchmark and Evaluation.   \n**[R2]** Enhancing Adversarial Transferability Through Neighborhood Conditional Sampling.   \n**[R3]** Strong Transferable Adversarial Attacks via Ensembled Asymptotically Normal Distribution Learning. CVPR. 2024.   \n**[R4]** Boosting Adversarial Transferability across Model Genus by Deformation-Constrained Warping.   \n**[R5]** Learning to Transform Dynamically for Better Adversarial Transferability", "limitations": "Limitations are discussed in the Appendix.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper tackles a practical and challenging problem of attacking downstream models fine-tuned from a publicly available foundation model without knowledge of the downstream task or data.\n2. The proposed UMI-GRAT method is well-motivated and technically sound. The authors provide theoretical insights into the gradient deviation problem and propose a robust solution using gradient noise augmentation.\n3. The paper presents extensive experiments demonstrating the effectiveness of UMI-GRAT in attacking SAM and its downstream models", "weaknesses": "See Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Qxcd,\n\nThank you so much for taking the time to read this paper and giving constructive feedback. Please find our response below.\n\n> 1. How does the performance of UMI-GRAT vary with different hyperparameters, such as the bound $\\epsilon$ and iterations?\n\nFollowing your good suggestion, we conducted experiments using two additional sets of hyperparameters: bound $\\epsilon=4$ with iteration $T_a=10$ and bound $\\epsilon=10$ with iteration $T_a=20$. We evaluated 5 different attacks and reported the experimental results in **Table R2** of our submitted PDF file. The results indicate that: \n\n1. In datasets that exhibit a large domain gap from natural image datasets (e.g., medical and shadow datasets), the attack bound $\\epsilon$ is critical for transferability. Reducing the bound leads to a substantial performance decline for all attack algorithms. For the camouflaged object segmentation that shares a small domain gap with the original SAM's task, reducing the norm bound only causes a marginal performance drop.\n2. Increasing the attack iterations markedly enhances the AEs when the surrogate and victim domains are proximate. However, for those tasks characterized by a substantial domain gap, increasing the iteration has little help.\n\n> 2. Is the proposed method a universal transfer attack method? Can the UMI-GRAT and SOTA be compared under a general transfer attack test setting?\n\n Following your good suggestion, we evaluated the effectiveness of our proposed UMI-GRAT on two new scenarios: \n\n1. **The transferability of MUI-GRAT to other pre-trained methods.** We attacked the **MAE ViT-S and ViT-B that are fine-tuned on Chexpert [2]** using solely **the pre-trained MAE ViT-S** and reported the results in **Table. R3** of our submitted PDF file. \n2. **The effectiveness of MUI-GRAT on the general transfer attack setting.** We attacked the **MAE ViT-B and MAE DenseNet-121** **fine-tuned** **on Chexpert [2]** using **MAE ViT-S fine-tuned** on the same dataset and reported the results in **Table. R4**.\n\nWe utilized models provided by [1] and followed the same experimental setting. In both scenarios, we evaluate 8 attack methods on the Chexpert [2] dataset, where the model needs to diagnose five predefined diseases in the chest X-ray images.  We reported the mean Area Under the Curve (mAUC) to evaluate the performance. \n\nThe results in **Table R3 and R4** demonstrate the great generalizability of our proposed method. **Table R3** shows that the proposed **MUI-GART maintains the best method** on attacking both the fine-tuned **MAE-ViT-S and MAE-ViT-B** models. In **Table. R4**, the evaluation of the transferability between different ViTs and transferability from the ViT to the CNN indicates that our MUI-GRAT maintains effectiveness (**the second-best method**) on general transfer tasks.\n\n> 3. How does it compare with NCS **[R2]**, ANDA **[R3]**, DeCowA **[R4]** and L2T **[R5]**\n\nFollowing your good suggestion, we evaluated two currently open-sourced methods, **ANDA[R3] and L2T[R5]**. We conducted experiments considering the above two scenarios along with attacking the **normally trained and adversarial trained Medical SAM**. We set the n_ens=5 for ANDA and num_ scale=3 for L2T and keep the rest hyperparameters the same. We reported the experimental results in **Table R3, R4, and R5** in our submitted PDF file.  \n\nThe results show that L2T performs well in both transfer tasks on MAE-based models while failing in attacking the SAM and adversarial-trained SAM. ANDA performs well on the general transfer tasks while failing on all pre-trained to fine-tuned transfer tasks. We hypothesize the reason for their failures as: \n\n1.  Due to the great misalignment in the gradient discussed in Definition 1 and Proposition 1, the transfer between pre-trained and fine-tuned models is much harder than the general one. \n2.  The feature-oriented attacking task may potentially undermine those output-oriented attack methods.  \n\n> 4.  How does the proposed method perform when the model has a certain robustness (such as adversarial training)?\n\nFollowing your constructive feedback, we conducted comparative experiments on attacking the **adversarial trained Medical SAM**. As our attacks aim to maximize the feature distance, we thereby consider two types of adversarial training (AT) mechanisms:\n\n1. **Feature-wise AT**: the defender is **aware of feature-wise attacks**, thus assuming an attacker to **maximize the distance of feature embedding** during AT, and the defender thereby minimizes the adversarial loss.\n2. **Output-wise AT**: the defender is **unaware of feature-wise attacks**, thus assuming an attacker to **maximize the final segmentation loss** during AT, and the defender thereby minimizes the adversarial segmentation loss.\n\nWe optimize the Medical SAM by incorporating the adversarial loss into the training loss with a weight factor $\\tau$. We used $\\tau=0.1\\,0.5$ and took the MI-FGSM with iteration $T_a =1, 5$ and the same bound $\\epsilon=10$ as the attacking strategy.  \n\nWe ran **9 different attacks on 6 different AT models** and presented the results in **Table R.5**, demonstrating that:\n\n1. The proposed UMI-GRAT remains the most effective method even if the model has a certain robustness via AT, demonstrating the effectiveness of our proposed method.\n2. Feature-wise AT surpasses output-wise AT towards those feature maximizing attacks in most scenarios.\n3. Enlarging the weight $\\tau$ benefits the robustness. An intriguing finding is that using a stronger gradient attack during AT (e.g . 5-steps MI-FGSM) may damage the robustness.  \n---\nWe greatly appreciate your constructive feedback and will add all those experiments with analyses and cite all mentioned papers in our revised version.\n\n[1]  Delving into masked autoencoders for multi-label thorax disease classification. In WACV 2023.\n\n[2]  Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In AAAI 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qlZmlzla2X", "reviewer": "Reviewer_Q28B", "summary": "This paper proposes an adversarial attack against fine-tuned derivatives to a publicly available foundation model, such as the Segment Anything Model (SAM). In the proposed threat model, attackers can potentially manipulate these downstream models even without knowing the specific task or data they are used for. Under this threat model, proposes a new attack method called UMI-GRAT (Universal Meta-initialized and Gradient Robust Adversarial Attack). Through a bi-level optimization procedure, this method leverages the information from the open-source SAM to create adversarial examples that can fool mislead the original SAM and its fine-tuned versions. Finally, this paper demonstrates the effectiveness of the proposed UMI-GRAT attack against SAM through extensive experiments.", "questions": "1. How does the domain gap between the natural image dataset used to obtain the universal adversarial trigger and the downstream dataset influence the effectiveness of the attack?\n2. How effective is the proposed method against adaptive defense? For example, if the downstream victim model has gone through adversarial training, how effective would the adversarial trigger obtained on the unguarded pretrained SAM be against the guarded victim model?", "limitations": "The authors acknowledge the limitations of this work in the appendix. They candidly acknowledge the limitations in evaluations as the proposed attack is only evaluated against SAM. I appreciate the authors openly acknowledging this limitation.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is motivated by real-world safety concerns for fine-tuning a public foundation model on private domain-specific datasets.\n2. The figures and tables are well-polished and generally reflect the overall message of the paper.\n3. The proposed UMI-GRAT attack method is unique and backed by theoretical analysis.", "weaknesses": "1. The effectiveness of the proposed attack is only demonstrated by attacking the SAM model. However, more experiment settings (e.g. against pretrained MAE models) are warranted to demonstrate the generalizability of the proposed attack.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Q28B,\n\nThank you so much for taking the time to read this paper and giving constructive feedback. Please find our response below.\n\n> 1. More experiment settings (e.g. against pretrained MAE models) are warranted to demonstrate the generalizability.\n\n Following your good suggestion, we evaluated the effectiveness of our proposed UMI-GRAT on two new scenarios: \n\n1. **The transferability of MUI-GRAT to other pre-trained methods.** We attacked the **MAE ViT-S and ViT-B that are fine-tuned on Chexpert [2]** using solely **the pre-trained MAE ViT-S** and reported the results in **Table. R3** of our submitted PDF file. \n2. **The effectiveness of MUI-GRAT on the general transfer attack setting.** We attacked the **MAE ViT-B and DenseNet-121** **fine-tuned** **on Chexpert [2]** using **MAE ViT-S fine-tuned** on the same dataset and reported the results in **Table. R4**.\n\nWe utilized models provided by [1] and followed the same experimental setting. In both scenarios, we evaluate 8 attack methods on the Chexpert [2] dataset, where the model needs to diagnose five predefined diseases in the chest X-ray images.  Following [1], we report the mean Area Under the Curve (mAUC) to evaluate the performance. Each experiment is run for 5 times.\n\nThe results in **Table. R3** shows that the proposed **MUI-GART maintains the best method** on attacking both the **MAE-ViT-S and MAE-ViT-B** models fine-tuned on the downstream data. Moreover, the combination of the MUI-GRAT with the second-best method BSR [3] brings a further enhancement to the performance.\n\nIn **Table. R4**, we evaluated the transferability between different ViTs and transferability from the ViT to the CNN.  The experimental results indicate that our MUI-GRAT maintains effectiveness on general transfer tasks (MUI-GRAT achieves **the second-best** attack performance compared to other SOTA methods), demonstrating its good generalizability.\n\n> 2. How does the domain gap between the natural image dataset used to obtain the universal adversarial trigger and the downstream dataset influence the effectiveness of the attack?\n\nThe efficacy of the universal adversarial trigger increases as the domain gap between the downstream dataset and the natural image dataset narrows. As mentioned in lines \"334-340\" of the submitted paper, the UMI extracts the intrinsic vulnerability inherent in the foundation model, thus being more effective when the victim model inherits significant information from the pre-trained model. The following table presents the performance of the SAM before and after being fine-tuned on downstream tasks:\n\n|                                  | Medical SAM (mDSC) | Camouflaged SAM (MAE) |\n| -------------------------------- | :----------------: | :-------------------: |\n| Performance pre/post-fine-tuning |     1.39/81.88     |      0.050/0.025      |\n| Attack performance gain by UMI   |        3.29        |       **2.98**        |\n| Attack performance gain by GRAT  |     **34.49**      |         0.37          |\n\nThe results show that the fine-tuning brings a significant performance gain for Medical SAM but a minor performance gain for Camouflaged SAM, indicating that the domain gap between the medical and natural datasets is huge and the model is accompanied by a pronounced gradient update. The domain gap between the camouflaged and natural datasets is small so that the gradient modification following fine-tuning is minor. The performance gain shown in the rest data demonstrates that the UMI derives a greater advantage than the GRAT when the domain gap is small and GRAT displays a contrary trend, which aligns with our analysis in Section 4.  \n\n> 3. How effective is the proposed method against adaptive defense?\n\nFollowing your constructive feedback, we conduct comparative experiments on attacking the **adversarial trained Medical SAM**. As our attacks aim to maximize the feature distance, we thereby consider two types of adversarial training (AT) mechanisms:\n\n1. **Feature-wise AT**: the defender is **aware of feature-wise attacks**, thus assuming an attacker whose objective is to **maximize the distance of feature embedding** during AT, and hence the defender minimizes the adversarial loss.\n2. **Output-wise AT**: the defender is **unaware of feature-wise attacks**, thus assuming an attacker whose objective is to **maximize the final segmentation loss** during AT, and hence the defender minimizes the adversarial segmentation loss.\n\nWe optimize the Medical SAM by incorporating the adversarial loss into the training loss with a weight hyperparameter $\\tau$. We evaluate $\\tau=0.1\\,0.5$ and use the MI-FGSM with iteration $T_a =1, 5$ with bound $\\epsilon=10$ as the attacking strategy.  \n\nWe ran 9 different attacks on 6 different AT models on CT-Scan and presented the results in **Table R.5**. The results demonstrate that:\n\n1. The proposed UMI-GRAT remains the most effective method even if the model has a certain robustness via AT, demonstrating the effectiveness of our proposed UMI-GRAT.\n2. Feature-wise AT surpasses output-wise adversarial training for those feature maximizing attacks in most scenarios.\n3. The robustness of AT varies drastically with different training hypermeters. Enlarging the weight $\\tau$ benefits the robustness. An intriguing finding is that using a stronger gradient attack (e.g . MI-FGSM $T_a =1\\,to \\, 5$) during AT may damage the robustness towards **adversarial examples generated from the pre-trained model**.\n\n---\n\nWe greatly appreciate your constructive feedbacks. We will add those experiments and analyses mentioned above in the appendix of our revised version.\n\n[1] Delving into masked autoencoders for multi-label thorax disease classification. In WACV 2023.\n\n[2] Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In AAAI 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SxWzavlAfX", "reviewer": "Reviewer_iuQV", "summary": "In this paper, the authors present a new approach for adversarial attacks on Segment Anything Model (SAM)-based downstream models, addressing the challenge of attacking without prior knowledge of the downstream task or data distribution. Their key contribution is a universal meta initialization-based algorithm that exposes inherent vulnerabilities in the foundation model. The authors also introduce a gradient robust loss, which simulates uncertainty through gradient-based noise augmentation. This loss is derived from a theoretical formulation of adversarial update deviation between the open-sourced SAM and its fine-tuned downstream models. The authors provide an analytical demonstration of how their proposed method enhances attack transferability. The effectiveness of their approach is thoroughly validated through comprehensive experiments.", "questions": "Please see the weakness section. While the paper presents an approach to attacking SAM-based downstream models, it largely combines existing methods rather than introducing new techniques. The current strategy, though effective, does not fully exploit SAM's unique architecture.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Originality: This is the first work to explore the feasibility of adversarially attacking various downstream models fine-tuned from the Segment Anything Model (SAM). The introduction of a universal meta initialization-based algorithm to uncover intrinsic vulnerabilities in foundation models is both effective and efficient. Additionally, the formulation of adversarial update deviation and the proposal of a gradient robust loss that simulates uncertainty with gradient-based noise augmentation further enhance the transferability of adversarial examples.\n\nQuality and Clarity: The writing is generally clear but has room for improvement. The methodology and results are well-structured, though some technical sections could benefit from additional clarification.\n\nSignificance: This work is highly significant given the increasing prevalence of foundation models like SAM. The proposed methods for enhancing attack transferability have important implications for AI system security and could influence future directions in both offensive and defensive strategies in adversarial machine learning for SAM.", "weaknesses": "1 - My major concern is related to the novelty of the proposed approach. Although I agree that this is the first work in the context of SAMs, the main components, such as downstream agnostic adversarial examples and meta learning-based fast initialization, have already been proposed in the literature.\n\n2 - The authors, in line 45, briefly highlight downstream agnostic examples in just one line. They should clarify in the related work section how their work is different from references 55 and 56 of the main paper, beyond just applying it to SAM. Similarly, another related work that the authors missed is [1] (given below), in which the generated adversarial examples are agnostic to downstream tasks.\n\n3 - Similarly, the authors did not mention any work related to meta-learning-based adversarial examples in the paper. There are multiple works that use meta-learning to craft universal adversarial examples, such as [1, 2] below. The authors use these meta-learning-based methods for initialization of adversarial examples, but this has already been explored in [3] below. The authors should mention these meta-learning-based approaches in their paper and discuss how their method is different from these approaches, beyond just the application to SAMs.\n\n4 - It is not clear to me when the authors claim in line 8 that they are attacking \"without accessing the downstream task.\" What is the task here? Is it not the segmentation task? In [1], their task-agnostic adversarial examples are effective against classification, detection, and segmentation. Since the downstream task here is segmentation-based, is it not obvious what the task is? Please clarify this.\n\n5 - The authors should include some specific aspects of SAM to make their attack more unique. Currently, they are utilizing the SAM image encoder, which, in my opinion, is not much different from the previous works listed below.\n\n6 - For experiments, why have the authors compared their method with intermediate-level feature-based approaches? They should also compare it with different downstream agnostic adversarial approaches as listed below.\n\n7 - In Equation 8, how did the authors choose the threshold lambda? \n\n[1] A Self-supervised Approach for Adversarial Robustness (CVPR-2020)\n\n[2] Learning to Generate Image Source-Agnostic Universal Adversarial Perturbations (IJCAI22)\n\n[3] Meta Adversarial Perturbations (AAI2022-Workshop)\n\n[4] Adversarial Initialization with Universal Adversarial Perturbation: A New Approach to Fast Adversarial Training", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer iuQV\n\nThank you so much for taking the time to read this paper and giving constructive feedback. Please find our response below.\n\n> Q1, Q2, and Q3: The novelty of the proposed method. The authors should discuss how their method is different from meta-learning-based approaches [2,3,4]. Clarify how their work is different from references 55 and 56. Another work[1] is missing.\n\nAlthough utilizing the UAP as the initial point is discussed in [2,3,4] and downstream-agnostic attacks are discussed in [1, 55, 56], the core methodology design in our UMI is significantly different from previous work. Moreover, the proposition of GRAT in mitigating the gradient misalignment is unique, novel, and effective. The main differences are:\n\n1. **Compared with meta-learning approaches:** [3] utilizes the UAP to generate AEs with a **one-step update**, and [4] uses UAP for **adversarial training**. [2] enhances the existing UAP under the **few-shot learning** scenarios:  learning a UAP with a few examples to attack the same victim model. Different from previous work, we first utilize the UMI to extract the intrinsic vulnerability **inherent in the pre-trained foundation model** and utilize it as the prior knowledge to enhance the **attack on fine-tuned downstream models**.\n\n2. **Compared with model-agnostic approaches:** [1] first proposed a representation-based adversarial attack to enhance the downstream-agnostic adversarial training. [55] proposed the first framework for generating downstream-agnostic UAP on self-learning models and [56] extends this attack to multimodality. However, those methods do not consider the utilization of **intrinsic vulnerability in the pre-trained model and the gradient misalignment brought by fine-tuning**, thus do not work well when the downstream dataset **exhibits a distinctive domain gap** from the pre-trained dataset (e.g. from a natural dataset to the medical dataset).\n\nThe novelty of our work is two-fold:\n\n1.  **Exploitation of intrinsic vulnerability via UMI**: we first utilize the UMI to extract the **intrinsic vulnerability inherent in the pre-trained foundation model** and utilize it to enhance the attack on fine-tuned downstream models. \n\n2.  **Rectification of gradient misalignment via GRAT**: Inspired by our **Proposition 1, which formulates the deviation** occurred on attacking the unknown fine-tuned model, we propose the **GRAT to effectively mitigate this misalignment**. The experimental results in Figure 4 and Table 2 demonstrate that **when the fine-tuned model has a pronounced gradient update, the proposed GRAT greatly rectifies the deviation and brings a great performance gain**.\n\n\nWe will discuss and cite all the mentioned work above in our revised paper.\n\n> Q2 and Q6: Why compare with intermediate-level feature-based approaches? They should also compare it with different downstream agnostic approaches.\n\n**The Reason for comparison with ILPD is**: as shown in Equation (7) of the submitted paper, our attack aims to **maximize the feature distance**. We thus discuss and compare our method with the SOTA feature-based attack. \n\n**Comparison with [1]**: according to Section 3.1 and Equation (4) in [1], the attack of [1] directly perturbs the input to **maximize feature distortion via MI-FGSM**. According to Equation (7) of the submitted paper, all methods compared in Table 1 aim to **maximize the feature distortion of SAM's image encoder**. This means **the compared MI-FGSM in Table 1 is totally the same as the attack algorithm proposed in [1]**. We will add more explanations in our revised version to dispel this confusion. \n\n**Comparison with [55, 56]**: as [55, 56] focus on **universal adversarial perturbation**, directly comparing them with **the input-specific attacks**  is somewhat unfair to them. We evaluated the Adv-PER [55] in attacking **normally trained and adversarial-trained Medical SAMs** and reported the experimental results in **Table R5** of our submitted PDF file, indicating that the Adv-PER does not work well when the downstream dataset exhibits a significant domain gap from the pre-trained dataset. \n\n> Q4. Clarify the claim \"without accessing the downstream task\".\n\nOur attack perturbs the feature encoder of SAM, which, by fine-tuning the decoder, can zero-shot and few-shot transfer to various **downstream tasks, such as edge prediction**. **Without accessing the task, the distortion in the feature encoder** will adversely affect the performance across different types of decoders. To substantiate this and demonstrate the generation, we conducted a new experiment on attacking the MAE model fine-tuned on the Chexpert [5], a **task for diagnosing chest X-ray diseases**. We reported the results in **Table R3 and R4** of our submitted PDF, demonstrating the effectiveness and generalization of our method across different downstream tasks.\n\n> Q5. The authors should include some specific aspects of SAM to make their attack more unique.\n\nAs stated in lines 548-549 in our submitted paper, our proposed UMI-GRAT is not contingent upon a prior on model’s architecture, suggesting its potential applicability across various model paradigms. The experiments on attacking the fine-tuned MAE ViTs and CNNs in Tables R3 and R4 demonstrate its good generalization. Following your suggestion, we will clarify this in our revised version. \n\n> Q7. How to choose the threshold lambda in Equation 9?\n\nThanks for your good suggestion, we will add the discussion below in our revised paper:\n\n\"We initialize the $\\lambda$ with a value of 0.05, which is easily satisfied by most inputs, at the first epoch and increase it by a factor of 2 if at least 50% of the inputs meet this threshold.\"\n\n\n---\n\nWe greatly appreciate your constructive feedback and will add all the experiments with analyses and cite all mentioned papers in our revised version.\n\n[5]  Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In AAAI 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C9cma5vwTW", "reviewer": "Reviewer_aXPE", "summary": "This work discusses an interesting security issue of deploying a model fine-tuned on a large foundational model in private downstream applications. It proposes a universal meta-initialized and gradient robust adversarial attack (UMI-GRAT) to break the powerful SAM and its various downstream models, without requiring prior knowledge of the specific downstream task and data distribution. The author explores the challenges associated with threating transfer-based adversarial attack without the task-related prior knowledge and provides the theoretical insights on the deviation in updating the adversarial perturbation when using the open-sourced model as the surrogate model. An extensive evaluation of UMI-GRAT's performance, transferability, and efficiency was conducted across five datasets and three different downstream tasks (medical image segmentation, shadow segmentation and camouflaged object segmentation), demonstrating the high effectiveness of the UMI-GRAT approach.", "questions": "1.\tIn Figure 3, the peak cosine similarity of the generated adversarial perturbation is observed between the 20th and 30th iteration. So will increasing the iterative step of generating adversarial perturbation enhance the transferability? \n2.\tModel ensemble is an effective method to enhance the adversarial attacks’ transferability. Will the ensemble of different SAMs benefit the UMI-GRAT?", "limitations": "The experimental results are all based on SAMs and their downstream models. It will provide more valuable insights if expanding the scope of analysis to assess whether this adversarial threat also applies to other large foundation models.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work discusses a critical adversarial issue of deploying large foundation model in real-world applications and for the first time considers a more challenging and practical scenario where the adversarial attacker breaks SAM and its downstream models in the absence of prior knowledge on the task and data distribution.\n2. This work provides the in-depth analysis on the challenge of threating the transferable adversarial attack via open-sourced SAM and proposes the corresponding theoretical insights and solution.  \n3. The work establishes a detailed experimental framework and the proposed UMI-GRAT shows superior performance on misleading various SAMs’ downstream models compared with previous methods, which serve as a preliminary exploration for future research.", "weaknesses": "1.\tIt’s recommended to give more comprehensive analysis of the UMI noise, including the size of the natural image dataset and the effect of various hyperparameters.\n2.\tThere are more metrics such as $E_\\phi$, $F_\\beta^\\omega$ in the camouflaged object detection task. It would be beneficial if the author could provide further data pertaining to these evaluation metrics to enrich the analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer aXPE,\n\nThank you so much for taking the time to read this paper and giving constructive feedback. Please find our response below.\n\n> Give a more comprehensive analysis of the UMI noise.\n\nFollowing your valuable suggestion, we conducted an evaluation to analyze the impact of the natural dataset size $N$ and the total meta iteration times $T_m$ on the UMI using the COD10K dataset.  We presented the results in the following table.\n\n|  Hyperparameters  | $S_\\alpha\\downarrow$ | MAE$\\uparrow$ |\n| :---------------: | :------------------: | :-----------: |\n| $T_m=7,N=20,000$  |        36.11         |   **24.74**   |\n|   $T_m=7,N=200$   |        37.54         |     20.83     |\n|  $T_m=7,N=5,000$  |        36.81         |     23.23     |\n| $T_m=3,N=20,000$  |        36.43         |     24.30     |\n| $T_m=5,N=20,000$  |      **36.01**       |     24.58     |\n| $T_m=11,N=20,000$ |        36.59         |     24.59     |\n\nThe results indicate that the size of the dataset $N$ has a significant impact on the UMI. As a smaller dataset can introduce substantial bias to the generated UMI, this thereby leads to the degraded performance of the generated UMI. The effect of  $T_m$ becomes minor when larger than 5.\n\n> Provide further data pertaining to $E_\\Phi $ and $F_\\omega ^\\beta $.\n\nWe presented the performance of $E_\\Phi $ and $F_\\omega ^\\beta $ on the COD10K dataset under five different attacks in the following table:\n\n| Attack strategy | $E_\\Phi\\downarrow$ | $F_\\omega ^\\beta\\downarrow$ |\n| :-------------: | :----------------: | :-------------------------: |\n| Without attack  |       0.918        |            0.801            |\n|     MI-FGSM     |       0.569        |            0.019            |\n|       PGN       |       0.537        |            0.021            |\n|       BSR       |       0.513        |            0.058            |\n|      ILPD       |       0.502        |          **0.017**          |\n|    UMI-GRAT     |     **0.478**      |            0.019            |\n\nThe results show that our method achieves the best attack performance in terms of the E-measure ($E_\\Phi$) and the second-best performance on the weight F-measure $F_\\omega ^\\beta $.  We will add those metrics in the table of our revised paper for better clarification.\n\n> Will increasing the iterative step of generating adversarial perturbation enhance the transferability?\n\nWe conducted comparative experiments under 5 different attacks with iteration times $T_a=20$ and the results are reported in **Table R2** of our submitted PDF file. \n\nThe results indicate that increasing the attack iterations **greatly enhances** the adversarial examples on attacking camouflaged SAM, where the **domain gap from the natural images is minor**. However, For those tasks characterized by a substantial domain gap, increasing the iteration cannot bring performance gain.\n\n> Will the ensemble of different SAMs benefit the UMI-GRAT?\n\nFollowing your valuable suggestion, we explored enhancing adversarial attacks by the ensemble under two types of scenarios:\n\n1. The transferability from pre-trained SAM to Medical-SAM.\n2. The transferability from the pre-trained SAM ViT-B to SAM ViT-H.\n\nWe adhered to the experimental settings outlined in Section 6.1 and employed an ensemble of **SAM ViT-B and SAM ViT-L**. We reported the mDSC for Medical SAM and mIOU for the original SAM. The results are presented in the following tables:\n\n|                     | Medical-SAM(mDSC$\\downarrow$ ) | SAM ViT-H(mIOU$\\downarrow$ ) |\n| :-----------------: | :----------------------------: | :--------------------------: |\n|      UMI-GRAT       |              5.22              |            15.41             |\n| UMI-GRAT + Ensemble |             13.54              |            12.09             |\n\nThe results indicate that the ensemble yields performance gains in general transfer tasks while degrading the performance on the pre-trained model to fine-tuned model transfer task. As the fine-tuned model solely inherits information from its pre-trained one, incorporating uncorrelated gradient information brings unnecessary deviation, thus degrades the performance.\n\n> Expanding the scope of analysis to assess whether this adversarial threat also applies to other large foundation models.\n\n Following your good suggestion, we evaluated the effectiveness of our proposed UMI-GRAT on two new scenarios: \n\n1. **The transferability of MUI-GRAT to other pre-trained methods.** We attacked the **MAE ViT-S and ViT-B that are fine-tuned on Chexpert [2]** using solely **the pre-trained MAE ViT-S** and reported the results in **Table. R3** of our submitted PDF file. \n2. **The effectiveness of MUI-GRAT on the general transfer attack setting.** We attacked the **MAE ViT-B and DenseNet-121** **fine-tuned** **on Chexpert [2]** using **MAE ViT-S fine-tuned** on the same dataset and reported the results in **Table. R4**.\n\nWe utilized models provided by [1] and followed the same experimental setting. We evaluate 8 attack methods on the Chexpert [2] dataset, where the model needs to diagnose five predefined diseases in the chest X-ray images.  We reported the mean Area Under the Curve (mAUC) to evaluate the performance. \n\nThe results in **Table. R3** shows that the proposed **MUI-GART maintains the best method** on attacking both the fine-tuned **MAE-ViT-S and MAE-ViT-B** models. In **Table. R4**, we evaluated the transferability between different ViTs and transferability from the ViT to the CNN.  The experimental results indicate that our MUI-GRAT maintains effectiveness on general transfer tasks, demonstrating its good generalizability.\n\n------\n\nWe do appreciate your constructive feedback. We will add those experiments and analyses mentioned above in the appendix of our revised version.\n\n[1]  Delving into masked autoencoders for multi-label thorax disease classification. In WACV 2023.\n\n[2]  Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In AAAI 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We express our sincere appreciation to all the reviewers for their elaborate and constructive feedback. We summarize our rebuttal as follows:

1. As suggested by reviewer **pDnz**, we conducted the **randomness experiment** and presented the experimental results in **Table R1** of the PDF document attached below.
2. As suggested by reviewer **Qxcd and aXPE**, we discussed the effect of different hyperparameters for different attack algorithms and presented the experimental results in **Table R2** of the PDF document attached below.
3. As suggested by reviewer **Qxcd**, we compared our method with **ANDA and L2T** and presented the experimental results in **Table R3, R4 and R5** of the PDF document attached below.
4. As suggested by reviewers **Qxcd, Q28B, and aXPE**, we evaluated our proposed method on **MAE pre-trained model and on a general transfer adversarial attack task**. The results are presented in **Tables R3 and R4** of the PDF document attached below.
5. As suggested by reviewers **Qxcd and Q28B**, we evaluated our proposed method on the **adversarial-trained Medical SAM** and reported the results in **Table R5** of the PDF document attached below. 
6. As suggested by reviewer **iuQV**, we evaluated the mentioned **model-agnostic method** and presented the results in **Table R5** of the PDF document attached below.

Meta-review: This paper explores the feasibility of adversarial attacks on downstream models fine-tuned from the SAM solely using information from the open-sourced SAM. They introduce a Universal Meta-Initialization and Gradient Robust Adversarial Attack (UMI-GRAT) to enhance the effectiveness of attacks on models fine-tuned on unknown datasets. By extracting intrinsic vulnerabilities from the foundation model and utilizing prior knowledge for adversarial perturbation generation, UMI-GRAT showcases improved transferability in attacking SAMs and their downstream models without accessing task-specific datasets.

On the positive side, the reviewers found the paper to be well-motivated and technically sound. The experimental results are sufficient and the proposed UMI-GRAT is well-supported by theoretical analysis. On the negative side, there are some concerns regarding the randomness, missing comparisons, and generalizability.

The authors have provided detailed responses. After fruitful discussions between reviewers and authors, these concerns were effectively addressed and most reviewers increased their scores. AC thus recommends acceptance.
Decision: Accept (poster)