Transferable Adversarial Attacks on SAM and Its Downstream Models

Authors: song  xia
wenhan  yang
yi  yu
xun  lin
henghui  ding
lingyu  duan
xudong  jiang
medical  datasets
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 09:11:47

Abstract: The utilization of large foundational models has a dilemma: while fine-tuning downstream tasks from them holds promise for making use of the well-generalized knowledge in practical applications, their open accessibility also poses threats of adverse usage.
This paper, for the first time, explores the feasibility of adversarial attacking various downstream models fine-tuned from the segment anything model (SAM), by solely utilizing the information from the open-sourced SAM. 
In contrast to prevailing transfer-based adversarial attacks, we demonstrate the existence of adversarial dangers even without accessing the downstream task and dataset to train a similar surrogate model.
To enhance the effectiveness of the adversarial attack towards models fine-tuned on unknown datasets, we propose a universal meta-initialization (UMI) algorithm to extract the intrinsic vulnerability inherent in the foundation model, which is then utilized as the prior knowledge to guide the generation of adversarial perturbations.
Moreover, by formulating the gradient difference in the attacking process between the open-sourced SAM and its fine-tuned downstream models, we theoretically demonstrate that a deviation occurs in the adversarial update direction by directly maximizing the distance of encoded feature embeddings in the open-sourced SAM.
Consequently, we propose a gradient robust loss that simulates the associated uncertainty with gradient-based noise augmentation to enhance the robustness of generated adversarial examples (AEs) towards this deviation, thus improving the transferability.
Extensive experiments demonstrate the effectiveness of the proposed universal meta-initialized and gradient robust adversarial attack (UMI-GRAT) toward SAMs and their downstream models.
Code is available at https://github.com/xiasong0501/GRAT.

Main paper: 
 introduction: 
 Large foundation models that are trained on a broad scale of data have gained massive success in various applications [6] , such as vision-language chatbot [1] , text-image generation [42, 44, 46] , image-grounded text generation [2, 31] , and anything segmentation [26] . The segment anything model (SAM) [26] , trained on vast amounts of data from the SA-1B dataset, is capable of handling diverse and complex visual tasks. The open accessibility of SAM makes it a promising foundation model, serving as the starting point for fine-tuning analytics models in certain domains and downstream applications, e.g., medical segmentation [61, 54, 39] , 3D object segmentation [9] , camouflaged object segmentation [11] , overhead image segmentation [45] , and high-quality segmentation [25] . However, many studies [5, 18, 57, 38, 27, 55, 58, 66, 65, 59, 48] have highlighted the secure issues of deep learning models towards adversarial attacks. By corrupting the clean input with a finely crafted and Figure 1 : An illustration of UMI-GRAT towards SAM and its downstream tasks. The UMI-GRAT can mislead various downstream models by solely utilizing information from the open-sourced SAM. nearly imperceptible adversarial perturbation, the attacker can mislead the well-trained model at a high success rate, with limited information available (e.g., the surrogate model or limited queries). Consequently, significant concerns arise regarding fine-tuning open-sourced models, as it inevitably leaks critical information on downstream models, increasing their vulnerability to adversarial attacks.Existing adversarial attacks can be roughly categorized into white-box attacks [3, 18] and black-box attacks [56, 23] , based on whether the attacker can fully access the victim model. The pre-requisite of fully accessing the victim model complicates the practical deployment of white-box attacks. Conversely, transfer-based black-box attacks that require less information pose a substantial threat to real-world applications. The prevalent transfer-based black-box adversarial approaches [8, 12, 17, 22, 32, 34, 36, 15, 60] typically suppose strong prior knowledge of the victim model's task and training data, such as a 1000-class classification task in ImageNet, thereby facilitating the training of a similar surrogate model to generate potent adversarial examples (AEs). However, few studies [64, 63] consider a more practical and challenging scenario wherein the attacker is unaware of the victim model's tasks and the associated training data, due to stringent privacy and security protection policies (e.g., datasets containing medical or human facial information). Moreover, the increasing size of large foundation models significantly amplifies the costs of training effective task-specific surrogate models. Thus, a more general and practical security concern is to explore the capability of attackers to mount adversarial attacks on any victim model even without the need of accessing to its downstream tasks-specific datasets to train a closely aligned surrogate model.Given the practical security concerns of utilizing large foundation models, this paper investigates the potential risks associated with fine-tuning the open-sourced SAM on a private and encrypted dataset. We introduce a strong transfer-based adversarial attack called universal meta-initialized and gradient robust adversarial attack (UMI-GRAT), which effectively mislead SAMs and their various fine-tuned models without accessing the downstream task and training dataset, as illustrated in Figure 1 .The contributions of our paper are summarized as follows:• We begin an investigation into a more practical while challenging adversarial attack problem: attacking various SAMs' downstream models by solely utilizing the information from the open-sourced SAMs. We provide the theoretical insights and build the experimental setting and benchmark, aiming to serve as a preliminary exploration for future research. • We propose an offline universal meta-initialization (UMI) algorithm to extract the intrinsic vulnerability inherent in the foundation model, which is utilized as prior knowledge to enhance the effectiveness of adversarial attacks through meta-initialization. • We theoretically formulate that, when using the open-sourced SAM as the surrogate model, a deviation occurs inevitably from the optimal direction of updating the adversary. Correspondingly, we propose a gradient robust loss to mitigate this deviation. • Extensive experiments demonstrate the effectiveness of the proposed UMI-GRAT toward SAMs and its downstream models. Moreover, UMI-GRAT can serve as a plug-and-play strategy to significantly improve current state-of-the-art transfer-based adversarial attacks. 
related work: 
 The adversarial attacks aim to mislead the target (or victim) model by adding a small adversarial perturbation in the clean input. Existing black-box attacks can be broadly categorized into query-based and transfer-based adversarial attacks.The query-based attacks consider the scenarios where the attacker does not have enough information to train a satisfying surrogate model, thus generating adversarial examples by interacting and analyzing the outputs from the victim model. This kind of attack can be divided into score-based query attacks (SQAs) [12, 24, 47] that update the AEs by observing the change of the model's prediction (e.g., the logits or softmax probability) and decision-based query attacks (DQAs) [7, 10] that only rely on the model's top-1 prediction to update the AEs. However, this black-box search is naturally the NP-hard problem, and solving the optimal update strategy is non-differentiable. This makes query-based attack requires thousands of interactions with the victim model, making query-based attacks characterized by low throughput, high latency, and marked conspicuousness to attack real-world deployed systems.The transfer-based adversarial attacks generate the AEs to mislead the victim model based on a similar surrogate model. Existing work mainly focuses on improving the transferability of AEs, which can be categorized into four groups: input-augmentation-based attacks [8, 56, 52] that enhances the effectiveness of generated AEs by augmenting the clean input (e.g., using crop or rotation), optimization-based attacks [13, 35, 51, 62] that utilizes a better optimization strategy to guide the update of AEs, model modification-based attacks [53, 4] or ensemble-based attacks [19, 43, 37, 33] that enhances the AEs by utilizing a more powerful surrogate model, and feature-based attacks [32, 34] that attack the extracted feature in the intermediate layer. However, most of the work makes a strong assumption that the surrogate and victim models are optimized for the same task with identical data distribution, for example, both surrogate and victim models are optimized on the ImageNet dataset to complete the classification task. In real-world deployed systems, due to privacy and security concerns, attackers typically cannot access the training data (e.g., datasets containing private information) or obtain the optimization objectives of the victim model, making training a similar surrogate model exceedingly difficult and unfeasible.To address the challenge of deploying transfer-based black-box attacks without knowing the victim model's task and training dataset, this paper investigates the feasibility of attacking any victim model optimized for unknown tasks and distributions that significantly differ from the open-sourced surrogate model. We provide both theoretical and analytical evidence demonstrating that our proposed method can enhance the transferability and effectiveness of the generated AEs. Moreover, the proposed MUI-GRAT can serve as a plug-and-play adversarial generation strategy to enhance most existing transfer-based adversarial attacks for this challenging task. 
adversarial attacks: 
 Let f be any deep learning model and L be the loss function (e.g.,, the cross-entropy loss) that evaluates the quality of the model's prediction. Let B ϵ (x) = x ′ : ∥x ′ -x∥ p ≤ ϵ be an ℓ p -norm ball centered at the input x, where ϵ is a pre-defined perturbation bound. For each input x, the untargeted adversarial attacks aim to find an adversarial perturbation δ by solving:EQUATIONAn effective solution to Equation 1 is iteratively updating the adversarial perturbation δ based on the gradient of the loss function, for example, the iterative fast sign gradient method (I-FGSM) [28] , which iteratively updates δ by:EQUATIONwhere ∇ calculates the gradient and sign returns the sign (i.e.,-1 or +1). α is a pre-defined step size to update the adversarial perturbation. clip constrains the magnitude of the perturbation by projecting δ into the boundary of the ℓ p -norm ball B ϵ .The SAM consists of three parts: an image encoder f ϕim , a lightweight prompt encoder f ϕpt , and a lightweight mask decoder f ϕ mk . SAM gives the mask prediction based on the image input x and prompt input p, which is expressed as:EQUATIONwhere f ϕim is the image encoder that provides the fundamental understanding by converting natural images into feature embeddings and f ϕpt extracts prompt embeddings. f ϕ mk is the mask decoder that gives the mask prediction by fusing the information from both feature and prompt embeddings. where AT is the adversarial attack strategy and f ϕs is the open-sourced SAM. A solution to that is fine-tuning an optimal surrogate model f ϕ * s that closely aligns with the victim model. However, this approach becomes extremely challenging, when the downstream dataset is inaccessible to the attacker. Alternatively, an effective solution is to design an optimal attack strategy AT * such that:EQUATIONNotably, f ϕs and f ϕτ are optimized on two distinctive distributions D and D τ with losses L and L τ , leading to a significant input-output mapping gap and gradient disparity, such as Cosine_similarity(∇f ϕs (x τ ), ∇f ϕτ (x τ )) ≪ 1. This misalignment critically undermines the effectiveness of current gradient-based adversarial attack strategies.Further analysis on attacking SAMs. The standard operation to deploy SAM on downstream tasks τ involves fine-tuning the image encoder f ϕim to inherit some well-generalized knowledge. Concurrently, the lightweight prompt encoder f ϕpt and the mask decoder f ϕ mk are trained from scratch to better accommodate the task. Considering the pivotal importance of feature embeddings and the substantial variation caused by full retraining, an intuitive approach to generate effective adversarial perturbation δ s is utilizing the common information in f ϕim to achieve:EQUATIONwhere ϕ τ im denotes the updated parameters for the downstream model's image encoder after finetuning. Unless otherwise specified, we denote ϕ im as ϕ in our subsequent content and take the general SAM image encoder f ϕ as the surrogate model f ϕs . We aim to generate the transfer-based AEs to attack any fine-tuned image encoder f ϕτ on task τ , thereby misleading the entire prediction.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
extract the intrinsic vulnerability via universal meta initialization: 
 Considering the great variation brought by fine-tuning the model on a new task τ , we aim to extract the intrinsic vulnerability of the foundational model that remains invariant after fine-tuning. Subsequently, this extracted vulnerability is leveraged as prior knowledge to initialize and enhance the adversarial attack AT * . Inspired by the universal adversarial perturbation [40] that maintains effectiveness across various inputs, we propose the universal meta initialization (UMI) algorithm, which optimizes the initialization of adversarial perturbation to ensure both effectiveness and fast adaptability by meta-learning [41, 16] . We define the universal and meta-initialized perturbation δ as follows. Definition 2 (Universal and meta-initialized perturbation δ). Given the foundation model f ϕ and its fine-tuned models f ϕτ on downstream tasks τ , the universal and meta-initialized perturbation δ that extracts the intrinsic vulnerability ensures both effectiveness and fast adaptability, which are:1. Effectiveness (universal adversarial perturbation): δ extracts the intrinsic vulnerability in the foundation model, which can mislead the f ϕ successfully on most natural inputs x, which is:EQUATIONwhere λ is a pre-defined threshold for one successful attack, and I {•} is the indicator function that returns 1 if the inside condition is satisfied, else 0.2. Fast adaptability (meta-initialization): for any downstream task τ with the corresponding private downstream dataset D τ and model f ϕτ , the attackers can maximize the loss L on downstream model f ϕτ by updating the initialization δ via the surrogate model f ϕ in t steps, which is:EQUATIONU t is the operation to update δ for t steps based on input x τ , which is defined as:EQUATIONwhere∆δ j+1 = α τ • sign ∇L f ϕ (x τ ) , f ϕ x τ + U j (δ)if we attack the surrogate model f ϕ and update the adversarial perturbation based on the first-order gradient.Generally, Equation 8 aims to extract the intrinsic vulnerability inherent in the model, which remains effective towards the input variation. Equation 9guarantees that utilizing the perturbation δ as the initialization can rapidly threaten strong adversarial attacks for any downstream model. However, in Equation 9, D τ and f ϕτ are unknown if the attacker is precluded from the downstream dataset. An approximated solution to that involves using the dataset D that covers the distribution of most natural inputs and a general model f ϕ that can approximately represent the expectation of f ϕτ , which is:EQUATIONTo optimize the above two objectives simultaneously, our learning aims to move towards the direction that maximizes the inner product of the gradients computed on both objectives. We utilize a first-order meta-learning algorithm called Reptile [41] , which defines the noise δ update in each round as:EQUATION)where η is the update step size, and δµi = U t µi (δ) is the updated perturbation on objective µ i after optimizing t iterations. Here we set n = 2, corresponding to the two objectives in Equation 8 and 11. For µ 1 that aims to optimize Equation 11, we set t = 5 and U t µ1 (δ) the same as U t defined in Equation 10. For µ 2 that aims to optimize Equation 8, we set U t µ2 (δ) as:EQUATIONEquation 13 aims to find a minimal update ∆δ nearby δ to mislead the model f ϕ . This can be achieved by using enough iterations t and a small but gradually increased norm-ball boundary ϵ.While finding an effective UMI requires a substantial number of inputs and iterations, this process can be conducted fully offline, thus not hindering real-time adversarial attacks.Besides the utilization of intrinsic weakness inherent in the foundation model to enhance the adversarial attack AT * , another method involves generating the adversarial perturbation that sustains robustness against the deviation arising from updates through a surrogate model that exhibits significant gradient disparity compared to the fine-tuned downstream model. . For the victim model f ϕτ with updated parameter ∆ϕ τ , the modules are denoted as{f 1 ϕ 1 +∆ϕ 1 τ , . . . , f m ϕ m +∆ϕ m τ }.The output y τ of each module with the input x τ is denoted as:EQUATIONwhere y 0 τ = x τ and h i ∆ϕ i τ is a hypothetical function that characterizes the update brought by ∆ϕ i τ . Proposition 1 (Deviation in updating adversarial perturbation). Let f ϕτ be the victim model finetuned on any unknown task τ , the deviation in the direction of updating the adversarial perturbation by maximizing a predefined loss L in the surrogate model f ϕ can be formulated as:EQUATIONIn Equation 15, ∆δτ ← ∇L(y m τ ) • m i=1 ∇f i ϕ i y i-1 τ + ∇h i ∆ϕ i τ y i-∇f i ϕ i y i-1 τis the update of adversarial perturbation by maximizing the pre-defined L on feature embeddings of the surrogate model. Proposition 1 establishes by simultaneously considering the white-box scenarios for both surrogate and victim models to derive the gradient using the chain rule. It claims that h i ∆ϕ i τ leads to a great deviation in updating the adversarial perturbation δ s towards the optimal solution in attacking victim model if directly maximizing the feature embedding distance of the surrogate model, thus degrading the effectiveness of the generated AEs.Mitigate the deviation caused by gradient disparity. To enhance effectiveness of the generated adversarial perturbation δ s under the hypothetical update ∇h ∆ϕτ , we propose a gradient robust loss L GR , that aims to mitigate the deviation in Equation 15 by gradient-based noise augmentation. Denote N (ε; µ, σ 2 I) as the isotropic Gaussian noise with mean µ and variance σ 2 , which has the same dimension as ∇h ∆ϕτ . The robust update of adversarial perturbation ∆δ * s on the surrogate model based on the noised augmented gradient is:EQUATION)By ignoring higher-order uncertain terms in Equation 16, we can simplify it as:EQUATION)Following the adversarial perturbation update guidance in Equation 17, the corresponding gradient robust loss L GR is defined as : for each x in D do 8:EQUATIONupdate δµ 1 by U t µ 1 ( δµ 1 ) in Equation 109:update δµ 2 by U t µ 2 ( δµ 2 ) in Equation 1310: update δ = δ + η • 1 where y i adv is the extracted adversarial feature by layer i and ∥•∥ p is a predefined norm-based measure that is decided by the L (e.g., p = 1 for L1 loss).Discussion with the intermediate-level attacks. The intermediate-level attacks (ILAs) [34, 22, 32 ] also aim to maximize the dissimilarity of feature embeddings between the clean and adversarial inputs. However, the main concern in ILAs is how to find a directional vector v to guide the update direction of f ϕ (x) -f ϕ (x adv ), thus assuring that this featurewised dissimilarity can maximally mislead the final prediction. Different from that, our L GR considers the problem one step further: given an optimal directional vector v, how to generate the adversarial perturbation that is roust towards the potential gradient variation in the victim model, thus maximally misleading the victim model along the direction of v. Our core idea is hence in parallel with ILAs and can be well combined with them to enhance the attack's effectiveness. We experimentally analyze and visualize the cosine similarity of the generated perturbations on CTscan images by white-box attacking open-sourced SAM and medical SAM using MI-FGSM [13] , ILPD [34] , and our gradient robust attack in Figure 3 , illustrating that the proposed L GR effectively reduce the deviation caused by gradient variation and achieves a much transferability. 
implementation of the proposed mui-grat: 
 The detailed implementation of our proposed MUI-GRAT is illustrated in Algorithm 1 and Figure 2 .Our UMI-GRAT method consists of two stages. The initial stage involves the offline learning of a universal meta-initialization (UMI), which aims to find the intrinsic vulnerability inherent in the foundation model. In the subsequent stage, we utilize the learned UMI as the prior knowledge to enhance the gradient-variation robust adversarial attack.We utilize the image encoder from Vit-H-based SAM as the general foundation model f ϕ for generating the UMI. The natural image dataset D consists of a total of 20,000 images, with 10,000 from ImageNet and 10,000 from the SA-1B dataset. We set the meta iterations T m as 7 and the universal step size η as 1. The function Uni_Meta_Ini returns the learned UMI δ that can be used to enhance the generation of subsequent input-specific adversarial perturbation.In GR_Attack, we first adapt the calculated UMI δ with the task-specific image x τ by one-step update using FGSM [18] with the step size α adp = 4. Assume that ỹ represents the mean of the feature embedding of natural images calculated by f ϕ , our L adp is defined as:EQUATIONEquation 19 aims to minimize the domain difference between vx τ and the natural images by a specific generated perturbation. The UMI δ is then added by δ adp and utilized to initialize δ * s . We set T a to 10, and update the adversarial perturbation δ * s by maximizing the gradient robust loss L GR .  
experiment setup: 
 Evaluation details: we conduct experiments on SAMs' downstream models including, medical image segmentation SAM [61] , shadow segmentation SAM [11] , and camouflaged object segmentation SAM [11] . The datasets include: the synapse multi-organ segmentation dataset [29] that contains 3779 abdominal CT scans with 13 types of organs annotated, the ISTD dataset [49] that contains 1870 image triplets of shadow images, the COD10K dataset [14] that contains 5066 camouflaged object images, the CHAMELEON dataset that contains 76 camouflaged images, and the CAMO dataset [30] , that contains 1500 camouflaged object images. We report the mean dice similarity score (mDSC) and mean Hausdorff distance (mHD) for evaluating medical segmentation, mean absolute error (MAE) and structural similarity (S α ) for camouflaged object segmentation, and the bit error rate (BER) for shadow segmentation. In medical SAM, the image encoder is based on SAM-Vit-B and fine-tuned with LoRA [21] . In shadow segmentation and camouflaged object segmentation SAM, the image encoders are based on SAM-Vit-H and fine-tuned with the adapter [20] . The decoders in those models are all fully retrained.Compared methods: We mainly compare and evaluate our method with current transfer-based adversarial attacks including gradient-based attacks called MI-FGSM [13] and PGN [17] , inputaugmentation based attacks called DMI-FGSM [56] and BSR [50] , and intermediate-level feature based attack called ILPD [34] .Implementation details: we use the MI-FGSM [13] as our basic attack method. For all methods reported, we set the attack update iterations T a as 10, with the l ∞ bound ϵ = 10 and the step size α = 2. For our UMI, we set the meta iterations T m = 7, universal step size η = 1. For PGN and BSR, we set the number of examples as 8 for efficiency.We report our main results in Table 1 . The first row of the data presents the model performance with clean inputs. The second part of the data shows the model performance under different adversarial attacks, where the data with the strongest attack is bold. The results demonstrate that the adversarial examples generated by our proposed MUI-GRAT are more effective and generalizable than others, consistently posing significant adversarial threats across various downstream models. In medical segmentation and shadow segmentation tasks that share a great difference with the natural segmentation tasks, our proposed MUI-GRAT greatly surpasses others (e.g., MUI-GRAT reduces the mDSC from 81.88 to 5.22 while the previous best is 25.73.). This demonstrates the exceptional effectiveness of our proposed MUI-GRAT when the attacker lacks information about the victim model, thereby generating AEs using a surrogate model distinct from the victim model. In the camouflaged object segmentation task where the data closely resembles natural images, all methods exhibit strong transferability. Our MUI-GRAT achieves the best performance on the COD10K and CHAME datasets and performs comparably to the SOTA method on the CAMO dataset. In the last part of Table 1 , we analyze the performance of our proposed MUI-GRAT when combined with other SOTA transfer-based attacks. Notably, all methods achieve an overall performance gain after combining with ours. Though a slight performance drop occurs when attacking the CAMO dataset, combining the MUI-GRAT brings great performance gain on attacking other tasks. This demonstrates the versatility of our MUI-GRAT, which can be seamlessly applied in a plug-and-play manner to bolster existing transfer-based attacks. We report the experiment results for attacking open-sourced SAMs in Appendix A.2 and analyze the real-time attack efficiency for each method in Appendix A.3.The transferability property across diverse models ensures the effectiveness of the adversarial example to mislead an unknown victim model. As discussed in Section 4.1, an intuitive objective for attacking SAM's downstream models is to maximize the dissimilarity of feature embedding extracted from the clean input x and adversarial input x adv . Based on this, to numerically evaluate the improvement of transferability brought by MUI-GRAT, we present the l 2 distance of clean and adversarial feature embeddings attacked by MI-FGSM and ours and then analyze the distance gap between the surrogate and victim models. We propose that a viable transferable attack methodology should induce a substantial feature distance ∆f in the victim model, while simultaneously ensuring minimal performance degradation ϵ during the transition from the surrogate to the victim model.We show this comparison result in Figure 4 , where we randomly pick a subset of inputs and show the distance of feature embedding for each clean-adversarial input pair. The average distance gap between the surrogate and victim models indicates the overall transferability of the attack method. In the medical and shadow segmentation SAM, where the data and task are distinct from the original SAM, we find a great performance drop for MI-FGSM when transferred from the surrogate model to the victim model. Though the adversarial examples generated by MI-FGSM induce a large feature distance in the surrogate model, their effect on the victim model is relatively minor. Conversely, the adversaries generated by MUI-GRAT maintain much better transferability, suffering from a small performance drop when transferred from the surrogate model to the victim model. In the camouflage object segmentation task, where the data are natural images and the segmentation objective is similar to the original SAM, both attack algorithms show good transferability (nearly no performance drop when transferred from the surrogate model to the victim models). Our MUI-GRAT shows better transferability with nearly no performance drop.In this section, we explore the contribution of the proposed MUI and GRAT by integrating them with MI-FGSM and PGN attacks. The ablation results are shown in Table 2 . In scenarios where the task and dataset distributions of surrogate and victim models differ markedly (e.g., medical image segmentation and natural image segmentation), we observe that the GR loss significantly enhances effectiveness. Meanwhile, across all scenarios, the proposed MUI consistently contributes to enhancing the adversarial attacks. Particularly in camouflaged object segmentation when the surrogate and victim models exhibit close similarities, the MUI yields substantial benefits. This observation aligns with our analysis in Section 4. By assuming a hypothetical update h ∆ϕτ in the victim model, the proposed gradient robust loss greatly enhances the effectiveness of the generated AEs towards the gradient variation, thus benefiting more for medical and shadow segmentation tasks. Moreover, the MUI aims to find the intrinsic vulnerability inherent in the basic foundation model through a broad general dataset, which is then provided as the prior knowledge for generating a more effective adversary. Therefore, in scenarios where the victim model inherits substantial information from the surrogate model, this prior knowledge becomes increasingly reliable and effective. 
conclusion: 
 The security of utilizing large foundation models is a critical issue for deploying them in real-world applications. This paper, for the first time, considers a more challenging and practical attack scenario where the attacker executes a potent adversarial attack on SAM-based downstream models without prior knowledge of the task and data distribution. To achieve that, we propose a universal metainitialization (UMI) algorithm to uncover the intrinsic vulnerabilities inherent in the foundation model. Moreover, by theoretically formulating the adversarial update deviation during the attacking process between the open-sourced SAM and its fine-tuned downstream models, we propose a gradient robust loss that simulates the corresponding uncertainty with gradient-based noise augmentation and analytically demonstrates that the proposed method effectively enhances the transferability.Extensive experiments validate the effectiveness of the proposed UMI-GRAT toward SAM and its downstream tasks, highlighting the vulnerabilities and potential security risks of the direct utilization and fine-tuning of open-sourced large foundation models. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This research does not have this kind of risk. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We have cited the original paper that produced the code package or dataset. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: A.1 Randomness test We evaluated 10 attack methods presented in our paper over 5 random seed runs on the subset of SAM's downstream tasks and reported the mean performance with its standard deviation. We use the same experimental setting provided in Section 6.1. The results indicate that the randomness is small and similar among all attacking methods. The uncertainty level of UMI-GRAT in mean Hausdorff Distance (mHD) is marginally higher compared to the other methods. This can account for the higher mHD value achieved by the UMI-GRAT. We report the mean average precision (mAP) and mean intersection-over-union (mIOU) metrics to evaluate the performance of attacking open-sourced SAMs. We compare the performance of MI-FGSM and ours MUI-GRAT. The implementation of these attacks adheres to the details described in Section 6.1. We randomly select 500 images from the SA-1B dataset and evaluate the performance of SAMs under the 'AutomaticMaskGenerator' mode. We white-box attacks SAM-Vit-B, SAM-Vit-L, and SAM-Vit-H models. Meanwhile, we also discuss the black-box transferability of generated AEs across different models. The results are shown in We analyze the real-time attack efficiency of the methods mentioned above. We report the average time required for generating one AE when the input resolution is 512 × 512 on the SAM-Vit-B model using one RTX 4090 GPU. The results are shown in Table A . 5 .We find that our proposed MUI-GRAT achieves secondhigh efficiency when compared with others. The ILPD requires extra attack iterations (e.g., 10-step MI-FGSM) to find a directional guide vector v. The input augmentation-based attack methods, such as BSR and DMI-FGSM, require multiple samples in each iteration to do the augmentation and the PGN also requires multiple samples to obtain a stable gradient direction. However, our method utilizes an offline generated MUI and only conducts one-time gradient augmentation in each iteration, thus achieving a much higher efficiency than others.We run our experiments for attacking the medical segmentation model using one RTX 4090 GPU with 24 GB memory. We run the rest of the experiments using one RTX A6000 GPU with 48 GB memory. The increasing reliance on large foundation models in various real-world applications amplifies the critical importance of ensuring their secure utilization. This paper mainly discusses the potential risks of adversarial threats associated with the direct utilization and fine-tuning of the open-sourced model even on private and encrypted datasets. To accomplish that, we begin an investigation into a more practical while challenging adversarial attack problem: attacking various SAM's downstream models by solely utilizing the information from the open-sourced SAM. We then provide the theoretical insights and build the experimental setting and benchmark, aiming to serve as a preliminary exploration for future research in this area. Experimentally, we validate the vulnerability of SAM and its downstream models under the proposed MUI-GRAT, indicating the security risk inherent in the direct utilization and fine-tuning of open-sourced large foundation models, thus highlighting the urgent need for robust defense mechanisms to protect these models from adversarial threats.The limitations of our paper are:• The proposed UMI-GRAT is not contingent upon a prior regarding the model's architecture, suggesting its potential applicability across various model paradigms. However, the experiments only tested MUI-GRAT on the prevalent SAMs and their downstream models. The capability of UMI-GRAT to pose a threat to other large foundation models remains a topic for further exploration. • While this paper highlights the risk of direct utilization of SAM and fine-tuning it on the downstream task, this paper does not provide and validate an effective solution for this secure concern.NeurIPS Paper Checklist • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: This paper explores the feasibility of adversarial attacks on downstream models fine-tuned from the SAM solely using information from the open-sourced SAM. They introduce a Universal Meta-Initialization and Gradient Robust Adversarial Attack (UMI-GRAT) to enhance the effectiveness of attacks on models fine-tuned on unknown datasets. By extracting intrinsic vulnerabilities from the foundation model and utilizing prior knowledge for adversarial perturbation generation, UMI-GRAT showcases improved transferability in attacking SAMs and their downstream models without accessing task-specific datasets.

On the positive side, the reviewers found the paper to be well-motivated and technically sound. The experimental results are sufficient and the proposed UMI-GRAT is well-supported by theoretical analysis. On the negative side, there are some concerns regarding the randomness, missing comparisons, and generalizability.

The authors have provided detailed responses. After fruitful discussions between reviewers and authors, these concerns were effectively addressed and most reviewers increased their scores. AC thus recommends acceptance.

Reviews: review: paper_summary: paper_summary: In this paper, the authors propose an adversarial attack method that can contaminate downstream tasks from the perspective of adversarial transferability. They address the problem that SAM models do not have similar optimisation routes after fine-tuning for different downstream tasks by designing universal meta initialization. In this paper, the authors address the problem that SAM models do not have similar optimisation routes after fine-tuning for different downstream tasks by designing UMI noise. The authors introduce the idea of meta-learning to allow their algorithm to quickly adapt to different situations, i.e., downstream tasks.
strength_weakness: strength_weakness: 1. The readability of the Methodology section of this article is somewhat poor. The authors define the problem to be solved through the form of propositions. Similarly, if the authors could summarise the formulas as Theorem and put the proof process (both formulas and reasoning) specifically in the supplementary material, it would make the article more coherent.



2. The randomness of the experimental results is unknown. I understand that due to the larger computational effort, it is not practical to report error lines on all major experiments. But it would be better for the authors to report a set of randomness on a smaller dataset and simpler settings, which will influence the reviewers' opinion of the results of this method.
questions: questions: Two questions listed, see Weaknesses for details. Note that if the authors can demonstrate the randomness of their algorithms, that will help to get a higher rating.
limitations: limitations: The authors correctly list the implications of their work for the use of large models such as SAM in downstream tasks.
paper_summary: paper_summary: This paper investigates the vulnerability of Segment Anything Model (SAM) and its downstream models to transferable adversarial attacks. The authors propose a novel attack method called Universal Meta-Initialized and Gradient Robust Adversarial attack (UMI-GRAT) that leverages the open-sourced SAM to generate adversarial examples effective against fine-tuned downstream models, even without access to the downstream task or dataset.
strength_weakness: strength_weakness: See Questions.
questions: questions: 1. How does the performance of UMI-GRAT vary with different choices of hyperparameters, such as the perturbation bound ε and the number of iterations in UMI and LGR. Especially, line 278 mentions that the perturbation bound is 10, which is a bit too large.
2. According to the latest benchmark **[R1]**, baselines used in the paper are not SOTA methods. How does it compare with NCS **[R2]**, ANDA **[R3]**, DeCowA **[R4]** and L2T **[R5]**?
3. Is the proposed method a universal transfer attack method? Although this question is mentioned on line 548, can the performance of UMI-GRAT and SOTA be compared under a general transfer attack test setting?
4. UMI-GRAT is a gradient-based attack method. How does the proposed method perform when the model has a certain robustness (such as adversarial training)?

---
**[R1]** Devling into Adversarial Transferability on Image Classification: A Review, Benchmark and Evaluation.   
**[R2]** Enhancing Adversarial Transferability Through Neighborhood Conditional Sampling.   
**[R3]** Strong Transferable Adversarial Attacks via Ensembled Asymptotically Normal Distribution Learning. CVPR. 2024.   
**[R4]** Boosting Adversarial Transferability across Model Genus by Deformation-Constrained Warping.   
**[R5]** Learning to Transform Dynamically for Better Adversarial Transferability
limitations: limitations: Limitations are discussed in the Appendix.
paper_summary: paper_summary: This paper proposes an adversarial attack against fine-tuned derivatives to a publicly available foundation model, such as the Segment Anything Model (SAM). In the proposed threat model, attackers can potentially manipulate these downstream models even without knowing the specific task or data they are used for. Under this threat model, proposes a new attack method called UMI-GRAT (Universal Meta-initialized and Gradient Robust Adversarial Attack). Through a bi-level optimization procedure, this method leverages the information from the open-source SAM to create adversarial examples that can fool mislead the original SAM and its fine-tuned versions. Finally, this paper demonstrates the effectiveness of the proposed UMI-GRAT attack against SAM through extensive experiments.
strength_weakness: strength_weakness: 1. The effectiveness of the proposed attack is only demonstrated by attacking the SAM model. However, more experiment settings (e.g. against pretrained MAE models) are warranted to demonstrate the generalizability of the proposed attack.
questions: questions: 1. How does the domain gap between the natural image dataset used to obtain the universal adversarial trigger and the downstream dataset influence the effectiveness of the attack?
2. How effective is the proposed method against adaptive defense? For example, if the downstream victim model has gone through adversarial training, how effective would the adversarial trigger obtained on the unguarded pretrained SAM be against the guarded victim model?
limitations: limitations: The authors acknowledge the limitations of this work in the appendix. They candidly acknowledge the limitations in evaluations as the proposed attack is only evaluated against SAM. I appreciate the authors openly acknowledging this limitation.
paper_summary: paper_summary: In this paper, the authors present a new approach for adversarial attacks on Segment Anything Model (SAM)-based downstream models, addressing the challenge of attacking without prior knowledge of the downstream task or data distribution. Their key contribution is a universal meta initialization-based algorithm that exposes inherent vulnerabilities in the foundation model. The authors also introduce a gradient robust loss, which simulates uncertainty through gradient-based noise augmentation. This loss is derived from a theoretical formulation of adversarial update deviation between the open-sourced SAM and its fine-tuned downstream models. The authors provide an analytical demonstration of how their proposed method enhances attack transferability. The effectiveness of their approach is thoroughly validated through comprehensive experiments.
strength_weakness: strength_weakness: 1 - My major concern is related to the novelty of the proposed approach. Although I agree that this is the first work in the context of SAMs, the main components, such as downstream agnostic adversarial examples and meta learning-based fast initialization, have already been proposed in the literature.

2 - The authors, in line 45, briefly highlight downstream agnostic examples in just one line. They should clarify in the related work section how their work is different from references 55 and 56 of the main paper, beyond just applying it to SAM. Similarly, another related work that the authors missed is [1] (given below), in which the generated adversarial examples are agnostic to downstream tasks.

3 - Similarly, the authors did not mention any work related to meta-learning-based adversarial examples in the paper. There are multiple works that use meta-learning to craft universal adversarial examples, such as [1, 2] below. The authors use these meta-learning-based methods for initialization of adversarial examples, but this has already been explored in [3] below. The authors should mention these meta-learning-based approaches in their paper and discuss how their method is different from these approaches, beyond just the application to SAMs.

4 - It is not clear to me when the authors claim in line 8 that they are attacking "without accessing the downstream task." What is the task here? Is it not the segmentation task? In [1], their task-agnostic adversarial examples are effective against classification, detection, and segmentation. Since the downstream task here is segmentation-based, is it not obvious what the task is? Please clarify this.

5 - The authors should include some specific aspects of SAM to make their attack more unique. Currently, they are utilizing the SAM image encoder, which, in my opinion, is not much different from the previous works listed below.

6 - For experiments, why have the authors compared their method with intermediate-level feature-based approaches? They should also compare it with different downstream agnostic adversarial approaches as listed below.

7 - In Equation 8, how did the authors choose the threshold lambda? 

[1] A Self-supervised Approach for Adversarial Robustness (CVPR-2020)

[2] Learning to Generate Image Source-Agnostic Universal Adversarial Perturbations (IJCAI22)

[3] Meta Adversarial Perturbations (AAI2022-Workshop)

[4] Adversarial Initialization with Universal Adversarial Perturbation: A New Approach to Fast Adversarial Training
questions: questions: Please see the weakness section. While the paper presents an approach to attacking SAM-based downstream models, it largely combines existing methods rather than introducing new techniques. The current strategy, though effective, does not fully exploit SAM's unique architecture.
limitations: limitations: Yes.
paper_summary: paper_summary: This work discusses an interesting security issue of deploying a model fine-tuned on a large foundational model in private downstream applications. It proposes a universal meta-initialized and gradient robust adversarial attack (UMI-GRAT) to break the powerful SAM and its various downstream models, without requiring prior knowledge of the specific downstream task and data distribution. The author explores the challenges associated with threating transfer-based adversarial attack without the task-related prior knowledge and provides the theoretical insights on the deviation in updating the adversarial perturbation when using the open-sourced model as the surrogate model. An extensive evaluation of UMI-GRAT's performance, transferability, and efficiency was conducted across five datasets and three different downstream tasks (medical image segmentation, shadow segmentation and camouflaged object segmentation), demonstrating the high effectiveness of the UMI-GRAT approach.
strength_weakness: strength_weakness: 1.	It’s recommended to give more comprehensive analysis of the UMI noise, including the size of the natural image dataset and the effect of various hyperparameters.
2.	There are more metrics such as $E_\phi$, $F_\beta^\omega$ in the camouflaged object detection task. It would be beneficial if the author could provide further data pertaining to these evaluation metrics to enrich the analysis.
questions: questions: 1.	In Figure 3, the peak cosine similarity of the generated adversarial perturbation is observed between the 20th and 30th iteration. So will increasing the iterative step of generating adversarial perturbation enhance the transferability? 
2.	Model ensemble is an effective method to enhance the adversarial attacks’ transferability. Will the ensemble of different SAMs benefit the UMI-GRAT?
limitations: limitations: The experimental results are all based on SAMs and their downstream models. It will provide more valuable insights if expanding the scope of analysis to assess whether this adversarial threat also applies to other large foundation models.

Comments: title: Response to Reviewer iuQV
Response to the Reviewer pDnZ
Response to the Authors
Response to reviewer aXPE
Response to Reviewer Q28B
Comments Adequately Addressed
Response to the Authors
The missing PDF
Thanks for Your Response & Missing PDF
comment: Thank you for your positive feedback. We are glad that our responses address your concerns. In our revised version, we will properly cite all the mentioned papers [1,2,3,4]. All codes will be open-sourced for reproducibility.

We are always available and eager to address any further questions you may have during our discussion.
The authors have addressed most of my concerns. While this is the first work in the context of SAM, components like meta-learning-based adversarial examples already exist in the literature and should be properly credited. Additionally, related work [1] should be properly cited. I have raised the score and hope the authors will open-source their code for reproducibility.
Dear Reviewer pDnZ,

Thank you so much for your positive and constructive feedback, which is very helpful and makes our paper stronger!

We are glad that our responses address your concern. We are always available and eager to address any additional questions you might have during our discussion.

Best regards,

The Authors
It looks like the randomness of most of the data is controlled. The authors also promised to fix the mentioned errors in the next draft. Therefore I decided to raise my score.
Dear Reviewer aXPE,

Thank you so much for your positive and constructive feedback, which is very helpful and makes our paper stronger!

We are glad that our responses address your concern. We are always available and eager to address any additional questions you might have during our discussion.

Best regards,

The Authors
Dear Reviewer Q28B,

Thank you so much for your positive and constructive feedback, which is very helpful and makes our paper stronger!

We are glad that our responses address your concern. We are always available and eager to address any additional questions you might have during our discussion.

Best regards,

The Authors
Thank you for the clarification! I can now see your follow-up PDF. I think my comments are adequately addressed, so I raise my score to seven.
After carefully reviewing the comments from the other reviewers and the author's rebuttal, all of my concerns have been adequately addressed. Therefore, I decide to raise my score.
Dear Reviewer,

Thank you so much for your follow-up. The PDF file is attached to the overall Author Rebuttal. However, it seems that there is a bug in the OpenReview website that makes this rebuttal invisible to reviewers. We believe that NeurIPS will address this issue soon, and you can find the submitted PDF in the Author Rebuttal section.

Best regards,
The Authors
Thank you for your detailed responses and follow-up experiments! However, I couldn't find the PDF file you submitted. If you could provide me with a pointer to the revised PDF, I will make sure to go over it in the upcoming days. Thank you!