Generate Universal Adversarial Perturbations for Few-Shot Learning
Authors: Yiman Hu
Yixiong Zou
Ruixuan Li
Yuhua Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Adversarial Attacks
Universal Adversarial Perturbations
Few-Shot Learning
Primary area: machine_vision
TL;DR: We find the ineffectiveness in generating UAP for open-set scenarios such as Few-Shot Learning. Through in-depth analysis, we point out the two shifts and address them to finally achieve a unified attacking framework.

Abstract: Deep networks are known to be vulnerable to adversarial examples which are deliberately designed to mislead the trained model by introducing imperceptible perturbations to input samples. Compared to traditional perturbations crafted specifically for each data point, Universal Adversarial Perturbations (UAPs) are input-agnostic and shown to be more practical in the real world. However, UAPs are typically generated in a close-set scenario that shares the same classification task during the training and testing phases. This paper demonstrates the ineffectiveness of traditional UAPs in open-set scenarios like Few-Shot Learning (FSL). Through analysis, we identify two primary challenges that hinder the attacking process: the task shift and the semantic shift. To enhance the transferability of UAPs in FSL, we propose a unifying attacking framework addressing these two shifts. The task shift is addressed by aligning proxy tasks to the downstream tasks, while the semantic shift is handled by leveraging the generalizability of pre-trained encoders.The proposed Few-Shot Attacking FrameWork, denoted as FSAFW, can effectively generate UAPs across various FSL training paradigms and different downstream tasks. Our approach not only sets a new standard for state-of-the-art works but also significantly enhances attack performance, exceeding the baseline method by over 16\%.

Generate Universal Adversarial Perturbations for Few-Shot Learning
Yiman Hu
Yixiong Zou" Ruixuan Li Yuhua Li
School of Computer Science and Technology, Huazhong University of Science and Technology (imane, yixiongz, rxli, idcliyuhua) Chust. edu.
Abstract
Deep networks are known to vulnerable to adversarial examples which are deliberately designed to mislead the trained model by introducing imperceptible perturbations to input samples. Compared to traditional perturbations crafted specifically for each data point, Universal Adversarial Perturbations (UAPs) are input-agnostic and shown to be more practical in the real world. However, UAPs are typically generated in close-set scenario that shares the same classification task during the training and testing phases. This paper demonstrates the ineffectiveness of traditional UAPs in open-set scenarios like Few-Shot Learning (FSL). Through analysis, we identify two primary challenges that hinder the attacking process: the task shift and the semantic shift. To enhance the transferability of UAPs in FSL, we propose unifying attacking framework addressing these two shifts. The task shift is addressed by aligning proxy tasks to the downstream tasks, while the semantic shift is handled by leveraging the generalizability of pre-trained encoders. The proposed Few-Shot Attacking Frame denoted as FSAFW, can effectively generate UAPs across various FSL training paradigms and different downstream tasks. Our approach not only sets new standard for state-of- sotte-of-the-se of-the-art works but also significantly enhances attack performance, exceeding the baseline method by over 16%.
Introduction
Deep neural networks[16] 12 have made significant advancements variety of computer vision tasks. Nowadays, there is growing trend of pre- pre-training model that achieves strong generalization capabilities and subsequently fine-tuning it for different unseen downstream tasks. A promising method to handle this open set problem is Few-Shot Learning (FSL), which learns model that rapidly adapt to unseen tasks with only a limited number of samples.
which deliberately designed to deceive a trained model by introducing imperceptible perturbations to the input samples. Given the widespread adoption of the pre-training and fine-tuning paradigm, it is crucial to acknowledge and address the security concerns associated with such approaches.
Mcanwhile, deep networks have shown to be vulnerable adversarial attacks 41 14 5
For more applicable way to attack various downstream tasks, we focus on Universal Adversarial Perturbations (UAPs) 26 This kind of perturbation can applied to images once generated, eliminating the need for crafting image-dependent perturbations for each task.
However, current approaches for generating UAPs have primarily concentrated close-set scenar- ios (26) 15 | (3), where the classification tasks for both training and finetuning are essentially identical. Despite achieving high Attack Success Rate (ASR) in their respective settings, our research reveals
*Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Arrocerr
Base Chaselfe
Uhe Duraut
Task
Somantic Shift
Encnder
Tack
Novel Cinnsifer
Tisk
Datacii
Sunt 1oho
S40p Sclee
Figure 1: (Left) A decline in the Attack Success Rate (ASR) can be observed when conventional UAPs are applied to the FSL scenario. The yellow bar represents performance in traditional scenarios, while the other three indicate performance in different FSL training paradigms. For the metric-based paradigm, traditional methods even fail to generate UAPs. The detailed setting can be found in Section 4.11 (Right). Solid arrows represent the pre-training phase, while dotted arrows depict the testing phase. The attacker can only manipulate the pre-trained model, which suffers from the semantic shift of different datasets and the task shift from base to novel tasks.
that in FSL scenarios these techniques are ineffective, and in some cases, incapable of crafting UAPs, as shown in the left panel of Fig. [
To better understand how to effectively attack the FSL tasks, we conducted a comprehensive evaluation and analysis of the challenges involved, namely task shift and semantic shift, depicted in the right panel of Fig. [ To further fill up these two shifts and achieve a more generalizable UAP, we establish baseline attacking framework as an initial point and progressively improve upon it. To address the task shift, we emphasize the significance of learning task bias for UAP, and handle this shift by constructing proxy tasks. To tackle the semantic shift, we demonstrate the effectiveness brought by the encoder's generalizability and eliminate the need for fine-tuning in the process of UAP generation.
By systematically addressing the shifts, successfully generate the single perturbation only based on the parameters of the pre-trained encoder, without any prior knowledge the pre-traine pre-training dataset or downstream tasks. The ASRs of universal perturbation surpass others by 16.49% for 5-way 1-shot tasks and 17.27% % for 5-way 5-shot tasks. Additionally, our attacking framework(FSAFFV unifies the generation of UAPs in various FSL. training paradigms, including finetuning- based [TO 47 7
meta-based 13 35 39]. and metric-based 43 36 50) 20 approaches, yielding state-of-the-aa results
in all cases. We summarize our main contributions follows:
We propose a new standard for the study of UAP in FSL scenarios, which highlights the limitations of traditional UAPs in the context of FSL.
We provide a thorough analysis of the associated challenges, the task shift and semantic shift, which diminish the effectiveness of UAPs on downstream FSL tasks.
We construct attacking framework and fill up the shifts step by step enhance the attack performance, which is effective across various FSL paradigms.
Our proposed method significantly advances state-of-the-atr methods in FSL attacking performance, with an increase over 16% in ASR in our standard.
Related Works
Adversarial attacks can be categorized into two types: image-dependent attacks and universal attacks. Srage-dependent attacks have been widely studied [14]
22 [1] The concept of Universal Adversarial Perturbations (UAPs) was initially introduced by [26] using an iterative Deepfool attack
23
271 applied to individual image samples. (28 developed data- inta-inneeendeet approach to generate
UAPs. Furthermore, 49 validated the efficacy using random source images. Several studies have explored the use of generative models to produce more generalized and natural-appearing UAPs.
15 were the first to utilize the generative network. (30] introduced GAP, a framework applicable to both classification and semantic segmentation models. [52] trained generator by leveraging

---

Tradittiona
FSL ASLAmtck
FSI. Setting
Clasiher
Step::
Enaader
Bicc
Enooder
Premain
Pany
Encoder
UAP
Powyy
Encedde
UAP
Classifer
Cassitee
Stepi:
Tesi-ime
Dovor
Encoder
Enseder
Dasse
Encoder
C Chosifern
Clasifies ni
EAP
Figure 2: Comparison different scenarios. In contrast traditional settings, the FSL framework introduces two distinct challenges: the task shift from pre-training to testing, and the semantic shift from the base to the novel dataset. Attackers must overcome the two shifts to mount successful attacks in the FSL scenario.
contrastive loss function. However, none of these studies address the specific challenges associated with attacking few-shot learning tasks.
Few-shot learning is a machine learning paradigm that aims to recognize novel classes from
few labeled samples [25]. Existing FSL methods can be broadly grouped into three categories meta- meta-based, and metric- based methods. Finetuning based
according to [21] finetuning-basee
strategies [47
32 24] follow a transfer learning process that includes pre- trainininng on base
classes and fine-tuning novel classes. Meta-based approaches [ 33
38
35
39
3
31] adopts
a meta-learning paradigm to learn the cross-task knowledge through the optimization between the meta-learner and base-learner. In this way, the model adopts quick adaptation the novel dataset.
Metric-based techniques [2] 40] 45 36, [5] (0 focus on learning transferable representations and making predictions based on the distance between feature representations. This strategy eliminates the need for test-time fine-tuning. These paradigms gimm have significantly advanced the progress of few-shot learning, yet a unified method for generating UAPs across all paradigms remains undeveloped.
3 UAP Setting for Few-Shot Learning
In this section, outline the threat model of attacking the few-shot tasks and present key definitions and notations to facilitate a clearer understanding of the concepts involved.
3.1 Threat Model
We consider an attacker who aims to craft universal adversarial perturbation to attack a pre-trained victim model and further impair the performance downstream few-shot tasks. The attacker only has access to the pre-traines model (e.g., by downloading from public repositories), but can not obtain datasets used for pre-training and have no knowledge of following few-shot tasks. Once the perturbation is generated, the attacker attaches it to each query sample. The crafted perturbation is imperceptible, and expected greatly mislead the few-shot classification.
3.2 Definition and Notations
In the domain of few-shot learning, abundant annotated images of the base dataset Dd can be used for pre-training. Subsequently, the model would be fine-tuned using limited samples from the novel dataset D,,, where the categories in D, do not overlap with those in Dj. In the pre-training stage, the victim model f-) that composed of an encoder fe(-) and base classifier f-c(-) is trained on De. In the evaluation stage, different forms of tasks are sampled from D,,. Each task contains a support set S used for finetuning and a query set used for evaluation. The support set includes n different classes with & samples per class, referred to as n-way k-shot. Similarly, the query set contains classes with q samples per class. For each task, a novel class classifier fnc(.) is newly trained on S. For any image in Q, (x), yields the classifier's 's output. The goal of FSL is to learn

---

85
80
80
75
Attack Success Rate
Proxy datasel Mini
70.41
69.49
70
70
63
65
60
60
.00
Cifar
54.58
54.09
60
50
60
50
40
30
10
Mini
Ticred
Number of ways
Novel dataset
(a) An illustration of the ASRs under different down- stream tasks.
(b) An illustration of the ASRs under dif-
ferent datasets.
Figure 3: The ASRs under different task shifts and semantic shifts. In (b), Cifar, Mini, Tiered are different datasets detailed in Section 6.1
generalizable Se(.) that exhibits good performance on Q across various tasks after learning from the limited examples S.
Once get the pre-trained encoder fe(-), an attacker can craft the adversarial images as follows: ande = x x where d represents the perturbation. To ensure that the adversarial images are visually indistinguishable from the original ones, 8 is usually restricted by chosen distance metric, such as the lrc. norm. This constraint can be expressed as ||æ x||>xx = |||||s S e. An effective approach generate is to train a generator go(-) the proxy dataset D,- The generator takes random vector as input and it aims to produce perturbations that satisfy the given constraints while maximizing adversarial impact.
4 A Closer Look at UAP Generation in Few-Shot Learning
In this session, we conduct thorough analysis of the challenges associated with generating Universal Adversarial Perturbations (UAPs) in the context Few-Shot Learning (FSL). We first compare the attack performance in the traditional scenario with that in the FSL scenario. Then we point out the lower performance in the FSL scenario due to the presence of two shifts.
4.1 Poor Performance of Traditional UAPs in the FSL Scenario
To better understand the performance of existing attacking methods on FSL tasks, we directly apply the traditional method in the FSL scenario. We compare the Attack Success Rate (ASR) in the FSL scenario with that observed in the original scenario. The ASR refers to the success rate of the UAP in fooling the classifier. higher ASR means stronger attack capability.
We adopt the Generative Adversarial Perturbation (GAP) method, due to its straightforward applicability FSL tasks and its high attack performance within its original context. The GAP method generates UAP through a ResNet Generator [7]. denoted as gCap. The perturbation is produced follows:
= (2),
(1)
where = represents a random vector. The generator is optimized with respect to the following loss term:
Leap = lf + 8),
(2)
Here, (x, y) is the data point sampled from proxy dataset, f(-) denotes the model pre-trained on the base dataset, and (2) ), y) represents the cross-entropy loss.
For the traditional GAP setting, we compute the mean ASR on VGG16, VGG19, and ResNet152 reported in the original paper to gauge the average performance scenario, where the attacker does not have access to the base or novel datasets, we utilize CIFAR-FS as the proxy dataset to generate UAPs. attack three pre-trained FSL. models representative of different paradigms: finetuning- based, metric-bann based, and meta-based approaches.
traditional scenario. the FSL

---

Attacker
M
Clansiferr
ASR
Methods
TS SS
1-shot
5-shot
UAP
Encoder
Logit
Base classifier x 65.07
61.73+0000
Proxy linear / X 70.87++..6
68.87+0z1
Base linear
V < 76. 76.75 74.67+021
Proxy
Thoxy
Linear
Protos
Proxy protos
< 80.04.10.30 77.691021
Proxy Danaet
(a) An illustration of the attacking framework.
(b) illustration different ASRs.
Figure 4: (a) The Base Classifier is the pre-trained classifier. Proxy Linear and Proxy Protos newly designed to address the task shift and the semantic shift, as detailed Section 5 (b) The detailed Attack Success Rate(ASR) of different methods on 5-way 1-shot and 5-way 5-shot tasks. TS, SS represents whether the Task Shift and the Semantic Shift are handled respectively.
As illustrated in the left panel of figure I GAP achieves pretty high mean ASR of 89.43% in traditional tasks. However, when switched to the FSL setting, there is a marked decrease in performance, with reduction of at least 25% observed in both 5-way 1-shot and 5-way 5-shot tasks. Moreover, the GAP method unable to generate UAPs for metric-based models, its generation process depends on the presence of a fixed classifier, which is lacking for metric- methods. This observation leads us to pose the question: What makes the attack success rate of UAPs decrease so much in FSL? In the next subsection, we will discuss the factors.
4.2 Two Shifts in FSL Affect the Attack Transferability
We compare the differences in the UAP generation process between the traditional scenario and the FSL scenario, as depicted in Figure 2 In traditional scenarios, UAP generation relies on a fixed classifier that remains unchanged throughout both the training and testing phases. In contrast, FSL scenarios present a variety of classification tasks during desting-encompasiing different categories and shapes, such 5-way 1-shot and 5-way 5-shot, thereby posing challenges UAP generalization. Therefore, our first hypothesis is that the task shift in FSL hinders the transferability of the UAP.
To validate our hypothesis, we conducted experiments to evaluate the performance of the UAP across different downstream tasks. The experiments vary the number of ways from 5 to 64 and the number of shots from to 20. To maintain the semantic consistency between the pre- vre-training and testing datasets, we sampled an additional 100 images for each category in the training split of mini-Imagen ImtgeNet to serve the downstream dataset. The results presented in Figure 3 (a), demonstrating that as the downstream task becomes less similar to the pre-training mask(64-way full shot), the attacking performance of the UAP diminishes.
From Figure 2 can also be observed that the categories in training and testing datasets do not overlap in the FSL scenario. This semantic shift may contribute to reduction in the attacking performance, Therefore, we derive second hypothesis that the semantic shift in FSL hinders the transfcrability of the UAP. To verify the impact of the semantic shift, we keep the downstream task constant and vary the proxy dataset and downstream dataset. We choose a model pre-trained on the mini- Imageee the vietim model. As illustrated in 3 the attack performance declines with increasing distances between both the downstream dataset and the pre-trained dataset(mini), as well as between the proxy dataset and the pre- pre-trained dataset(mini).
To summarize, we empirically demonstrate that the two shifts that existed the FSL scenario degrade the attack performance of traditionally generated UAPs.
Method
In this section, we propose a baseline framework aimed at generating effective Universal Adversarial Perturbations (UAPs) within Few-Shot Learning (FSL) scenario. Starting with an initial attacking framework, we progressively refine our approach to address the two shifts mentioned before. Finally, we have developed a universally applicable UAP generation strategy, capable of attacking models
5

---

trained under all kinds of FSL paradigms and showing strong generalizability to different downstream tasks.
5.1 A Baseline Attacking Framework
The attacking framework is illustrated in Figure 4 (a). The victim model, built on R backbone, is pre-trained using the training split of the Imageee3ageNe dataset, following the training paradigm established in [TO]. For the attack, we train a generator to transform random vectors into a UAP, which also used in [52]. Since the attacker lacks access to both the training and testing data from the mini-ImageNet dataset, we opt for the training split of the CIFAR-FS dataset [ as a proxy. To optimize our generator go, we employ a negative cross-entropy loss as the attacking objective, defined by the following equation:
oool - (
(3)
As the original approach, we utilize a 64-way classifier (ac(.) from the baseline model to infer, without any fine-tuning. The results presented in the first row of F (b), denoted as base classifier. The results are far from satisfactory compared to the performance in traditional tasks, highlighting the need for further refinement in the FSL scenario.
5.2 Fill Up the Task Shift through Task Alignment
As demonstrated in Section 4.2 a misalignment between upstream and downstream tasks can adversely affect the performance of UAPs. Given that the attacker cannot foresee the specific downstream tasks, we modify the tasks used in UAP generation to better align with those downstream tasks. Importantly, we only aim approximate the shape of the downstream tasks rather than achieving exactness, which is sufficient to let the UAP acquire the necessary task bias. This is supported by the ablation study detailed in Section 6.3] We sample praxy tasks with the shape of 5-way 1-shot from the proxy dataset Dz. For each task, we construct a linear classifier fye(? based on its support set Sp. We refer to this method as proxy linear. The objective function for training the classifier is given by:
Lpc = ==]fff((((()))
where (rr, .) represents the data sampled from Sp- Once the proxy linear classifier has been trained, the attacker generates perturbations guided by the perturbed query set Qade from the proxy task, as depicted in Figure 4 (a). The fooling loss is quite like equation Beuationneexcepl exxcept that the classifier is newly constructed and the data (z, y) sampled from 2p. rather than the entire proxy dataset. The fooling loss is calculated as follows to optimize the generator go
foot - Hl(Spoffe (=))))),))
(5)
where (I. is sampled from 2p.
By integrating task alignment into the training of UAPs, we empower them to generalize more effectively to downstream tasks, even in the absence of direct access to those tasks. From the results presented in the second row of Figure ( (b), it is evident that by addressing the task shift, there is marked improvement in the ASR, with an increase of 5% for 1-shot tasks and 7% for 5-shot tasks.
5.3 Fill Up the Semantic Shift by Leveraging the Encoder's Generalizability
Analysis of the semantic shift. In Section 4.2 we discussed the impact of the semantic shift on UAPs, a phenomenon widely present in FSL. As attackers lack access to upstream or downstream datasets, strategy emphasizes utilizing the generalization capabilities of the pre-trained encoder. well-trained encoder possesses the capability to transfer useful information to the novel dataset, even trained on the base dataset. Consequently, we hypothesize that the proxy dataset closely aligns with the base dataset, the generated UAP can be more transferable to the novel dataset.
We substitute the proxy dataset with the base dataset to create an ideal scenario for testing our assumption. While attackers cannot access the base dataset in real-world situations, utilizing it in our experiments allows us to better understand and validate our hypotheses. To be specific, we sample different tasks from the base dataset and train linear classifier for each task, following the process outlined in equation 4 The results are documented in the third row of Figure 4 (b), denoted as base

---

Encoder
Praxy Protes
Dirchake
Proxy Labels
Log
1.S
ASR
Supervise
1-shot
5-shot
X
X
80.04 +0.30 77.69
V
X
+0.72+0.0 78.35 +0:21
Prrty
81.56 79.00+a.1s
Encoder
Prony Classifier
(a) A qualitative illustration.
(h) An illustration of different ASRs.
Figure 5: (a) A qualitative illustration showing the effect of the supervisory signal. (b) The attack
performance comparison of different fooling losses.
linear. Compared to the Proxy liner, further improvement can be observed in the table. The results suggest that closer simulation of the base dataset distribution enhances the transferability of the UAP to the novel dataset.
Alleviation of the semantic shift. While utilizing the base dataset proves effective, the attacker can only utilize the proxy dataset. As discussed in 46), when the proxy dataset is fed into the encoder, its output distribution tends to shift towards the base dataset, which is exactly what we need.
However, the previously mentioned proxy linear method introduces a supervisory signal that corrects the distribution, as shown in Figure (a). The output distribution, initially altered by the encoder, is adjusted by the constructed proxy classifier.
This causes the UAP to lose its connection with the base dataset during training, thereby diminishing its ability to generalize to downstream tasks. Consequently, we abandon the proxy linear classifier and adopt metric-based evaluation mitigate the influence of labels from the proxy dataset. For each task sampled from the proxy dataset, we extract the output features from the support set and compute the mean of each class (i.e., class prototypes). The prototype for class k in support set Sp is calculated using the equation:
1 Σ fa(x). N
(x, y) is sampled from Sp and N₂ represents the number of samples for class ie in the support set. We then employ a distance metric :(-.)) to quantify the distance between the query feature and each class prototype. The classification probability for query sample in class is then determined by:
D(fa(s)eew)
ply = k|xx) Σn e((f.()).ew))
where k' represents class in the task. accordance with prior work 36), we utilize cosine similarity as the distance metric.
The proxy protos strategy facilitates closer approximation the novel dataset, regardless of the specific proxy dataset used. The efficacy this approach is documented in the final row Figure (b), which presents the most favorable results among all methods tested.
further improvement. As suggested by 30 an alternative formulation for the fooling loss is as follows:
L
log H (fe(x g(x))),
(8)
where fproto denotes the predicted classification computed using the prototype- approach previ- ously described. Furthermore, to avoid reliance on the proxy query label and enhance transferability. we can supervise the perturbation generator with smoothed labels, denoted as ÿ. The fooling loss redefined as:
L foot - log H H(Sproro (fe g(2))),
(9)
7

---

Table 1: 5-way 1-shot ASR results our framework different FSL victim models.
Victim
Proxy
Backbone
Baseline
Baseline++
ANIL-1
R2D2-1
ProtoNet
DN4
RNI2
81,561029
58,94.1.43
07.84=0.28
70.34
69.03
=3.31
Cifar
RN18
11.55
14.67
=2.11
70.00
60.02
67.66.0030
Mini
RN12
81.63.028
68,55
78.08 0.22
77.22 1:022
68.54
74,91 20033
Mini
RN18
+2.32
80.01 4:266
76.87
77.32
76.09 4031
69.36
Tiered
RN12
79.68
67.51 +0.43
77.81 --0288
77.09
4:0.27
72.87
4034
75.23+4033
RN18
79.45
76.36=0.30
77.11
74.80
73.31
RN12
36.031032
62.5611029
68.26=0.28
76,49
36.731032
78.47
Cifar
RN18
69.271032
73.8220077
70.31 69.27=0.29
2:030
67.60
63.66
Tiered
RN12
+0.9 +5.67
75.02-0.31
77.68
76.7 +032
78.80
Mini
RN18
74.74
80.03
75.68=0332
78.08
71.65
70.58
RN12
78
75.99
75.48=0.11
77.64
76.83
78.94
Tiered
RN18
73.51
80.44:10.25
-..3
78.25
71.45 1033
69.15
Table 2: 5-way 5-ohot ASR results of our framework different FSL victim models.
Victim
Proxy
Backhone
Baseline
Baseline++
ANIL-1
R2D2-1
ProtoNet
DN4
RN12
79.00=0.18
63.41
78.31
70.42:1022
74.88 67.9620.2
Cifar
RN18
71.82=021
73.87
71.37
69.744021
59.94=031
69.75
Mini
RN12
79.05
75.01
78.401021
79.14
70.09
76.90 76.90+0.20
Mini
RN18
=3.89
79.31
77.461021
79.09
76.34
68.70+022
RN12
76.78
74.03+0023
78.43
79.02+0.18
76.91 74.26
Ticred
RN18
73.24
78.99+0.00
77.42
78.92+0.17
74.75
75.93
RN12
75.09=0.21 59.40
68.96102s
76.94.0.19
78.33
75:91
Cifar
RN18
70.32.0.24
71.30
70,731023
71.55
70.87
66.0910.28
Tiered
RN12
78 28.43 78.07
76.05
78.72:0.16
78.34
76.4211022
Mini
RN18
28.11
79.41
77.901021
79.25
76.65
74.85
RN12
78.48
78.09
76.61
78.7940.6
20.7 77.09:0.22
Tiered
RN18
76.60
79.83
77.881021
19.14.0016
25.4 73.0610027
The adoption of this revised fooling loss yields further performance enhancements, as evidenced in Figure 5 (b). Consequently, we have adopted equation aas the final formulation for our fooling loss.
5.4 Conclusion and Discussion
Through thorough comparison of the traditional scenario and the FSL scenario, point two critical shifts that reduce the attack performance on FSL tasks. To address the task shift, we sample proxy tasks mirror the shape of the downstream tasks. To handle the semantic shift, we employ class prototypes to train UAPs, circumventing the introduction of the proxy labels. Moreover, the application of smoothed labels yields additional enhancements in attack performance.
Our final method can be summarized the following steps: (1) Sample 5 5-way 1-shot tasks from the proxy dataset. (2) Caleulate prototypes for each class in the proxy tasks. (3) Compute the fooling loss based on the distance between the query sample and class prototypes. (4) Optimize the perturbation generator produce highly transferable UAP in FSL. By following these steps, method systematically generates universal adversarial perturbations that are robust and generalizable across different tasks.
Experiments
6.1 Implementation Details
Datasets. We utilize three widely used datasets in Few-Shot Learning (FSL) as the proxy dataset: CIFAR-FSEA mini- ImageNe]3,NN,, and Secred-ImageNee[333 Both CIFAR-FS and mini- ImageNet

---

Table Comparison of different attack methods on ASR for 5-way 1-shot tasks.
Victim
Method
Mark
Baseline
Baseline++
ANIL-1
R2D2-1
ProtoNet
DN4
UAN
SPW-18
52.27
47.6810.2
43264.1226
GAP
CVPR-18
40.31
49.40
66.47.0.2
Mini
AdvEncoder
ICCV-23
76.68=am1
57.37
68.51
59.1010.0
66.63
22.85=oa1
FSAFW
Ours
81.56
18.94
77.84
70.34:029 69.03 73.31 0.33
UAN
SPW-18 40.3420:1
33.0020m
51.91 10
GAP
CVPR-18 49.7220.33 58,23
61.194030
Tiered
AdvEncoder
ICCV-23 75.99
53.82 62.16
+130 60.23 +0.33 68.86
FSAFW
Ours
76.0
62.56:0.29
68.26
76.4910227
76.73
78.47=034
Table 4: Comparison of different attack methods on ASR for 5-way 5-shot tasks.
Victim
Method
Mark
Baseline
Baseline++
ANIL-11
R2D2-1
ProtoNet
DN4
UAN
SPW-18
46.092n.3
45.42.000
42.23
GAP
CVPR-18 40.94 45.71
64.23+029
Mini
AdvEncoder
ICCV-23
24.19
55.12
67.34
$9.47
67.76
74.72
FSAFW
Ours
79.00
63.41
78.31 70.42+0.1 67.9
74.88
UAN
SPW-18 32.97
23,45
$1.75+o27
GAP
CVPR-18
44.90
52.40
59.52:1028
Tiered
AdvEncoder
ICCV-23
75.03
58,23
53,25:1028
70.93
60.40
69.55
FSAFW
Ours
25.09
59.40 68.96
76.94:1.19 28.3 75.91 20.21
datasets comprise images from 100 categories, divided into 64 classes for training, for validation, and 20 for testing. Tiered-ImageNe (779,165 images) organized into 34 high-level categories, which are further split into 351 training classes, 97 validation classes, and 160 testing classes. To prevent the attacker from gaining knowledge of the downstream images, only the training portions of these datasets are utilized for the generation of UAPs.
more extensive subset of ImageNet [9. consists of 608 classes
Training Details. All the victim models are downloaded from the Libfewshot(21] (2],, comprehensive Library for FSL. In line with the library's classification of FSL training paradigms, we selected Baseline and Baseline++ [00 as representatives of the finetuning- paradigm, ANIL [31 and
R2D2 3 for the meta-based paradigm, and ProtoNet (36) along with DN4 20 for the metric-based paradigm. For each paradigm, we adopted ResNet12 and ResNet 16 as the backbone of the victim models, following the previous works 8 48). All these victim models are pre-trained on the mini-ImageNet and Tiaged-ImageNet.
We adopt the Attack Success Rate (ASR) to evaluate the attack performance of UAPs on 2000 FSL tasks. The generator network, following the approach described in (52), was optimized using the Adam optimizer with an initial learning rate of 0.0002. The perturbation was kept within an lx -norm bound of e 10, considering pixel values in the range of 255. Additional FSL testing configurations follow the details provided in Libfewshot.
6.2 Results
Effectiveness of our attacking framework. Our investigation into the effectiveness of our attacking framework (FSAFW) across different FSL models on different tasks demonstrated in [ and For each FSL paradigm, we targeted four models pre-trained on mini-ImageNe and TieredImageNet, utilizing ResNet12 and R as backbone architectures. We report the mean Attack Success Rate (ASR, %, top-1) alongside 95% confidence intervals. For the meta- meta-based methods ANIL and R2D2, we chose the victim models trained by 5-way I-shot episodes denoted as ANIL-I and R2D2-1 the table (ANIL-5 and R2D2-5 are present in Section A)) Notably, when the victim's 's dataset is consistent with the proxy dataset, attack performance generally improves. Our proposed attacking framework can achieve comparable ASR even when the proxy dataset is different from the victim one.

---

Table 5: Comparison of different contrastive losses on ASR.
Tasks
Ours
InfoNCE
SimCLR
MSE
Cosine
SupCon
feature scatter
5 5-way 1-shot 81.56.029 77.67:001
77.26
64. (4,19=0.3 79.32.10.30
77.48
1038
5 5-way 5-shot 79.00 75.40
74.69.0.23
60.38 76.87100.2
14.94
72.90
I-shot
S-shot
14bhot
S-abot
Attack success rale
81
Attack success rate
80
76
75
0
5
10
15
20
25
30
35
5
10
15
25
30
35
The number of ways
The number of shots
Figure 6: An illustration of the ASR that different forms of proxy tasks bring.
Comparison with other attacking methods. We compare the ASR of our attacking framework with the classic cenerator-based adversarial methods, such as UAN 13), GAP [0), and AdvEncoder [52), against the aforementioned FSL victim models. For this comparison, we utilized ResNet12 as the backbone and CIFAR-FS as the proxy dataset. ASR results for both 5-way 1-shot and 5-way 5-shot tasks are detailed in 3 and 4 respectively. Our framework consistently outperforms the other methods across victim models trained under various configurations. Note that some attacking methods rely on a fixed classifier in testing to generate UAPs, while models like R2D2, ProtoNet, and DN4 do not have one, These methods are not applicable to such models, and we have indicated this incompatibility with dash ("-") in the relevant table cells.
6.3 Ablation Study
The attack performance of contrastive losses. To generalize the attack ability to the downstream tasks, a simple way adopt the contrastive (5s) In this section, we probe the effectiveness of various contrastive losses in crafting generalizable UAPs. We adopt the InfoNCE loss (2)), SimCLR loss/6), MSE loss, Cosine similarity loss, SupCon loss[T8] and feature scatter loss (51 train the generator. We use a victim model trained on mini-ImageNN and with a backbone of ResNet12. The model trained and tested under the Baseline [0] paradigm. We compare their ASR performance with that of our method in Table 5 The results suggest that contrastive losses in attacking FSL tasks are inferior to our FSAFW.
The influence of different forms of proxy tasks. We evaluate how different shapes of the proxy tasks affect the attack performance of the generated UAP. For the proxy tasks, we keep the number of shots constant at I while varying the number of ways, and vice versa, maintaining 5 ways while changing the number of shots. We utilize 5-way 5-shot and 5-way 1-shot as the shapes for downstream tasks. The attack results are displayed in the left and right panels of Figure 6 It can be observed much from the downstream tasks, the
that when the shapes of the proxy tasks do not deviate attack performance can be maintained. This suggests that when constructing the proxy tasks, rough estimation of the downstream tasks' shape is enough.
7 Conclusion
In this work, we propose unifying Few-Shot Attacking FrameWork (FSAFW) to generate transfer- able UAPs on the FSL scenario. We identify and analyze two major challenges in FSL: the task shift and the semantic shift. We adopt proxy tasks to handle the task shift and proxy prototypes to address the semantic shift. Our framework significantly outperforms existing methods, establishing new benchmark for launching attacks on FSL tasks.
10

---

Acknowledgements
This work is supported by the National Natural Science Foundation of China under grants 62206 102, 62376103, 62436003 and 62302184; the Science and Technology Support Program of Hubei Province under grant 2022BAA046; Hubei Science and Technology Talent Service Project under grant 2024DJC078; and Ant Group through CCF-Ant Research Fund.
References
[1] Yuanhao Ban and Yinpeng Dong. Pre-trained adversarial perturbations. Advances in Neural Information Processing Systems, 35:1196-1209, 2022.
[2] Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved few-shot visual classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14493-14502, 2020.
[3] Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. arXiv preprint urXiv: 185.08600 2018
[4] Luca Bertinetto, Joao Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. arXiv preprint a 36, 2018.
[5] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). pages 39-57. leee, 2017.
[6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[7] Wei-Yu Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. arXiv preprint arXiv:1904.044 042332, 2019.
[8] Wei- Yu Chen, Yen-Cheng Liu, 7solt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. arXiv preprint arXiv:1904.04233 2019.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Patlern Recognition, pages 248-255. 2009.
[10] Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. arXiv preprint (tt0::/999.02200 2019.
[11] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 9185-9193, 2018.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprins arXiv:2010.1 19999 2020.
[13] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. International conference on machine learning, pages 1126-1133 1135. PMLR, 2017.
[14] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint arXiv:1412.6572. 2014.
[15] Jamie Hayes and George Danezis. Learning universal adversarial perturbations with generative models. In 2018 IEEE Security and Privacy Workshops (SPW), pages 43-49. IEEE, 2018.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.

---

[17] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Esion-ECCY' 2016: 14th European Conference, Amsterdam, The Netherlands, October I 11-14, 2016, Proceedings, Part II 14, pages 694-711. Springer, 2016.
[18) Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Samna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18673, 2020.
[19] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[20] Wenbin Li, Lei Wang, Jinglin Xu, Jing Huo, Yang Gao, and Jiebo Luo. Revisiting local descriptor based image-to- toce-to-clue cllss measure for few-shot learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7260-7268, 2019.
[21] Wenbin Li, Ziyi Wang, Xuesong Yang, Chuanqi Dong, Pinzhuo Tian, Tiexin Qin, Jing Huo, Yinghuan Shi, Lei Wang, Yang Gao, et al. Libfewshot: A comprehensive library for few-shot learning. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2023.
[22] Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. arXiv preprint
2016.
[23] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706 06088, 2017.
[24] Puneet Mangla, Nupur Kumari, Abhishek Sinha, Mayank Singh, Balaji Krishnamurthy, and Vineeth N Balasubramanian. Charting the right manifold: Manifold mixup for few-shot learning. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2218 222-222722222 2020.
[25] Erik G Miller, Nicholas E Matsakis, and Paul A Viola. Learning from one example through shared densities on transforms. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No. PR00662), volume 1, pages 464-471. IEEE, 2000.
[26] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Univer- sal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1765-1773, 2017.
[27] Seyed-Mohsen Moosavi-Dezfool Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2582, 2016.
[28] Konda Reddy Mopuri, Utsav Garg, and R Venkatesh Babu. Fast feature fool: A data independent approach to universal adversarial perturbations. arXiv preprint arXiv:1707.055: 777.0552, 2017.
[29] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 00000007788.... 2018
[30] Omid Poursaeed, Isay Katsman, Bicheng Gao, and Serge Belongie. Generative adversarial per- turbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4422-4431, 2018
[31] Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint (x0iv:190.009) 2019.
[32] Jathushan Rajasegaran, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Mubarak Shah. Self-supervised knowledge distillation for few-shot learning. arXiv preprint w0X.::2006.09785. 2020.
[33] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In Interna- tional conference on learning representations, 2017. 2017.
12

---

[34] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenen- baum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classification. arXiv preprin! arXiv: 00666, 2018.
[35] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-auggmented neural networks. In International conference on machine learning, pages 1842-1850. PMLR, 2016.
[36] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
[37] Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: multi-class classification competition. In The 201 international joint conference on neural networks, pages 1453-144 1460. IEEE, 2011.
[38] Qianru Sun, Yaoyao Liu, Zhaozheng Chen, Tat-Seng Chua, and Bernt Schiele, learning through hard tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence. 44(3):1443-1555
2020.
[39] Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. deta-traansfer learning for few- shot learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 403-412, 2019.
[40] Flood Sung. Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Tomr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1199-1208, 2018
[41] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties neural networks. arXiv preprint 111.v9:3 2013.
[42] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? In Computer Vision-ECC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 266-282. Springer, 2020.
[43] Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Daan Wierstra, et al. Matching networks for one shot learning. Advances neural information processing systems, 29, 2016.
[44] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
[45] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
[46] Jing Xu, Xu Luo, Xinglin Pan, Yanan Li, Wenjie Pei, and Zenglin Xu. Alleviating the sample selection bias in few-shot learning by removing projection to the centroid. Advances in Neural Information Processing Systems, 35:21073-21086 2022.
[47] Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. arXiv preprint 210.62200..0639, 2021.
[48] Sung Whan Yoon, Jun Seo, and Jackyun Moon. Tapnet: Neural network augmented with task-adaptive projection for few-shot learning. In International conference on machine learning, pages 7115-7123. PMLR, 2019.
[49] Chaoning Zhang, Philipp Benz, Tooba Imtiaz, and In So Kweon. examples from the mutual influence of images and perturbations. In Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition, pages 14521-14530, 2020.
Understanding adversarial
[50] Chi Zhang. Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classifica- tion with differentiable earth mover's distance and structured classifiers. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 12203-12213, 2020. 2020.
13

---

[51] Haichao Zhang and Jianyu Wang. Defense against adversarial attacks using feature scattering- based adversarial training. Advances in Neural Information Processing Systems, 32, 2019.
[52] Ziqi Zhou, Shengshan Hu, Ruizhi Zhao, Qian Wang, Leo Yu Zhang, Junhui Hou, and Hai Jin. Downstream-agnooste adversarial examples. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 4345 4355, 2023.
14

---

Table 6: 5-way 1-shot ASR results our framework different FSL victim models.
Victim
Proxy
Backbone
ANIL-5
R2D2-5
ProtoNet-5
DN4-5
RNI2
67.80
74.771034
61.101039
66.701037
Cifar
RN18
59,42
65.95
70.11 4434
63.99
Mini
RN12
29.48
74.761034
08.80
68.77
Mini
RN18
60. 14.4
79. 22
70.45
65.364032
RN12
69.31 +0.43
74.66 +0.35
68.34
70.22
Tiered
RN18
60.06=0447
28
65. 1032
65.324
Table 7: 5-way 5-shot ASR results of our framework in different FSL victim models.
Victim
Proxy
Backhone
ANIL-5
R2D2-5
ProtoNet-5
DN4-5
RN12 RN18
72.75
79.39=017
63.65
69.05_0.26
Cifar
67.93
65.90=a.25
76.06=0118
64.32_a.27
Mini
RN12
74.96
79.40
75.84
74.30=0.23
Mini
RN18
71.93
79.48=a16
75.53
66.28-0.27
RN12
74.51 1::26
79.38
75.80
74.44=0.23
Tiered
RN18
69.07 10:26
79.33
68.52.0.24
66.18=0.27
A More Victim Models
Due to page limitations, we present the Attack Success Rate (ASR) our method on more victim models trained under different paradigms in this section. Specifically, we selected the victim models trained by 5-way 5-shot episodes under the four paradigms: ANII [I], R2D2 33. ProtoNet [36), and DN4 20 These models are referred to as ANIL-5, R2D2-5, ProtoNet-5, and DN4-5 in the table. The victim dataset we chose ImageNe,,,,,, while CIFAR-FS, mini- meteeem and maaeed were used as proxy datasets. The corresponding results can be found in Table O and Table 7 As
observed, our method consistently maintains a high Attack Success Rate (ASR) across all victim models, across different Few-Shot Learning (FSL) settings. This demonstrates the effectiveness of our method on models trained with 5-way 5-shot episodes.
B Compared with An Additional UAP Method
We have incorporated the P method under the few-shot learning scenario and made compara- tive analysis. The results in Table 1a show that our proposed method demonstrates effectiveness not only on the base dataset but also exhibits superior cross-dataset performance without necessitating access to the base dataset.
Table 8: The ASR results compared with the PAP method.
1-shot
5-shot
Proxy
Method
Baseline
ANIL-1
ProtoNet-1
Baseline
ANIL-1
ProtoNet-1
PAP
1030
44.52.0..9
37.70
36.58
42.34
35.69
Cifar
Ours
81.561209
77.84
69.03
79.00=a.1s
78.31
67.964222
PAP
45.59.030
44.73
37.92.1.30
37.86=0.26
42.65
36.21
CUB
Ours
81.17
77.83
71.22::0.6
28.2
78.25
72.24
PAP
47.91
45,88 20.28
38.52
40.69
43.92
36.634026
Omniglot
Ours
76.32.10.1
77.81 -20.28
67.70
74.46
77.87 :0.21
69.05
PAP
42.624029
44.54-t0..5
$7.96+#0.29
34.67 +.026
42.39 40.27
49, +026
GTSRB
Ours
79.9940.0
78.13
69.17
77.55
77.93
69.98
15

---

Table 5-way 1-shot ASR results of cross domain datasets.
Proxy
Downstream
Method
Baseline
ANIL-I
ProtoNet-1
UAN
52.47
62.85
GAP
48.4
67. 17,7 +0.28
CUB
Mini
AdvEncoder
76.57
72.86
70.80
Ours
81.17
77.83++..5
71.22 -0.36
UAN
44.90
37.58 +0026
GAP 60.95 1:223
63,82 10.35
Omniglot
Mini
AdvEncoder
74.16
47.97 1:034
54.72
Ours 76.32.10.1
77.81 1:022
67.70...35
UAN
54.45
50.17 10026
10.25
GAP
54.09
68, 22.2211331
GTSRB
Mini
AdvEncoder
67.24:1032
62.86:10.0
67.16.0.34
Ours
79.99
78.13
69.17 00.3
UAN
76.82.1036
73.75
GAP
69.701035
75.08
Mini
CUB
AdvEncoder
76:21
76.82-10.6
71.40=0040
Ours
79.22
78.09
71.60
UAN
36.20+0.0
56.63
GAP
20.09+0.2
25.40 20.33
Mini
Omniglot
AdvEncoder
64.41
66.27 1:065
71.80
Ours 76.4410.27
69.64..064
75.27 0.336
UAN
74.86+003
79,82
GAP 73.30+004
79.04
Mini
GTSRB
AdvEncoder
77.18
78.27 20.37
76. 10.37
Ours
80.29
80.31 1:.36
78.41 =0.35
C Cross Domain Experiments
To better validate the generalizability of our attack approach, we conducted experiments various cross-domain datasets. We used CUB [45). [44] and GTSRB [37] as cross-domain datasets, employing them both proxy datasets and downstream datasets to implement the attacks. We selected one victim model each from the fine-tuning, meta-based, and metric-based FSL paradigms for our experiments, comparing approach with previous attack methods. The results are shown in Table and Table I0 demonstrating that our attacking framework exhibits broader generalizability.
D Datasets
The CIFAR-FS 4 dataset is randomly sampled from CIFAR- 100 19 which contains 100 classes with 600 images of 32x32 size per class. The 100 classes are split into 64 classes for training, 16 classes for validation and 20 classes for testing. The average inter-class similarity is sufficiently high to represent a challenge for the current state of the art. Moreover, the limited original resolution of 32x32 makes the task harder and the time allows fast prototyping. The samples of this dataset are listed in Figure 7
The mini- ImageNet dataset proposed by [43 for few-shot learning evaluation. Its complexity is high due to the use of ImageNet images but requires fewer resources and infrastructure than running on the full ImageNet dataset. In total, there are 100 classes with 600 samples of 84x84 color images per class. These 100 classes are divided into 64, 16, and 20 classes respectively for sampling tasks for meta-traing meta-training, meta-validation, and meta-test. The samples of this dataset are listed in Figure
The mageed dataset 34 a larger subset of ILSVRC-12 with 608 classes (779,165 images) grouped into 34 higher-level nodes in the ImageNet human-curated hierarchy. This set of nodes partitioned into 20, 6, and disjoint sets training, validation, and testing nodes, and the corresponding classes form the respective meta-sets. As argued in Ren al., this split near the root
16

---

Table 10: 5-way 5-shot ASR results of cross domain datasets.
Proxy
Downstream
Method
Baseline
ANIL-I
ProtoNet-1
UAN
45.93
63.53
GAP
65.91
67. 17.58 1:025
CUB
Mini
AdvEncoder
73.67 4022
72.67 4:0.2
71.99
Ours
78.29
78.25 +-020
72.24
UAN
37.34+t027
35,05 4-0.26
GAP
57.22
61.09 1:033
Omniglot
Mini
AdvEncoder
72.07::00
44.73
57.38 0.226
Ours
74.46-10.21
77.87 1:021
69.05 0.26
UAN
48.46:1029
13.22
GAP
48.68
65,94.1028
GTSRB
Mini
AdvEncoder 63.991027
60.5 1 10.27
68. 06.2
Ours
+..23 77.55+00.0
69.98
UAN
77.42.10:0
64.42-10.s
GAP
68.901027
68.89 20.52
Mini
CUB
AdvEncoder 75.29
67.31 1:068
72.80
Ours
77.46:021
69.43
72.86
UAN
26.15
52,25
GAP
10.1440
21.69 +0.32
Mini
Omniglot
AdvEncoder 60.57
62.99 62.99.0066
68.25 25 0.33
Ours
75.04:10.5
67.52
73.15 0..30
UAN
74.27+t023
78.3 +:0.88
GAP 68.57
77.77 +:029
Mini
GTSRB
AdvEncoder
73.961006
77.50.2029
78.38
Ours
77.22.1024
78.50
78.77 -0.23
Figure 7: Samples of CIFAR-FS.
of the ImageNet hierarchy results in more challenging, yet realistic regime with test classes that are less similar to training classes. The samples of this dataset listed in Figure 9
E Broader Impact
We point out the security problem in widely used open-set setting, i.e Few-Shot Learning (FSL). Under this setting, we propose a unified attack framework to generate the UAP, which can fool all kinds of FSL models. The attack method can be used in many vision-based real-world scenarios, as
Figure 8: Samples of mini-ImmgeNa ImageNet.
17

---

Figure 9: Samples of ticredImageNet.
doesn't 't need acquire the pre-training and downstream data. Also, the attack method can draw attention to the development of a defense method in FSL.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
We do not have theoretical results or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have reported the detailed settings and hyper-parameters. We will release our codes.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification: We have reported the detailed settings and hyper-parameters. We will release our codes.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (ltttps:/nniss.cc/ public/guides
Cowe//ooeeeeeeeee ss http:///uu.deeo/oooeeeeaaaasl///
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
7 //nips.cci http:/uulllgggucceee
htte//ooeeSeiieieeeeeeee iobiisssiiii. onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have reported the detailed settings and syper-paramette
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We have reported the confidence interval in our ablation studies.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have reported the computer resources the implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the broader impact in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not have the risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have respected the license of assets used in this paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not have assets in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not include human subjects information.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not include human subjects information.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "KPPa2M26LT", "reviewer": "Reviewer_id8n", "summary": "Authors argue that Univeral Adversarial Perturbation (UAPs) do not work well in few-shot learning (FSL) scenarios due to task and semantic shifts between the data. To this end, they proposed a method for crafting UAPs in FSL scenarios. The threat model assumes access to a pre-trained model with no access to pre-training data and downstream few-shot tasks. First, the authors conducted several experiments to understand the effect of semantic and task shifts for generating UAPs for few-shot scenarios. Next, they utilized several ablation experiments to design this method. The proposed method consists of training a UAP on a 5-way-1-shot FSL task with a cross-entropy-like fooling loss.", "questions": "- For training of UAP generator ($g_\\theta(.)$), just one 5-way-1-shot task is used or 5-way-1-shot tasks are sampled from a datasets? \n  - It could be an extremely efficient method if, for one UAP, a single 5-way-1-shot is utilized.\n\n- Three FSL datasets used (two ImageNet variants and CIFAR10) are closely related. How does your method work for a significantly different downstream task?\n\n- Why was GAP chosen as a UAP generator, even though it is a 6-year-old paper? How does your method compare with recent UAP methods?\n\n- Why is a comparison with state-of-the-art UAPs not provided?", "limitations": "The authors have added a broader impact statement.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper presents an interesting and novel method to craft UAP examples for FSL scenarios. Interesting empirical studies are conducted to understand different aspects of the method, including the failure of previous methods and components of the proposed method. Comprehensive experiments are performed to evaluate its performance.", "weaknesses": "- Authors apply GAP (an existing UAP method) and show its bad performance on few-shot learning scenarios. They attributed the bad transferability of GAP in FAL to the task shift. However, one more relevant issue could be the small number of examples used for generating GAP-based UAP.\n\n- The writing of the paper can be improved. Parts of it are difficult to understand and confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your valuable comments. In the following, we respond to the concerns.\n> Q1: Authors attributed the bad transferability of GAP in FSL to the task shift. However, one more relevant issue could be the small number of examples used for generating GAP-based UAP.\n\nThanks for the insightful comments. We would like to clarify that **whether in the traditional scenario with GAP or in the FSL scenario, the generation of UAP is safeguarded by a substantial amount of image data.** When generating the UAP using the GAP method in FSL, the attacker leverages a pre-trained model(comprising an encoder and a classifier) to conduct the pre-training classification task (such as 64-way classification) and optimizes the classification loss in reverse to achieve the attack effect. Consequently, all images from the proxy dataset are utilized during the generation of UAP. In the downstream finetuning stage, the model provider fine-tunes the model based on the downstream tasks (training a new classifier with a small set of samples). The attacker cannot manipulate this procedure as he doesn't know what the downstream tasks will be. When the fine-tuning is complete and the user is ready to infer with the new model, the attacker implants the unique UAP onto the images uploaded by the user, thereby executing the attack.\n\n**Consequently, we claim that the decline in GAP's effectiveness when applied to FSL is attributed to the task shift and the semantic shift.** As these two shifts are progressively mitigated, we achieve a substantial enhancement in attack performance.\n> Q2: For training of UAP generator, just one 5-way-1-shot task is used or 5-way-1-shot tasks are sampled from a dataset?\n\nThank you for your question. We sample multiple 5-way 1-shot tasks from a dataset. **The fundamental distinction between UAPs(image-agnostic) and image-dependent adversarial perturbations lies in the data required in their generation process.** A single UAP is designed as a cumulative effect of perturbations across many individual samples, which enables it to be effective against a wide range of images. This process is crucial for its universal applicability across different inputs. Meanwhile, **in the setting of few-shot learning, it is also necessary to first pre-train a transferable base model using a large amount of samples**, and then generalize it to new downstream tasks with a limited number of samples.\n> Q3: Why was GAP chosen as a UAP generator? How does your method compare with recent UAP methods?Why is a comparison with state-of-the-art UAPs not provided?\n\nWe chose GAP due to its representativeness as a classical method and its convenience for an extension to the FSL scenario, but our comparisons extend well beyond it in the paper(lines 312-320, table4 and table5). **We benchmarked our approach against several methods(including the recent AdvEncoder), across six different few-shot learning paradigms, two distinct downstream tasks (5way-1shot and 5way-5shot), and two victim datasets, outperforming all**.\n\nRegarding the similar motivation of the PAP[1] method to ours (both are generalizing UAPs to unseen downstream tasks), we've also evaluated its performance in FSL. PAP perturbs only the low-level feature to mitigate the effect of fine-tuning, which has proven effective in the original paper. However, given the scarcity of classes and samples in few-shot scenarios, PAP underperforms in FSL,  The results are shown in table3 and table4.\n\nTable 3: 5-way 1-shot results.\n| proxy    | method   | baseline       | anil         | protonet       |\n| -------- | -------- | -------------- | -------------- | -------------- |\n| cifar    | PAP      | 44.39±0.30     | 44.52±0.29     | 37.70±0.29     |\n|          | **ours** | **81.56±0.29** | **77.84±0.28** | **69.03±0.36** |\n| cub      | PAP      | 45.59±0.30     | 44.73±0.29     | 37.92±0.30     |\n|          | **ours** | **81.17±0.30** | **77.83±0.28** | **71.22±0.36** |\n| Omniglot | PAP      | 47.91±0.31     | 45.88±0.28     | 38.52±0.29     |\n|          | **ours** | **76.32±0.31** | **77.81±0.28** | **67.70±0.35** |\n| GTSRB    | PAP      | 42.62±0.29     | 44.54±0.29     | 37.96±0.29     |\n|          | **ours** | **79.99±0.30** | **78.13±0.20** | **69.17±0.38** |\n\nTable 4: 5-way 5-shot results.\n| proxy    | method   | baseline       | anil         | protonet       |\n| -------- | -------- | -------------- | -------------- | -------------- |\n| cifar    | PAP      | 36.58±0.26     | 42.34±0.28     | 35.69±0.26     |\n|          | **ours** | **79.00±0.18** | **78.31±0.21** | **67.96±0.28** |\n| CUB      | PAP      | 37.86±0.26     | 42.65±0.27     | 36.21±0.26     |\n|          | **ours** | **78.29±0.21** | **78.25±0.20** | **72.24±0.28** |\n| Omniglot | PAP      | 40.69±0.27     | 43.92±0.27     | 36.63±0.26     |\n|          | **ours** | **74.46±0.21** | **77.87±0.21** | **69.05±0.26** |\n| GTSRB    | PAP      | 34.67±0.26     | 42.39±0.27     | 49.44±0.26     |\n|          | **ours** | **77.55±0.20** | **77.93±0.28** | **69.98±0.30** |\n> Q4: Three FSL datasets used are closely related. How does your method work for a significantly different downstream task?\n\nThank you for your valuable suggestion. In addition to the three FSL datasets, we have augmented our experiments with three out-of-domain datasets to further validate the generalizability of our method. These include CUB(fine-grained bird images), Omniglot(handwritten characters), and GTSRB(traffic sign images). We utilized these datasets both as proxy datasets and downstream datasets in extended experiments.\n\nThe experimental results are summarized in the **table1 and table2 in the 'Author Rebuttal'**, which shows that **our method is capable of achieving state-of-the-art adversarial attack performance even when there is a significant domain gap between the datasets.**\n\n[1] Pre-trained Adversarial Perturbations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j3ZtpLvnC3", "reviewer": "Reviewer_gmrg", "summary": "This paper addresses the adversarial attack on image classification tasks, especially for the few-shot learning setup with universal adversarial perturbations. Authors claim that the traditional approach does not work well in the FSL setup, which has two properties including task shifts and semantic shifts. The paper addresses these issues by proposing to do the attack on few-shot tasks, with proxy protos. Experiments are carried out on multiple FSL frameworks, including finetuning/meta/metric-based paradigms, and show the effectiveness of the proposed method on all of those frameworks.", "questions": "In Fig 1 and line 142, we mention that GAP cannot generate UAPs for metric-based models. I am curious can we use the base classifier for the attack? Which setup is used for reporting numbers in Fig 4(b)?", "limitations": "Limitations are sufficiently discussed.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The results reported in Fig 4(b) are very promising, and the proposed method also achieves good results comparing with other attack methods as reported in Tab 4.\n* The investigated problem is important yet less studied.\n* The proposed method is simple.", "weaknesses": "* All datasets are natural images, and I wonder if the experiments can be improved by trying on out-of-domain datasets, e.g. Omniglot or CUB.\n* In Tab 2 and 3, sometimes the ASR is the best when victim and proxy datasets are different. E.g. in Tab 3 DN4, when backbone=RN18 and victim=Tiered, the Mini-proxy version has better ASR (74.85) than the Tiered-proxy version (73.06). This may need more analysis and ablation.\n* More ablations on the algorithms (line 275 to 278) are needed. For example, is 5-way-1-shot always the best proxy task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your valuable comments. In the following, we respond to the concerns.\n> Q1: All datasets are natural images, and I wonder if the experiments can be improved by trying on out-of-domain datasets, e.g. Omniglot or CUB.\n\nThanks for your constructive suggestion. In addition to the original three natural image datasets, we have supplemented our study with three cross-domain datasets: CUB, Omniglot, and GTSRB (Traffic Signs), using them alternately as proxy and downstream datasets. Meanwhile, we select Baseline, ANIL, and ProtoNet as representatives of the finetuning, meta, and metric-based FSL paradigms. The experimental results are demonstrated in **table1 and table2 in 'Author Rebuttal'**, which demonstrate that **even with a substantial domain gap between datasets, our approach can still achieve SOTA attack efficacy across all three FSL paradigms.**\n> Q2: In Tab 2 and 3, sometimes the ASR is the best when victim and proxy datasets are different. This may need more analysis and ablation.\n\nThank you for raising this point. We have carefully examined the data and observed that this phenomenon occurs specifically between the miniImageNet and tieredImageNet datasets. We speculate that this is due to the similarity and overlap between the images in the mini and tiered datasets. Consequently, in some cases, we observe higher ASR when the proxy and victim datasets are different.\n> Q3: More ablations on the algorithms (lines 275 to 278) are needed. For example, is 5-way-1-shot always the best proxy task?\n\nWe appreciate your suggestion, and we would like to clarify that **we have indeed conducted ablation studies to explore the impact of varying the number of ways and shots in our proxy tasks, as outlined in lines 331 to 339**. Aligning with the setup detailed in Section 5.1, we fixed the number of shots at 1 and varied the ways (from 5-way up to 35-way), and conversely, we held the way constant at 5 and increased the shots (from 1-shot up to 30-shot). The downstream tasks remain at 5-way 1-shot and 5-way 5-shot.\n\nThe results are depicted in Figure 6, which shows that the ASR for downstream tasks remains relatively high when the proxy tasks are within 25-way and 20-shot. Meanwhile, as both the way and shot numbers deviate further from the actual downstream task, a notable decline in attack effectiveness can be observed. This suggests that a rough estimation of the downstream tasks is sufficient for an attacker to discern the task bias and produce transferable UAPs.\n> Q4: In Fig 1 and line 142, we mention that GAP cannot generate UAPs for metric-based models. I am curious can we use the base classifier for the attack? \n\nThank you for the insightful suggestion. Theoretically, the suggestion is feasible. When attacking metric-based models using GAP, since the attacker does not have direct access to the pretraining and downstream datasets, he could train an additional base classifier based on the proxy dataset, which would then be used to generate UAP aimed at the downstream tasks. **However, there's a crucial consideration: the classifier trained on this proxy dataset inherently introduces semantic shift, as it is tailored to the distribution of the proxy data rather than the actual target data**. Consequently, UAPs generated through such a classifier are likely to exhibit diminished effectiveness.\n\nTo provide empirical evidence, we conducted experiments where CIFAR-FS served as the proxy dataset, while mini-ImageNet was employed both for pre-training and as the downstream dataset. We utilized the ProtoNet with ResNet12 as the backbone for our metric-based model. Our results, outlined below, illustrate that when using this approach to generate GAP-based UAPs, the performance in both 5-way-1-shot and 5-way-5-shot downstream tasks is significantly inferior to the methods that directly utilize the original model for UAP generation.\n| method                              | 5way 1shot ASR | 5way 5shot ASR |\n| ----------------------------------- | -------------- | -------------- |\n| GAP (proxy trained base classifier) | 51.20±0.39     | 61.36±0.26     |\n| AdvEncoder (ICCV2023)               | 66.63±0.34     | 67.76±0.26     |\n| **Ours**                            | **69.03±0.36** | **67.96±0.28** |\n\n> Q5: Which setup is used for reporting numbers in Fig 4(b)?\n\nFigure 4(b) illustrates the performance of our attacking framework, whose setting is detailed in section 5.1(lines 180-193). Within this framework, we choose a model pre-trained on the train split of the mini-ImageNet dataset with a ResNet12 backbone as our victim model. The train split of the CIFAR-FS dataset is chosen as the proxy dataset while the test split of the mini-ImageNet dataset is chosen as the downstream dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XfueECi5e4", "reviewer": "Reviewer_L9xm", "summary": "This paper proposed a framework for generating Universal Adversarial Perturbations (UAPs) by considering the task and the semantic shift in the context of few shot learning. The authors introduces proxy tasks and proxy protos to enhance the transferability of UAPs in different FSL tasks. Results are validated on miniImagetNet and tieredImageNet.", "questions": "1. In section 5.3, authors found that using base dataset as the proxy dataset improves the performance. The authors thus interpret it as “if the proxy dataset can be distributed near the base dataset, the generated UAP can be more transferable to the novel dataset.” Could the improvement be due to that the base distribution is more like that of the novel dataset (compared with the proxy distribution), instead of due to enhanced transferability?\n2. What’s the rationale of removing log() in eq(3) in section 5.1 to “demonstrate the fundamental performance of the attack within this framework”?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors consider a practical problem of attacking FSL with UAP settings\n2.  Paper is well written with dedicated analysis on potential causes of performance gap of traditional UAPs", "weaknesses": "1. Evaluation of the proposed UAP framework is conducted on smaller datasets like miniImageNet, it’s not clear if the task and semantic shifts still hold on larger scale datasets like meta-dataset.\n2. The paper lacks discussions regarding how the proposed proxy tasks relate to/differ from the existing meta-learning UAP with similar use of few shot learning tasks (e.g., https://arxiv.org/pdf/2009.13714).\n3. The paper can benefit from a more comprehensive comparison against recent UAP frameworks (e.g., Pre-trained Adversarial Perturbations/PAP), with more in-depth discussions regarding how these alternatives address the gap from task and semantic shifts, making clearer the advantage of the proposed approach.\n4. The uses of “base classifier” and “base linear” are ambiguous in Figure 4(b), it could help to add explanations of these terms to the caption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your valuable comments. In the following, we respond to the concerns.\n> Q1: It’s not clear if the task and semantic shifts still hold on larger-scale datasets like meta-dataset.\n\nThank you for your valuable suggestion. Due to constraints of time and space, we select three distinct sub-datasets from the meta-dataset that significantly differ from miniImageNet and conduct experiments on it. These datasets include CUB, Omniglot, and GTSRB. We utilized these datasets alternately as proxy and downstream datasets to conduct comprehensive tests for our framework's generalizability. The results are shown in **table1 and table2 in 'Author Rebuttal'**, which demonstrate that **even when scaled up to larger and more diverse datasets such as those within the meta-dataset, our approach can still achieve SOTA attack efficacy across all three FSL paradigms.** This consistency in performance highlights the efficacy of our method in tackling the challenges posed by both semantic and task shifts. \n> Q2: The paper lacks discussions regarding how the proposed proxy tasks relate to/differ from the existing meta-learning UAP with the similar use of few-shot learning tasks.\n\nThank you for your valuable comment. We discussed the connections and distinctions between our proposed proxy protos and existing meta-learning-based UAP methods(e.g., LFT).\n\nIn terms of similarities, both our work and LFT share the same goal of generating UAPs that can effectively impair model performance on a wide range of unseen test images.\n\nHowever, several key differences distinguish our approach from LFT and similar meta-learning-based UAP methodologies:\n\n1. **Threat Model**: A fundamental difference lies in the threat model we adopt. Unlike LFT, which assumes access to both source data and downstream data during the generation of UAPs, our framework is proposed under a more strict attacker scenario. Specifically, our attacker is not only denied access to the source dataset but also has no visibility into the downstream data. This setup reflects the real-world scenarios where attackers may have severely limited information.\n2. **Attack Scenario**: While LFT focuses on generating UAPs tailored to attack standard classification tasks, leveraging few-shot learning techniques as part of its training regime, our method is aimed at a broader spectrum of few-shot tasks. \n3. **Application of UAPs**: Another distinction relates to how the generated UAPs are applied. In the case of LFT, even after utilizing a meta-learning scheme to generate UAPs, these perturbations need to be additionally fine-tuned on the specific downstream task using a small number of examples. Conversely, our methodology employs a generator that yields generalized UAPs capable of being deployed directly without the need for further fine-tuning on the target tasks. \n\nIn summary, while both our work and prior studies like LFT explore the generation of UAPs within the context of few-shot learning, our research introduces a stricter threat model, extends the scope to a variety of downstream tasks, and innovates by generating UAPs that do not require task-specific fine-tuning. These points of divergence highlight the novelty and significance of our contribution to the field.\n> Q3: The paper can benefit from a more comprehensive comparison against recent UAP frameworks (e.g., PAP), with more in-depth discussions.\n\nThank you for the insightful suggestion. We have incorporated the PAP method under the few-shot learning scenario and made a comparative analysis. Through the experimental results, we delve into how PAP tackles the challenges posed by task and semantic shifts. Due to the space limit, PAP results are demonstrated in **table3 and table4 in the reply to the reviewer id8n**.\n\nPAP mitigates the impact of fine-tuning by perturbing only the lower-level features. Additionally, it uses learnable Gaussian noise for data augmentation. However, it is crucial to note that **PAP achieves generalizability by sacrificing model parameters used in generating UAPs, which comes at the cost of requiring access to the base dataset and a reduction of performance on the base dataset.** In contrast, our proposed method demonstrates effectiveness not only on the base dataset but also exhibits superior cross-dataset performance without necessitating access to the base dataset. This feature enables our approach to bridge wider semantic gaps effectively. Moreover, our work uniquely addresses the task shift issue in the few-shot context, a dimension overlooked by PAP. \n> Q4:Could the improvement be due to the base distribution being more like that of the novel dataset instead of due to enhanced transferability?\n\nWe acknowledge your insightful observation. We agree that the improved performance could stem from the base distribution's inherent similarity to the novel dataset's distribution, which potentially leads to better results compared to the proxy distribution. This is the exact explanation for why Base Linear outperforms Proxy Linear. \n\nHowever, an attacker does not have access to the base distribution in practice. So, during the process of crafting UAPs, the key lies in approximating the base distribution as closely as possible to effectively generalize to the unseen novel distribution. Our approach leverages the generalization capability of the encoder to extend its understanding to the UAP. By doing so, the generalizable UAP can be obtained through various proxy datasets and models.\n\n> Q5: What’s the rationale of removing log in eq(3)?\n\nThanks for the question. We need to mention that the $log()$ function in eq(3) is not the $log()$ inside the Cross-Entropy loss, but an additional component introduced from the GAP method, aimed at boosting performance. We remove it to demonstrate the fundamental performance of our framework, ablating the effect of the additional $log()$. The most basic form of the negative cross-entropy loss is $loss = -\\mathcal{H}(f(x+\\delta), y)$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uZ3RQcDh9H", "reviewer": "Reviewer_yKwa", "summary": "The authors proposed an attacking framework to generate transferable universal adversarial perturbations on the few-shot learning scenario. The authors align proxy tasks to the downstream tasks and leverage the generalizability of pre-trained encoders to tackle the task shift and the semantic shift. Experiments show that the proposed method outperforms the existing attacking methods using CIFAR-FS as the proxy dataset.", "questions": "1.\tThe idea of the proxy dataset is good. However, the authors did not mention why the attacker cannot access the base dataset and there is no definition of the proxy dataset in the previous works (e.g. [29, 48]) of “Traditional Attack”.", "limitations": "1.\tThere are no limitations mentioned in the manuscript. The statement in Broader Impact is too weak to serve as the description of limitation. The authors should present and analyze the performance degradation such as the one of diverged proxy tasks in Fig. 6.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1.\tThe combination of universal adversarial perturbations and few-shot learning is new.", "weaknesses": "The overall writing quality of the manuscript does not meet the requirements of the NeurIPS community. Especially in terms of:\n1.\tThe confusion of chosen words. The words the authors used should be easy to understand and consistent between paragraphs, tables, and figures (e.g. “approximate the form” of the downstream tasks). It is confusing between the meaning of task and class.\n2.\tThe erroneous sentences like: “Since the attacker cannot predict the downstream tasks, we alter the tasks we use during UAP training to align them with the downstream tasks”.\n3.\tLacking sufficient explanation of the newly invented/used words. The authors need to explain the newly defined words right after using them (e.g. Base linear).\n4.\tThe unclear figures. In Fig. 4, the reviewers cannot understand the difference and relationship between Base Classifier, Proxy Linear, and Proxy Protos by looking at the different colors and lines. Fig. 5 is an imagination instead of a qualitative illustration with real results that cannot show the effectiveness of proxy protos. The legends in Fig. 6 are confusing.\n5.\tThe unclear motivation. How can the authors state that the downstream task becomes less similar to the pre-training task (64-way full shot) with higher ASR in Fig. 3 (a)? Why did the authors choose the combination of {Cifar, Mini} and {Mini, Tiered} in Fig. 3 (b)? Why the performance of the Mini proxy dataset is generally higher than the Cifar dataset? To sum up, the motivation section is misleading and should be rewritten.\n6.\tWhy using GAP in Sec. 4? Why don’t the authors use the proposed Base classifier?\n7.\tThe performance of the proposed proxy linear is not as good as base linear and there is no explanation for it, making the proposed method unconvincing.\n8.\tThe reviewers would like to see the visualization of t-SNE visualization of the feature space like Fig. 2 in [48] to demonstrate the effectiveness of the proposed proxy protos.\n9.\tCould the authors better explain the identical and different part between the proposed proxy protos and the [48] in the related work section?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your valuable comments. In the following, we respond to the concerns.\n> Q1: It is confusing between the meaning of task and class. How can the authors state that the downstream task becomes less similar to the pre-training task with higher ASR in Fig. 3 (a)? \n\nBased on the introduction of FSL(lines 91-101), 'task' means a unique classification problem that the model encounters. 'class' or 'way' denotes the individual categories within those tasks. The pre-training task is a 64-way classification problem. As the downstream tasks transition from 64-way to 5-way, the distance to the pre-training task is increased, and a decrease in the ASR can be observed. The decline in ASR suggests that the UAP's performance degrades when faced with downstream tasks that diverge significantly from its pre-training tasks, which supports our proposition of the task shift.\n> Q2:Why using GAP in Sec. 4? Why don’t the authors use Base classifier?\n\nIn Section 4, our motivation for employing GAP is to compare the conventional UAP generation approach under the scenario of FSL. This section is to examine the challenges for UAP in FSL. The Base classifier is introduced in Section 5 as part of our attack methodology. **The sequential presentation is intentional, allowing us first to explore the existing methodologies and their limitations, then emphasizing the need for our novel framework.** \n> Q3: The reviewers cannot understand the difference and relationship between Base Classifier, Proxy Linear, and Proxy Protos. Why the performance of the proposed proxy linear is not as good as base linear?\n\nThank you for the reminder. Allow us to clarify the distinctions and relationships among the terms.\n- **Base Classifier**: As defined in line 192 of our paper, this refers to the classifier pre-trained on the base dataset. It serves as our starting point before further adaptations.\n- **Proxy Linear**: As mentioned in line 201, the Proxy Linear classifier is trained through proxy tasks. This is a key step to handle the task shift.\n- **Base Linear**: Defined around line 229, this refers to the linear classifier that trained on tasks sampled from the based dataset, which is a further improvement from the proxy linear classifier as it diminishes the semantic gap from proxy dataset to base dataset.\n- **Proxy Protos**: Defined at line 253, this represents our final approach, where we employ prototypes derived from the proxy tasks to enhance the performance.\n\n> Q4:Why the attacker cannot access the base dataset?There is no definition of the proxy dataset in the previous works\n\nRegarding the attacker's inability to access the base dataset, **we have outlined this threat model in Section 3.1, where we introduce a more challenging scenario**(same threat model with [48]). This setup reflects real-world conditions more accurately, as a large amount of data, especially proprietary corporate datasets, are typically not publicly available due to privacy concerns and intellectual property rights.\n\nAs for the definition of the proxy dataset, **we would like to clarify that it has been introduced in reference [45]**. This prior work conducts a comprehensive analysis of the mutual information between images and perturbations, thereby establishing the feasibility of employing a proxy dataset to perform the UAP attack.\n> Q5: Why did the authors choose the combination of {Cifar, Mini} and {Mini, Tiered}? Why the performance of the Mini proxy dataset is generally higher than the Cifar dataset? \n\nThank you for your question. As mentioned in lines 165-170, the victim model is trained on MiniImageNet. Consequently, when the proxy dataset coincides with MiniImageNet, it simulates a scenario where the attacker has access to the pre-training dataset, which is advantageous and leads to generally higher performance due to the diminished semantic shift between the proxy and the base dataset.\n\nThe choice of different datasets is intentional and serves to illustrate four distinct scenarios. {CIFAR, Mini} represents the cases that the proxy dataset matches (Mini) and doesn't match(CIFAR) the pre-training dataset. {Mini, Tiered} represents the cases that the downstream dataset matches (Mini) and doesn't match(Tiered) the pre-training dataset.\n\nWhen CIFAR is selected as the proxy and Tiered as the downstream, it embodies the most general and challenging scenario where the attacker suffers both semantic shift from the pre-training to the proxy and also to the downstream dataset. Conversely, using MiniImageNet for both proxy and downstream datasets reflects an idealized condition for the attacker, leading to optimal attack performance.\n> Q6: The legends in Fig. 6 are confusing.The authors should analyze the performance degradation in Fig.6.\n\nThank you for the reminder, we have revised the presentation. The analysis is provided in lines 331-339. From fig6, we observe that the ASR remains relatively high when transitioning from 5-way 1-shot to 25-way 1-shot, and from 5-way 1-shot to 5-way 20-shot scenarios(5-1 and 5-5 for downstream tasks). Meanwhile, a noticeable degradation in performance emerges as the ways and shots deviate further from the downstream tasks. This aligns with our task shift claim and suggests that a rough estimation of the downstream tasks is sufficient for an attacker to discern the task bias and produce transferable UAPs.\n> Q7:What is identical and different part between the proposed method and the [48]?\n\nThanks for the suggestion. While our threat model aligns with that of [48], we have introduced key advancements in addressing the two shifts:\n1. Task Shift: Unlike [48], our method is designed to handle scenarios comprising diverse downstream tasks with limited samples, which [48] doesn't take into consideration.\n2. Semantic Shift: [48] employs contrastive learning loss to mitigate semantic shift, our ablation studies (lines 322-329) reveal that our strategy outperforms that using contrastive losses in the FSL scenario.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for the detailed and constructive reviews. We have revised the paper based on your suggestion. Here are some highlights in the paper revision:

1. **Expanded Experimental Scope**: We have augmented our study with additional experiments conducted on the CUB, Omniglot, and GTSRB datasets, utilizing them alternately as proxy and downstream datasets. We chose three victim models as representatives of finetuning, meta, and metric-based FSL paradigms, and compared them with three SOTA methods. The results, summarized as follows, offer a broader perspective on the generalizability of our proposed method.

   **Table1: 5-way 1-shot results.**

   | proxy    | downstream | method     | baseline       | anil           | protonet       |
   | -------- | ---------- | ---------- | -------------- | -------------- | -------------- |
   | CUB      | mini       | UAN        | 52.47±0.31     | 62.85±0.27     | -              |
   |          |            | GAP        | 68.47±0.31     | 67.47±0.28     | -              |
   |          |            | AdvEncoder | 76.57±0.31     | 72.86±0.27     | 70.80±0.36     |
   |          |            | **ours**   | **81.17±0.30** | **77.83±0.28** | **71.22±0.36** |
   | Omniglot | mini       | UAN        | 44.90±0.30     | 37.58±0.26     | -              |
   |          |            | GAP        | 60.95±0.33     | 63.82±0.35     | -              |
   |          |            | AdvEncoder | 74.16±0.33     | 47.97±0.34     | 54.72±0.31     |
   |          |            | **ours**   | **76.32±0.31** | **77.81±0.28** | **67.70±0.35** |
   | GTSRB    | mini       | UAN        | 54.45±0.32     | 50.17±0.26     | -              |
   |          |            | GAP        | 54.09±0.33     | 68.22±0.31     | -              |
   |          |            | AdvEncoder | 67.24±0.32     | 62.86±0.30     | 67.16±0.34     |
   |          |            | **ours**   | **79.99±0.30** | **78.13±0.20** | **69.17±0.38** |
   | mini     | CUB        | UAN        | 76.82±0.36     | 73.75±0.48     | -              |
   |          |            | GAP        | 69.70±0.35     | 75.08±0.45     | -              |
   |          |            | AdvEncoder | 76.21±0.35     | 76.82±0.56     | 71.40±0.40     |
   |          |            | **ours**   | **79.22±0.35** | **78.09±0.58** | **71.60±0.40** |
   | mini     | Omniglot   | UAN        | 36.20±0.40     | 56.63±0.59     | -              |
   |          |            | GAP        | 20.09±0.32     | 25.40±0.33     | -              |
   |          |            | AdvEncoder | 64.41±0.41     | 66.27±0.65     | 71.80±0.38     |
   |          |            | **ours**   | **76.44±0.27** | **69.64±0.64** | **75.27±0.36** |
   | mini     | GTSRB      | UAN        | 74.86±0.34     | 79.82±0.33     | -              |
   |          |            | GAP        | 73.30±0.34     | 79.04±0.36     | -              |
   |          |            | AdvEncoder | 77.18±0.33     | 78.27±0.37     | 76.71±0.37     |
   |          |            | **ours**   | **80.29±0.30** | **80.31±0.36** | **78.41±0.35** |

   **Table2: 5-way 5-shot results.**
   | proxy    | downstream | method     | baseline       | anil           | protonet       |
   | -------- | ---------- | ---------- | -------------- | -------------- | -------------- |
   | CUB      | mini       | UAN        | 45.93±0.28     | 63.53±0.24     | -              |
   |          |            | GAP        | 65.91±0.24     | 67.58±0.25     | -              |
   |          |            | AdvEncoder | 73.67±0.23     | 72.67±0.22     | 71.99±0.26     |
   |          |            | **ours**   | **78.29±0.21** | **78.25±0.20** | **72.24±0.28** |
   | Omniglot | mini       | UAN        | 37.34±0.27     | 35.05±0.26     | -              |
   |          |            | GAP        | 57.22±0.29     | 61.09±0.31     | -              |
   |          |            | AdvEncoder | 72.07±0.23     | 44.73±0.31     | 57.38±0.26     |
   |          |            | **ours**   | **74.46±0.21** | **77.87±0.21** | **69.05±0.26** |
   | GTSRB    | mini       | UAN        | 48.46±0.29     | 48.73±0.26     | -              |
   |          |            | GAP        | 48.68±0.29     | 65.94±0.28     | -              |
   |          |            | AdvEncoder | 63.99±0.27     | 60.51±0.27     | 68.96±0.24     |
   |          |            | **ours**   | **77.55±0.20** | **77.93±0.28** | **69.98±0.30** |
   | mini     | CUB        | UAN        | 77.42±0.20     | 64.42±0.58     | -              |
   |          |            | GAP        | 68.90±0.27     | 68.89±0.52     | -              |
   |          |            | AdvEncoder | 75.29±0.22     | 67.31±0.68     | 72.80±0.27     |
   |          |            | **ours**   | **77.46±0.21** | **69.43±0.71** | **72.86±0.27** |
   | mini     | Omniglot   | UAN        | 26.15±0.39     | 52.25±0.60     | -              |
   |          |            | GAP        | 10.14±0.23     | 21.69±0.32     | -              |
   |          |            | AdvEncoder | 60.57±0.44     | 62.99±0.66     | 68.25±0.34     |
   |          |            | **ours**   | **75.04±0.25** | **67.52±0.67** | **73.15±0.30** |
   | mini     | GTSRB      | UAN        | 74.27±0.23     | 78.31±0.28     | -              |
   |          |            | GAP        | 68.57±0.29     | 77.77±0.29     | -              |
   |          |            | AdvEncoder | 73.96±0.26     | 77.50±0.29     | 78.38±0.23     |
   |          |            | **ours**   | **77.22±0.24** | **78.50±0.28** | **78.77±0.23** |

2. **Enhanced Comparative experiments**: We have included additional comparative studies, including a comparison with the PAP method across multiple datasets and an exploration of the GAP approach in metric-based paradigms. 

3. **Improved paper presentation**: We have made additional modifications and clarifications and double-checked the paper writing for a better presentation. 

In the following, we respond to all concerns one by one.

Meta-review: This paper studies universal adversarial perturbations (UAP) for the few-shot learning (FSL) scenarios. Overall, all reviewers acknowledge the novelty and importance of the paper's contribution to addressing the "less-studied" UAPs in FSL scenarios, and appreciate the simplicity and effectiveness of the proposed method. But meanwhile, some concerns are raised, mainly regarding 1) more (diversed) datasets should be evaluated; 2) comparison to some more recent UAP methods should be included; and 3) some technique aspects should be further clarified and the overall presentation should be improved.

These concerns are well addressed in the rebuttal, and all reviewers stand on the positive side of this submission. The final decision is accept.
Decision: Accept (poster)