Attacking Few-Shot Classifiers with Adversarial Support Sets
Authors: Elre Talea Oldewage
John F Bronskill
Richard E Turner
Venue:  (2021)
Keywords: meta-learning
few-shot learning
adversarial attacks
poisoning

Abstract: Few-shot learning systems, especially those based on meta-learning, have recently made significant advances, and are now being considered for real world problems in healthcare, personalization, and science. In this paper, we examine the robustness of such deployed few-shot learning systems when they are fed an imperceptibly perturbed few-shot dataset, showing that the resulting predictions on test inputs can become worse than chance. This is achieved by developing a novel Adversarial Support Set Attack which crafts a poisoned set of examples. When even a small subset of malicious data points is inserted into the support set of a meta-learner, accuracy is significantly reduced. For example, the average classification accuracy of CNAPs on the Aircraft dataset in the META-DATASET benchmark drops from 69.2% to 9.1% when only 20% of the support set is poisoned by imperceptible perturbations. We evaluate the new attack on a variety of few-shot classification algorithms including MAML, prototypical networks, and CNAPs, on both small scale (miniImageNet) and large scale (META-DATASET) few-shot classification problems. Interestingly, adversarial support sets produced by attacking a meta-learning based few-shot classifier can also reduce the accuracy of a fine-tuning based few-shot classifier when both models use similar feature extractors.

Under review as a conference paper at ICLR 2021
ATTACKING FEW-SHOT CLASSIFIERS WITH ADVERSARIAL SUPPORT SETS
Anonymous authors Paper under double-blind review
ABSTRACT
Few-shot learning systems, especially those based on meta-learning, have recently made significant advances, and are now being considered for real world problems in healthcare, personalization, and science. In this paper, we examine the robust- ness of such deployed few-shot learning systems when they are fed an imper- ceptibly perturbed few-shot dataset, showing that the resulting predictions on test inputs can become worse than chance. This is achieved by developing a novel Ad- versarial Support Set Attack which crafts poisoned set of examples. When even a small subset of malicious data points is inserted into the support set of a meta- learner, accuracy is significantly reduced. For example, the average classification accuracy of CNAPS on the Aircraft dataset in the META-DATASET benchmark drops from 69.2% to 9.1% when only 20% of the support set is poisoned by im- perceptible perturbations. We evaluate the new attack on a variety of few-shot classification algorithms including MAML, prototypical networks, and CNAPs, on both small scale (minilmageNe)) and large scale (META-DATASER classification problems. Interestingly, adversarial support sets produced by at- tacking meta-learning based few-shot classifier can also reduce the accuracy of a fine-tuning based classifier when both models use similar feature extractors.
few-shot
INTRODUCTION
Standard deep learning approaches suffer from poor sample efficiency (Krizhevsky et al.) 2012) which is problematic in tasks where data collection is difficult or expensive. Recently, few-shot learners have been developed which address this shortcoming by supporting rapid adaptation to new task using only few labeled examples (Finn et al.) 2017 Snell et al. 2017]. This success has meant that few-shot learners are becoming increasingly attractive for real-life applications. They have been applied to user personalization in recommender systems (Lee et al. 2019). matching potential users to businesses (Li et al., 2020), personalized talking head models (Zakharov et al., 2019), and on-device gaze estimation (He al., 2019). As few-shot leamners improve, they also being applied to increasingly sensitive applications where the repercussions of confidently- wrong predictions glaucoma diagnosis (Kim et al. 2017). identification of diseases in skin lesions (Mahajan et al. 2020), and tissue slide annotation in cancer immuno-therapy biomarker research (Lahiani et al. 2018).
severe. Examples include clinical risk assessment (Sheryl Zhang al. 2019).
As few-shot learners gain popularity, it is essential understand how robust they and whether there are potential avenues for their exploitation. It is well known that standard classifiers are vul- nerable to inputs that have been purposefully modified in a minor way to cause incorrect predictions (Biggio & Roli, 2017). Such examples may be presented to model either at test time, called eva- sion attacks (Biggio et al. 2017) or adversarial examples (Szegedy et al., 20144), or at training time, which is referred to poisoning (Newsome et al., 20066; Rubinstein et al. 2209). While previous work has considered adversarial attacks on few shot learners, data poisoning attacks have not been studied and are the focus of this paper.
Data poisoning attacks are of particular relevance in the few-shot learning setting for two reasons. First, since the datasets small, a handful of poisoned patterns might have significant effect. Second, many applications of few-shot learning require labeled data from users to adapt the system new task, essentially providing direct interface for outsiders to influence the model's behaviour.

---

Under review as a conference paper at ICLR 2021
If sew-shot learning systems are not robust to poisoning of their training dataset, then this weakness could be exploited. An attacker performing a man-in-hhe-middle (Conti et al.) 2016) data poison- ing attack could cause recommender system's 's personalization to perform badly or suggest certain results to influence a user's decision. Applied at scale to many users, an attacker could cause signifi- cant damage. Similarly, doctor attempting to commit medical insurance fraud may submit images causing a benign skin condition to be incorrectly classified as a skin disease requiring expensive treatment; malicious party may ruin research study that uses automated annotation of samples by tampering imperceptibly with only few images. If these attacks could be achieved with mali- cious patterns that cannot be reliably distinguished from real training data, it would be difficult to defend against them.
Before detailing the key contributions of the paper, necessary to briefly introduce the lexicon of few-shot learning. During training, few-shot learners are typically presented with many different tasks. The model must learn to perform well on each task, hopefully arriving at a point where it can adapt effectively to new task at test time. At test time, the model is presented with an unseen task containing few labeled examples, the support set, and a number of unlabeled examples to classify, called the query set. The paper makes the following contributions:
1. We define novel attack on few-shot classifiers, called an Adversarial Support Set Attack, which applies adversarial perturbations to the support set that are calculated to minimize model accuracy over a set of query points. To the best of the authors' knowledge, this the first work considering the impact of poisoning attacks trained few-shot classifiers.
2. We demonstrate that few-shot classifiers surprisingly vulnerable Adversarial Support Set attacks. The adversarial support set attack is more effective than the baselines consid- ered, and generalizes well, i.e. the compromised classifier is highly likely to be inaccurate on a randomly sampled query set from the task domain.
3. We demonstrate the effectiveness of our approach against variety of few-shot classifiers including MAMI. (Finn et al., 2017), ProtoNets (Snell et al. 2017), and CNAPs (Re- queima et al. 2019a), on both small scale (minnimageNee ( (Vinyals et al., (016)) and large scale (META- DATASET (Triantafíllou et al. 2020)) few-shot classification benchmarks.
4. We show that adversarial support sets transfer effectively to fine-tuning based few-shot classifiers when the few-shot classifier and the fine-tuner utilize similar feature extractors.
The rest of the paper proceeds as follows: Section 2 provides background about the meta-learning models under consideration, relevant adversarial attack methods, and the threat model under con- sideration. Section discusses how evasion and poisoning attacks may generalized to few-shot learners. Section
4 presents the experimental results and Section concludes the paper. Additional results and experimental details are in the Appendix.
2 BACKGROUND
In this section we lay the necessary groundwork for adversarial support set attacks. We focus on image classification. We denote input images 2 E Bhhx where W is the image width, H the image height, ch the number of image channels and image labels y E (1,...,C) .C where C is the number of image classes. We use bold x and y to denote a set of images and labels, respectively.
2.1 META-LEARNING
We consider the few-shot image classification scenario using a meta-learning approach. Rather than a single, large dataset D, we assume access to a dataset D (r. { comprising large number of training tasks Te, drawn i.i.d. from a distribution p(r). The data for task consists of a support set Dg = (In, Fo Mo)) 1 comprising N elements, with the inputs I, and labels yn. observed, and )
a query set Dq = {1
- 2/hh )M with M elements for which we wish to make predictions. We
may use the shorthand Ds = xx, ÿ) and Dę = {x", y²} for brevity. Here the inputs x" are observed and the labels y" are only observed during meta-traie traitangaiinggggg (i.e. training of the meta-learning algorithm). Note that the query set examples are drawn from the same set of labels as the examples in the support set. At meta-test time, the classifier f is required to make predictions for query set
2

---

Under review as a conference paper at ICLR 2021
Support Set
N
2
3
Query Set Dq
M
x
y
stopwatch
clock
watch
watch
stapwatch
clock
Figure I: Example task with C' classes with N and M 4.
inputs of unseen tasks. Often, the meta-test tasks will include classes that have not been seen during meta-training, and Dy will contain only few observations. An example task is shown in Fig. T
Episodic Training The majority of modern meta-learning methods employ episodic training (Vinyals et al.) 2016). During meta-training, task 7 is drawn from p(r) and randomly split into support set Ds and query set Do- The meta-learner g takes as input the support set Ds and pro- duces task-specific classifier parameters 3 g(Ds) which
used to adapt the classifier f to the current task. The classifier can now make task-specific predictions f(x", v = g(Ds)) for any test input x" E Do. Refer to the diagram labeled Clean in Fig. 2 A loss function L(f(" <b), then computes the loss between the predictions for the label f( 36) and the true label y". Assuming that C. and 9 are differentiable, the meta-learning algorithm gradient descent by back nack-propagating the loss and updating the parameters of and g.
then be trained with stochastic
Common Few-shot Learning Algorithms There has been an explosion of meta-lcarning based few-shot learning algorithms proposed in recent years. For an in-depth review see HHssedales et al. (2020). Here, we briefly describe methods that are relevant to our experiments. Arguably the most widely used is the gradient-based approach, the canonical example for modern systems be- ing MAML (Finn et al.] 2017). With MAML, the task-specific parameters &b are the parameters of the classifier after applying other widely used class of meta-learners are ammoriized-inferennce or black box based approaches e.g. VERSA (Gordon et al.] 2019) and CNAPs (Requeima al. 2019a). In these methods, the task-specific parameters &b are generated by one or more hyper-networks, (Ha et al. 2016). An important special case of this approach Prototypical Networks (ProtoNets) (Snell al. 2017) which is based on metric learning and employs a nearest neighbor classifier, and the task speciipeiific parameters t/ are the mean feature vectors for each class c E C of the support set Ds- Finally, very competitive, non-episodic approach to few-shot learning is simple fine-tuning (Yosinski et al.) 2014 Tian et al. 2020). In this approach, pretrained feature extractor is used combination with final layer linear classifier, whose weights serve the task task-specific parameters zb that are learned by taking a number of gradient steps on the support set data.
more gradient steps taken on the support set Ds- An-
2.2 ADVERSARIAL ATTACKS
Szegedy al. (2014) have shown that it is possible to craft adversarial examples, inputs that are indistinguishable from legitimate examples to both humans and algorithms (Carlini & Wagner 2017). yet are classified incorrectly by deep neural networks with high confidence (Nguyen et al. 2014). Adversarial examples are typically generated by taking clean, unperturbed input image and adding a small amount of carefully crafted, almost imperceptible noise (Goodfellow et al. 2014 Madry al. 2017). The perturbation is calculated by performing the following optimization arg max, 2(0, +6, 8, where L is the loss function e.g. the cross entropy, denotes the model parameters, x is the input being perturbed, and is its label. The perturbation size is typically con- strained by some norm such as foo or lg. As stated above, the optimization produces an untargeted attack, in the sense that we do not require the generated image to be misclassified as any particu- lar class, only to be classified incorrectly. An alternative formulation of the optimization problem, arg min; L(O, x + 5. ū)), would produce targeted attack, i.e. an image that is misclassified as be- longing specifically to the class with label ÿ generated by shifting the clean class label by 1 mod C. Many algorithms can be used to generate adversarial examples, including Projected Gradient Descent (Madry et al.] 2017). the Carlini and & attack (Carlini Wagner] 2017). or ElasticNet (Chen et al. 2017a). Throughout this paper, without loss of generality,
In this paper, labels for targeted attacks are
Wagner
utilize Projected Gradient Descent (PGD) with an lo norm, because simple to implement, inexpensive to compute, and effective. We show that by careful application of PGDD, we can successfully poison meta-learners meta-test time.
3

---

Under review as a conference paper at ICLR 2021
2.3 THREAT MODEL
The threat model may be summarized in terms of the adversary's 's goal, knowledge and capabilities.
Goal The adversary could aim to compromise system's integrity, availability or confidentiality. Poisoning attacks may aim to compromise system integrity for example, backdoor poisoning (Chen et al. 2017b), where the aim is to misclassify only a specific query point at test time. In this paper, we consider poisoning attacks that reduce system availability, where the goal is simply to maximize system failure on any query point. Evasion attacks usually aim to compromise the integrity of the system by causing failure specific query points test time.
Knowledge We assume the adversary has full knowledge of the model's internal workings - in- cluding gradients and other internal state information. We also assume that they can access enough data to be able to form a seed query set of at least the same size as the adversarial support set.
Capabilities When performing adversarial support set attacks, we consider adversary who is able to manipulate some fraction of the support set. We constrain the attack-space by requiring that the adversary's 's modifications must be imperceptible. We achieve this by constraining the adversarial perturbation to be within some of the original image, measured using the Eo norm. As baseline, also considered an attacker who is able modify support set labels instead. See Other Possible Attacks in Section 3 for more details on this attack. When performing query attacks, we consider an adversary who is able to manipulate one or more points in the query set, again constrained by a maximum perturbation size.
ADVERSARIAL ATTACKS ON FEW-SHOP LEARNERS
In this section, we introduce a variety of attacks that can be perpetrated against few-shot learners, summarized in Fig. 2 Consider a few-shot learning system that has been trained and deployed as a service. In this service, users or groups of users provide their data as input to the service, either explicitly such as tagged photos or implicitly such as product preferences or browsing habits. This data forms support set and is used by the system to produce task-specific parameters which are then used to make predictions on novel queries from the users. malicious user, or a man-in- man-in-the-mice attacker could attack the system in number of ways, discussed below.
Query Attack The attacker may want the adapted classifier to misclassify specific input image. This corresponds to solving arg max, [(f(x* + 5, g ax, 9)), y²). Refer to Appendix AA. for details. These kinds of attacks relate adversarial examples considered in recent literature, in the context of evasion attacks (Biggio et al. 2017). Query attacks have been perpetrated successfully against few-shot learners (Goldblum al. 2019 Yin et al. 2018). but are not the main focus of this work.
Support Attack The attacker may want the system to fail on any query image. The attacker will achieve this by computing a perturbed support set Ds = (ë, y whose inputs are optimized to fool the system on a specific query set, which we call the seed query set, with the goal of generalizing to unseen query sets. This corresponds to solving arg max; [(f(x", ,g(x + 5, y)), y²) such that || < where DQ = fx= 3²} denotes the seed query set and is the maximum size of the perturbation. Refer to Algorithm [ for details. We call this novel few-shot learner attack an Ad- versarial Support Set Attack, or simply support attack. Our attack is poisoning attack, since the attacker is manipulating data that the model will use to do inference. However, it is important to note the the attack perpetrated at meta-test time, after the meta-learner has already been meta-trained. Example images from a support attack are shown in Fig. 3
In contrast to a query attack, which only considers the model's behaviour on single point at time, support attacks allow the adversarial optimization function to incorporate information regarding the model's 'ss behaviour on the entire query set. In real settings, an attacker might design an attack on their own query set, hoping it will generalize to other, unseen queries. The ability to generalize will depend on M, the size of the seed query set.
Adversarial support sets vs evasion attacks The adversarial support set attack differs from an evasion attack in the following ways: (i) We generate entire adversarial support set (or subset of
4

---

Under review as a conference paper at ICLR 2021
x PGD Po.f-g)
x PGDQ(Ds. Dof.9)
--(( y 4p
y
X
V/
D3 (x. y)
D3 (3. y
D3 (x, 9
Clean
Support
Query
Yn= Yutt mod C
x = PGDQ(DSs
X = PDs Do.
y
x²
1/
Dg- {x - 0
D3 (x* y²)
8
D2 y3
Label Shift
Swap
Transfer
Range of considered attacks on meta-learning based few-shot image classifiers. f and denote the classifier and trained meta-leamer, respectively. Each diagram depicts how an attack is applied and includes an expression for the attack's computation. Adversarially perturbed quantities are denoted with a tilde. The prime ()) marks in the Transfer attack indicate that the attack is transferred to a different meta-learner and classifier than the attack is derived from.
Figure 2:
Figure 3: Pairs of images from the minilmageNet dataset where the left is unperturbed. while the right is adversarially perturbed by PDG Support attack with 0.05, ? 0.0015, and L 100.
the support set) once, rather than generating just single adversarial input. (ii) The attack is con- structed with regard to an entire query dataset, rather than a single query point. Moreover, the goal not for single specific query to be misclassified, but for all future unseen queries be misclas- sified. (iii) Generation of the attack involves backpropagating the gradients through meta-learner which itself performing gradient based learning in the case of MAML, or is complex set neural network in the case of ProtoNets or CNAPs. In this context, it is prudent to use fairly simple attack method in the face of significant increased complexity over the standard evasion attack setting.
Other Possible Attacks We formulate a number of other few-shot learner attacks, also depicted in 2 Label Shift is simple attack on the support set which involves mislabelling the support set images by shifting the true label index by one in a modulo arithmetic fashion. We consider systematic mislabeling in this way to be strong attack for comparison, though this "attack" may be easily detected in practice. We also consider Swap attacks, a support set poisoning attack where set of images are perturbed with query attack and then inserted into the support set. Query attacks are typically cheaper to compute, since they do not require back-propagation through the meta-learner. Swap attacks thus a strong baseline. See Section for further details of the swap attack's 's implementation. Lastly, we also formalize transfer attacks in the context of few-shot learning, as shown Fig. [] Transfer, where the adversarial support set is computed on a first few- shot classifier system and is then applied to a different few-shot classifier. An attacker wishing to target specific system may perform a transfer attack from a surrogate system to the target the target's 's internal state or inner workings are unknown, or if attacking the surrogate is computationally cheaper than an attack against the target directly. In particular, our experiments transfer attacks from meta-learning based few-shot learners an expensive fine-tuning algorithm.
Related Work While there has been previous work on adversarial attacks against few-shot learn- ing systems (Goldblum et al.] 2019 Yin et al., 2018), attacks that poison the support set have re- ceived little attention. Goolblum al. (2019) devise technique called adversarial querying which significantly improves robustness against query attacks (refer to Fig. 2 They do not evaluate ro- bustness against support set attacks meta-test time, though they do test attacking the support set
5

---

Under review as a conference paper at ICLR 2021
Algorithm PGD for Adversarial Support Set Attack
Require:
1:
procedure PGDS(Ds,, De,
min:: Minimum image intensity
2:
8 ~~ (~(EE))))
Imaz Maximum image intensity
3:
f- clip(x + 5. Imins Imax) for i E l..... L do
L: Number of iterations
4:
6: Perturbation amount
5:
ô - [gn(VaL(Je
g( y)).
7y: Step size
6:
. < clip( @ y0, min: max)
Ds {x. y
7:
. f- a + clip( 2, E,
DQ =
8:
end for
> We use cross-entropy loss for L.
9: return æ
10: end procedure
along with the query set during meta-training. Yin et al. (2018) also describe a meta-training regime to increase robustness against query attacks, but only consider the relatively weak Fast Gradient Sign Method attack (Goodfellow et al., 2014). Edmunds et al. (2017) explore the transferability of query attacks between tasks on meta-traings traine models and find that the attacks are indeed highly trans- ferable, though their experiments were restricted to the MAMI. algorithm and the relatively simple Omniglot dataset. In this work, we focus on the effects of support set attacks on the performance of few-shot image classifiers using variety of learning algorithms and challenging datasets.
EXPERIMENTS
In this section we present our experiments that endeavor to answer the following questions: (i) How vulnerable are few-shot classifiers to adversarial support set attacks? (ii) What are the most effective settings for the support set attack parameters? (iii Do adversarial support set attacks crafted on a meta-learning based few-shot classifier transfer to fine-tuning based few-shot classifier? All of our experiments are carried out on meta-trained few-shot classifiers. Refer to Appendix [A. for details on the meta-training protocols. We divide the experiments into small-scale the minelmageNet dataset (Vinyals et al. (2016) using the MAML and ProtoNets algorithms, and large-scale on META- DATASET (Triantafillou et al. 2020) using the CNAPs algorithm
When perpetrating support attacks, we consider two variations of the loss function: all, in which the entire query set is used; and single, in which a single, random point in the query set is chosen for the loss calculation at each attack iteration, with the intention that the additional stochasticity may prevent the attack from getting stuck in local optima. We found that the all strategy performed best when combined with a targeted attack, whereas the single strategy worked better in combination with an untargeted attack. Unless otherwise specified, we use the targeted, all loss strategy.
For comparison purposes, we also consider uniform random noise. Uniform noise is added to an image æ as follows: = x + u, where u ~ U (- and U denotes the uniform distribution.
4.1 SMALL SCALE EXPERIMENTS
Attack Comparison Fig. depicts the relative decrease in 5-way classification accuracy due to a variety of attacks for two values of e. We compute the percentage relative decrease in classification accuracy as follows: 100% x (Aetean - -attacck) /Oddenn where dsteon is the clean classification accu- racy, and Gattack is the classification accuracy after the attack. We consider the model's performance over 500 randomly generated tasks. Each task is constructed by choosing the required number of classes from the dataset labels at random, then choosing the required number of images randomly within each class. Each task is composed of a support set, a seed query set and 50 unseen query sets used for evaluation of the attack. Although the shot of the seed query set may be varied, the query sets used for evaluation are the same size as the support set. The unseen query sets are all guaran- teed to disjoint from the seed query set to avoid information leakage. We proceed as follows: for each task, we generate an adversarial support set using the corresponding seed query set. We then evaluate the adversarial support set on the task's 's 50 unseen query sets. Note that we make no further
Source code for all experiments will be made available upon publication.

---

Under review as a conference paper at ICLR 2021
Label Shift
Noise
Support Specifiic
Support General (1x)
Support Generral (10x)
Swap
100.0
100.0
100.5
Relative Decrease Accuracy (%)
60
71.6
59.6
65.0
77.8
71.6
63.5
76.7
61.7
36.0
70.1
76.1
40-
20-
ProtoNets (Clean 46.4%) 4%)
MAML (Clean 46.8%)
ProtoNets (Clean 46 46.4%)
MAML (Clean 46.
(a) epsilon: 0.0314
epsilon:
100.0
100.0
83.3
69.5
85.
53.9
39.3
86.9
sa-
69.7
60-
60
9.7
ProtoNets (Clean 64.7%)
60.8%)
ProtoNets (Clean 64.7%)
(Clean
(c) epsilon: 0.0314
epsilon: 0.05
Figure 4: The relative drop in classification accuracy for a variety of attacks against MAML and ProtoNets models in the 5-way minilmageNet configuration, averaged over 500 tasks. For Support General (Ix), M=N, and for Support General (10x), M=1ON. All support images were perturbed. PGD settings were L=100, with 0.0015 for 0.05, and y 9.4e-4 for 0.0314.
changes to the adversarial support set when evaluating it on the unseen query sets. These unseen query sets also used to evaluate the other attacks and baselines included in Fig. We refer to the average classification accuracy on the seed query sets as the Support Specific attack accuracy, whereas the average classification accuracy when evaluating the attack unseen query sets is called the Support General attack accuracy. We include two different results for the Support General scenario: Support General (Ix), for which the seed query size is the same size as the support set (i.e. M = N) and Support General (10x), where M = 10N.
random sample of 50
When perpetrating swap attack a given task, Fig. 2 Query to generate a query attack. We then "swap" the role of the sets so that the adversarial query is presented the model to learn from i.e. as a support set. We evaluate the swap attack on 50 independent query sets, in the same way as the Support General scenario.
the task's support set and query
in
The Support Specific attack is 100% effective in all cases, indicating that the adversarial objective function is acting as desired. The attack's generalization to unseen query sets, as measured by the Support General (x10) attack, successfully beats all the baselines. The Support General (x/) attack, is noticeably weaker, but still beats the Swap baseline on Protonets. Since are measuring generalization these scenarios, is not unexpected that the attack with less data learn from does not generalize well. We further examine the effect of seed query set size later in this section. The Label Shift attack causes a large drop in accuracy, but is casily detected by inspection, and so does not provide like-for-like comparison. We consider the Swap attack, which is effective and simpler to compute than the adversarial support set attack, to be the strongest competitor. From Fig. 4 there does not appear to be difference between ProtoNets and MAML in terms of robustness to the adversarial support set attack, though MAML is significantly more vulnerable to the swap attack. For 5-shot classification, increasing the perturbation size does consistently increase the attack's impact. This is because the step size was tuned for the 1-shot case, but was not optimal for the 5-shot problem. For consistency, we used the same PGD settings for both in Fig, (details in A.3).
Since we are primarily interested in degrading model accuracy for unseen inputs, we consider only the Support General case for the remainder of the results presented, unless otherwise specified.
Fraction of Poisoned Patterns A realistic scenario may constrain attacker to perturb only fraction of the images in the support set. We consider two dimensions: specifying the fraction of classes that are adversarially perturbed and, within those classes, how many of the shots are poisoned. Fig. 5a Support shows the relative drop in 5-way classification accuracy for ProtoNets function of the number of poisoned classes and shots in support set. As expected, the attack strength increases as the number of poisoned shots are increased, though perturbing even one point causes significant drop in accuracy. We also see that the attack is stronger when the poisoned points

---

Under review as a conference paper at ICLR 2021
Poisoned Classes
Poisoned Classes
Polsoned Classes
-. Decroast in Accuracy 196)
5
5
Polsaned Shots
1-
12.8
38.8
71.9
1- 11.8
40.7
1- 2.0
6.7
3-
24.5
58.6
88.2
3
13.0
46,9
66.8
3-
12.7
39.8
66.0
26.3
61.4
89.7
13.0
47.1
69.7
20.8
54.4
90.1
1-chet MAMI 5-ehot
Support
Swap
Subselected
180
Seed Query Siee
Al
200 +is
300
(a)
(b)
Figure 5: (a) The relative drop in 5-way, 5-shot classification accuracy of ProtoNets the number of poisoned classes and poisoned shots within those classes are varied using three different methods of crafting the poisoned images. Darker colors indicate stronger attack. Attacks were calculated with e = 0.05, 7 0.0015, L 200, M 13N, averaged over 250 tasks. (b) Relative drop in 5-way classificatton accuracy for ProtoNets and MAML using a support attack as M is varied, for 1 I-shot and 5-shot. Both the targeted, all and untargeted. single loss strategies were considered and the more effective was chosen for each model. The attacks are generated using PGD with e = 0.05, 0.0015, I. 100, and half the classes and shots in the support set adversarially perturbed.
are spread across classes, e.g. perturbing just one image from every class causes 72% accuracy drop, whereas perturbing all five shots of one class only causes 26% accuracy drop.
For comparison, Fig. 5a Swap shows the effect of performing a swap attack with varying fractions of poisoned classes and shots. Poisoning the support set with a single malicious point, either an adversarial support pattern or an adversarial query pattern achieves similar drops in model accu- racy. However, increasing the number of poisoned shots for particular class is significantly more effective for the support set attack than the swap attack because the adversarial support set's joint optimization allows for collusion among its images. We confirm this with an additional experiment shown in Fig. Sa Subselected, where we generate an adversarial support set with all patterns poi- soned, then select subsets of these perturbed patterns poison the original support set. Even though the Support and Subselected attacks have the same fraction of poisoned patterns, the support set attack performs significantly better as is jointly optimized for a specific fraction be poisoned, whereas the patterns in the Subselected attack have been separated from their colluding images.
Seed Query Set Size Fig. 5b shows that in a low-shot scenario, attacker can obtain sig- nificantly stronger attack by increasing the seed query set's size, i.e. by increasing the number of examples that the adversarial support set is optimized to fool. The effectiveness of the attack using the ProtoNets algorithm plateaus at M = 100 for 1-shot and around M = 250 for 5-shot, whereas the effectiveness of MAML algorithm attack increases steadily with seed query set size. Fig. (bb) also shows that ProtoNets is more robust than MAMI. ProtoNets is generally more capable clas- sifier than MAML (Triantafilluu et al. 2020). Additionally, MAMI uses gradient steps to adapt to new tasks which can be noise-sensitive, whereas ProtoNets computes the mean of the feature vectors for each class in the support set and this averaging tends to reduce noise.
4.2 LARGE SCALE EXPERIMENTS
We now attack the CNAPs algorithm when performing classification on a subset of the META- DATASET constituent datasets (refer to Appendix I. details on our use of META- -ATASET) When following the META-DATASET protocol, task support sets may large (up to 500 images across all classes). We thus generate support set attack with approximately 20% poisoned shots per class, which is generally considered the upper limit of manipulated patterns conventional poisoning approaches (Jagielski et al. 2018), which use significantly larger datasets than few-shot learning. In Fig. 6 we show the relative decrease in accuracy of CNAPs when under attack (unnor- malized results are in A.4). Support Specific reduces the model's relative accuracy to almost for all datasets except MNIST. MNIST is the casiest classification problem in the large-seale suite because there are only ten classes and the input images are simplistic. The Support General attack
8

---

Under review as a conference paper at ICLR 2021
Noise
Support Specific
Support General
Swap
Relatwve Decnease
90.2
99-3
Accuracy (8)
100
00.
76.3
66
63.5
50
12.3
13.3
ILSVRC
Aircraft
Birds
Quick Draw
VGG Flower
MSCOCO
MNIST
OFAR10
CIFAR100
Figure 6: The relative drop in model accuracy for variety of attacks using the CNAPs algorithm on META-DATASET with = 0.05, = 0.0015, 100, averaged over 500 tasks, with all classes,
but only 20% of the shots poisoned.
Adversarial Classes
Adversarial Classes
Adversarial Classes
Adversarial Classes
5
1
5
3
3
5.2
1-
1.5
4.1
1-
0.6
1.0
Adversorial Shats
3.1
8.2
13.6
2.5
9.0
15.5
1.3
1.3
1.1
2.1
2.3
5- 4.5
11.8
20.3
5
4.9
14.7
23.8
5-
1.5
2.0
2.3
2.3
4.4
3.8
10-
11.0
29.3
49.1
10
13.8
40.8
67.3
10
3.7
7.3
3.3
10-
10.5
24.7
10.4
ResNet11, epsilon 0.05
ResNet18, epsilon 0.1
MNASNet, epsilon 0.05
MNASNet, epsilon 0.1
Figure 7: Relative drop in S-way classification accuracy when transferring adversarial support at- tacks as function of poisoned shots and classes using CNAPs, which uses ResNet18 network, to fine-tuners that use ResNet18 and MNASNet networks on ILSVRC 2012 from The attacks were generated with 7 = 0.0015, 100, M 6N, and averaged over 100 tasks.
significantly impacts classification aceuracy, easily out-performing all the baselines, in spite of the fact that only 20% of the support set shots are poisoned. Our results demonstrate that an attacker could cripple few-shot learning system, even in this more realistic scenario. The large impact of the attack may be partially due to the larger context sets, which allows for more poisoned patterns in total than the small-scale experiments; and partially because these classification benchmarks are much harder problems, with more classes, resulting in decision boundaries that are more fragile.
Attack Transfer Fig. 7 demonstrates that poisoning attacks developed for meta-learners can also impede the learning of fine-tuning based few-shot classifiers. Details of the fine-tuning protocol which involves learning the weights of the final layer classifier and FiLM layers (Perez et al. 2018) that modulate pretrained feature extractor weights of transferring the adversarial support sets from the meta-learning based CNAPS algorithm (with a
in Appendix A..15 Fig. 7 shows the effect
ResNet18 backbone) fine-tuning based few-shot classifiers (with Resnet18 MNASNET back- bones), as the number of poisoned classes and shots are varied. The MNASNet fine-tuner is largely robust the adversarial support set, potentially because the feature extractor is very different to that in CNAPs, though attacks with large perturbations and a large percentage of poisoned patterns do have some impact. Transferring the attack from CNAPs the ResNet18 fine-tuner more successful, with effects increasing significantly as the percentage of poisoned patterns are increased.
5 CONCLUSIONS
This paper introduced the Adversarial Support Set Attack, that attacks trained few-shot classifiers meta-lest time. We show that this attack is more effective than number baselines using variety of few-shot learning approaches, causing predictions on test inputs to be worse than chance. Future work may consider how to defend few-shot classifiers against adversarial support sets, for example by developing analogues of adversarial training algorithms, and investigating the effects of transferring adversarial support sets to conventional deep classifiers at training time.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Battista Biggio and Fabio Roli. Wild patterns: Ten years after the rise of adversarial machine learning. CoRR, abs/1712.03144 2017. Utt.tt/::/aaxxttt wrw. ab////b//112 03141
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Sındic, Pavel Laskov, Gior- gio Giacinto, and Fabio Roli. Evasion Attacks against Machine Learning at Test Time. arXiv e-prints, art, Xrxi: 1708 1778000001111 August 2017.
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop Artificial Intelligence and Secu- rity, AlSec 17, pp. 3-14, New York, NY, USA, 2017. ACM ISBN 1777---4550--52000 doi: 10. 1145/3128572.31444 140444. URL h acm. org 10. 31211443322722.2333
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017
IEEE Symposium Security and Privacy (SP), pp. 39-57, May 2017. doi: 10.1109/SP.2017.49.
Pin-Yu Yu Chen, Yash Sharma, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsieh. EAD: Elastic-Net Attacks to Deep Neural Networks via Adversarial Examples. arXiv e-prints, art. 17Xiv:1709.041110 Sep 2017a.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning. arXiv e-prints, art. urXiv:1712.05522 December 2017b.
Mauro Conti, Nicola Dragoni, and Viktor Lesyk. A survey of man in the middle attacks. 1EEE Communications Surveys Tutorials, 18(3):2027-2511, 2016.
Riley Edmunds, Noah Golmant, Vinay Ramasesh, Phillip Kuznetsov, Piyush Patil, and Raul Puri. Transferability of adversarial attacks in model-agnostic meta-learning. In Deep Learning and Security Workshop (DLSW) in Singapore, 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-lcarning for fast adaptation of deep networks. In Doina Precup and Yee Whye Tch (eds.). Proceedings of the 34th International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pp. 1126-1135, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL http::/prroceedingg mww. wwwps/wwssssepe Finn17a. html
Micah Goldblum, Liam Fowl, and Tom Goldstein. Adversarially robust few-shot learning: A meta- learning approach. arXiv, pp. arXiv-1910, 2019.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6577. 2014.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner. Meta- learning probabilistic inference for prediction. In International Conference on Learning Repre- sentations, 2019. URL ott.a//eee...oeeeeeeeeee////eees
/ Low.:e/.eeee.....S.CCC
David Ha, Andrew Dai, and Quoc V Le. Hypernetorrss. In International Conference on Learning Representations, 2016. http:pppee//peeeeeeeeeeeeeeeeeee
Junfeng He, Khoi Pham, Nachiappan Valliappan, Pingmei Xu, Chase Roberts, Dmitry Lagun, and Vidhya Navalpakkam. On-device few-shot personalization for real-time gaze estimation. Pro- ceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, Oct 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-leariing in neural networks: survey, arXiv preprint NXXv:2004.004399 2020.
10

---

Under review as a conference paper at ICLR 2021
Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru and Bo Li. Ma- nipulating machine learning: Poisoning attacks and countermeasures for regression learning. In 2018 IEEE Symposium Security and Privacy (SP), pp. 19-35, 2018.
Mijung Kim, Jasper Zuallaert, and Wesley De Neve. Few-shot learning using small-sized dataset of high-resolution fundus images for glaucoma diagnosis. In Proceedings of the 2nd International Workshop on Multimedia for Personal Health and Health Care, MMHealth 17. pp. 89-92, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 978 450355049, doi: 10. 1145/3132635.3 132650. URL htttps://d/t http://ii.rrg////////// 1141/ 32635 31 3132650
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 2009.
Alex Krizhevsky, lla Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Amal Lahiani, Jacob Gildenblat, Irina Klaman, Nassir Navab, and Eldad Klaiman. Generalizing multistain immunohistochemistyy tissue segmentation using one-shot color deconvolution deep neural networks. arXiv e-prints, art. arXiv: 1eci::180..00 $85.06658, May 2018.
Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. &T&T Labs [Online]. Available: http://yann. lecun. com/exdb/mnistt, 2:18, 2010.
Hoyeop Lee, Jinbae Im, Scongwon Jang, Hyunsouk Cho. and Schee Chung. McLU: Meta- Learned User Preference Estimator for Cold-Start Recommendation. arXiv e-prints, art. 1eXiv:1908000411 July 2019.
Ruirui Li, Xian Wu, Xian Wu, and Wei Wang. Few-shot learning for new user recommenda- tion in rrcationbbased social networks. In Proceedings of The Web Conference 2020, wwW '20, pp. 2472-2478, New York, NY. USA, 2020. Association for Computing Machinery. ISBN 9781450370233. doi: 10.1445/3364223379999 URL http:////oo....1.1111111.
3366423.33798
Alcksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. arXiv e-prints, art. arXiv:1706.06083, June 2017.
Kushagra Mahajan, Monika Sharma, and Lovekesh Vig. Meta-dermdiagnosis: Few-shot skin dis- ease identification using meta-learning In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR) Workshops, June 2020.
James Newsome, Brad Karp, and Dawn Song. Paragraph: Thwarting signature learning by training maliciously. In Diego Zamboni and Christopher Kruegel (eds.), Recent Advances Intrusion Detection, pp. 81-105, Berlin, Heidelberg. 2006. Springer Berlin Heidelberg. ISBN 978-3-5400 39725-0.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images. arXiv e-prints, art. 18X9 Dec 2014.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual reasoning with a general conditioning layer. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. In Pro- ceedings of the International Conference on Learning Representations, 2017. URL https:
//openzeview. pepperreview..eee/ddfiiiiee
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dÁlché-Buu Buc, E. Fox, and R. Gamett (eds.), Advances Neural Information Processing Systems 32, pp. 7957-7968. Curran Associates, Inc., 2019a.
11

---

Under review as a conference paper at ICLR 2021
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Code for "Fast and flexible multi-task classification using conditional neural adaptive processes". 2019b.
Benjamin I.P. Rubinstein, Blaine Nelson, Ling Huang, Anthony D. Joseph, Shing-hon Lau, Satish Rao, Nina Taft, and J. D. Tygar. Antidote: Understanding and defending against poisoning of anomaly detectors. In Proceedings of the 9th ACM SIGCOMM Conference on Internet Mea- surement, IMC '09. pp. 1-14, New York, NY, USA, 2009. Association for Computing Machin- ery. ISBN 9781 7755877457714 doi: 10.1145/1644893. 1644895. httthtttts////oii //o:: htg////ioorg.//..o///o T145/ 6448644433333 644895
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1153))211-255, 2015.
Xi Sheryl Zhang, Fengyi Tang, Hiroko Dodge, Jiayu Zhou, and Fei Wang. MetaPred: Meta-Learning for Clinical Risk Prediction with Limited Patient Electronic Health Records. arXiv e-prints, art. arXivv1905..0 90.99001.,,,... May 2019.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 4077-4087. Curran Asso- ciates, Inc., 2017.
Christian Szegedy, Google Inc, Wojeiech Zaremba, Ilya Sutskever, Google Inc, Joan Bruna, Dumitru Erhan, Google Inc, lan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In In ICLR, 2014.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-awwae neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 2820-2828, 2019.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B. Tenenbaum, and Phillip Isola. Rethinking few- shot image classification: good embedding is all you need? arXiv preprint arXiv:2003. 11539, 2020.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta- dataset: A dataset of datasets for learning to learn from few examples. In International Confer- ence on Learning Representations, 2020. URL httpp:://peeneeveer.. net www.eettfffomd---
TkgAGAVKPT
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Gamett (eds.), Advances in Neural Information Processing Systems 29, pp. 3630-3638. Curran Associates, Inc., 2016.
Chengxiang Yin, Jian Tang, Zhiyuan Xu, and Yanzhi Wang. Adversarial meta-learning. arXiv preprint (IXXv:88060033/66 2018
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in neural information processing systems, pp. 3320-3322 2014.
Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. Few-Shot Adversarial Learning of Realistic Neural Talking Head Models. arXiv e-prints, art. arXiv: 1905.08333, May 2019.
12


Reviews:
[{"review_id": "uihFTDo9-N6", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes adversarial attacks on few-shot learning systems. Evasion attacks are developed which can be viewed a simple applications of PGD/FGSM. The authors then apply evasion attacks to a support dataset, such that the n-shot classifier's loss is maximised on a query set. Additionally, poisoning attacks that cause the classifier to learn a shifted label set are introduced. Experiments on miniImageNet are performed with two meta-learning algorithms (MAML and ProtoNets) showing that these attacks lead to a large decrease in accuracy in comparison to baselines such as random noise attacks or universal adversarial perturbations (UAP) [1]. The authors then perform additional experiments such as measuring drops in accuracy for different numbers of n in n-shot learning and different query set sizes. \n\n\nI am not meta-learning expert, so I have assumed the authors choice of meta-learning algorithms and datasets is fair. Overall, I thought the paper was well-written, and the experiments were sound and relatively convincing. The downside is that this work introduces attacks, which as far as I can tell, are quite simple extensions or applications of gradient based evasion attacks such as PGD. I'm not sure if this represents a large enough contribution to warrant acceptance. The main contribution seems to lie in experiments showing how little data is required to make these kinds of attacks work, but what should we infer from e.g. figure 4a beyond \"if an adversary controls more data the attack is stronger\"? Is this a fair characterisation of the paper's contributions? I also have a few areas of confusion I hope the authors can clarify:\n\n\n1.  I found it curious that the UAP performed so poorly in comparison to the support attack, since the formulations are quite similar. I didn't understand how the UAP was constructed, it seems from Section 4.1 the perturbation was constructed from a separate classification task on 712 training labels from the full ImageNet dataset. Why was the UAP not constructed on the miniImageNet dataset? Or at least on a smaller task on the full ImageNet dataset? It is hard to reason about the veracity of comparison between the two attacks currently.\n\n2. Why was there no comparison with related work? For example, to (Goldblum et al., 2019; Yin et al., 2018) on the evasion attack side?\n\n3. Why is ProtoNets more robust than MAML?\n\n\n[1] Moosavi-Dezfooli, Seyed-Mohsen, et al. \"Universal adversarial perturbations.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_yPXh5HmnUV", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper introduces Adversarial Support Set Attack, an attack that perturbs the support set samples under the few-shot paradigm. It makes use of a seed query set to find adversarial samples that maximize the loss on this set, with the hope that it will generalize to any query sample. Empirical results show that this attack is effective against a variety of few-shot algorithms and datasets. Additionally, adversarial samples constructed on traditional few-shot algorithms are empirically shown to transfer to the fine-tuning few-shot algorithm (when using similar feature extractors).\n\nPros:\n1. This is the first paper that looks at support set poisoning as an adversarial attack in few-shot learning.\n\nCons:\n1. Swap attack is not explained comprehensively. Is the support set used as the seed query set in this case? Since it is one of the baselines, a better explanation for the method should be included.\n2. Figure 3 suggests that the seed query set is 20 times larger than the support set. This is unrealistic. Few-shot learning deals with scarce amounts of data. Under this paradigm, there are only a handful of samples per class. Assuming that an attacker has an order of magnitude more data is unrealistic.\n3. The paper mentions that the UAP perturbations were calculated on a 712-way classification problem, where smaller perturbations would be more effective than on a 5-way classification problem. If UAP is a baseline, the comparison should be made fair.\n4. As pointed out by the author's the transfer of adversarial samples in supervised learning is not new. It is not surprising that it happens in the case of few-shot learning as well, so that should not be counted as a major contribution.\n\nClarifications:\n1. Does the seed query set have an overlap with the query sets sampled for the Support General evaluation? If they are randomly sampled, there is a chance that they have some samples in common, especially when the size of the seed query set is large. If there is an overlap in samples, experiments should be run when the two sets are disjoint.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ScsPs453Q4L", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Pros:**\n+ The paper considers the construction of adversarial examples for a new learning paradigm which has practical relevance.\n+ A number of possible threat models under the few-shot learning paradigm are considered.\n+ The considered attack (a simple variation on PGD) is found to be effective against a variety of models and on a benchmark datasets.\n\n**Cons:**\n- The attack methodology is not particularly novel, as it is just a simple extension of standard PGD attacks. \n- The writing in the paper lacks clarity. The descriptions of meta-learning algorithms are not clear enough for a reader with knowledge of supervised learning but limited background on few-shot learning. In particular, there should be a dedicated section explaining the differences between attacks on traditional supervised learning and few-shot learning.\n- The related work refers to a couple of previous papers that have explored defenses against the query-based threat model for misclassifying particular examples. Similarly, this paper should have explored techniques for defending against the proposed support-set attacks. For example, the defender may add steps of training on the poisoned task to 'undo' the effects of support set attacks\n- The attack success metrics are not clearly defined. How are the 500 tasks mentioned in figure 3 chosen? Is the support set data at for the new tasks at test time modified at all? How does attack success vary with the similarity of tasks at training time vs those at test time?", "extra_scores": {}, "extra_text": {}}, {"review_id": "BtCt_BDg9vP", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a novel poisoning attack which is tailored for few shot learning classifiers. It performs an experimental evaluation of proposed attacks on various state of the art few shot learning / meta learning classifiers.\n\nOverall it’s a good paper which adapts poisoning attack on a few shot learning task. Thus I recommend to accept it.\n\nStrong points:\n* proposed poisoning attack on few shot learning classifiers\n* paper is well written and easy to understand\n* good experimental evaluation of the method\n\nWeak points:\n* security aspects of poisoning are not discussed in the paper. In particular paper does not clearly describe goals of adversary (it’s implied that adversary wants to make model always misclassify entire test set), does not discuss capabilities of adversary (white-box vs black box).\n* The main contribution of the paper is a poisoning attack, however authors talk quite a bit about adversarial examples (evasion attack) which could be distracting from the main point of the paper given that paper does not really add any new technique specifically related to evasion attack.\n* proposed taxonomy of few shot learning attacks (fig 1) is not comprehensive.\n* while the proposed attack is novel in a context of few shot learning, nevertheless it’s pretty straightforward generalization of poisoning attack for fully-supervised classifiers.\n\n\nRecommendations on how to improve the paper:\n* Add clear discussion of goals and capabilities of the adversary. If feasible, consider adaptation of the attack for different goals (i.e. change only a subset of predictions of the classifier on test set) and capabilities of the adversary (i.e. black box). Authors may refer to https://arxiv.org/pdf/1804.00308.pdf which discuss most of the necessary terminology and which is already cited by this paper.\n* Make paper more focused on poisoning attack, and remove references to adversarial examples, whenever they are not needed. \n* As mentioned above, proposed taxonomy of attacks is not comprehensive. It looks more like a list of attacks authors have tried, rather than comprehensive classification of all possible attacks. Thus I would recommend to change wording and call it “considered attacks” or something similar instead of “taxonomy”.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for attacking few-shot learners with poisoning a subset of support set. I believe this might be the first work to address adversarial examples for meta-learners (or few-shot learners), which is a timely issue. A common concern raised by most of reviewers is in the novelty of this work, in the sense that the method builds on a basic attack strategy (such as PGD) in the standard adversarial example setting. Authors responded to this, summarizing what's new in this paper. Episodic training for few-shot learners requires consuming support set (instead of single training data point). It is a nature of most meta-learning methods. Thus, it is easily expected that the adversarial attack for few-shot learners is naturally extended to poisoning a support set (or its subset) instead of a single data point. Certainly such extension may entail a new strategy. However, during the discussion period with reviewers, concerns on the novelty of such extension still remains. In particular, the few-shot learning algorithms do not allow big changes in the original model. The algorithms analyzed are prototypical networks that do not utilize fine-tuning, and MAML that fine-tunes for a small number of pre-fixed steps. So the transfer of adversarial samples may not be counted as a major contribution.


Decision: Reject