Poisoning Generative Models to Promote Catastrophic Forgetting
Authors: Siteng Kang
Zhan Shi
Xinhua Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: poisoning attacks
backdoor attacks

Abstract: Generative models have grown into the workhorse of many state-of-the-art machine learning methods. However, their vulnerability under poisoning attacks has been largely understudied. In this work, we investigate this issue in the context of continual learning, where generative replayers are utilized to tackle catastrophic forgetting. By developing a novel customization of dirty-label input-aware backdoor to the online setting, our attacker manages to stealthily promote forgetting while retaining high accuracy at the current task and sustaining strong defenders. Our approach taps into an intriguing property of generative models, namely that they cannot well capture input-dependent triggers. Experiments on four standard datasets corroborate the poisoner’s effectiveness.


Under review as a conference paper at ICLR 2023
POISONING GENERATIVE MODELS TO PROMOTE CATASTROPHIC FORGETTING
Anonymous authors
Paper under double-blind review
ABSTRACT
Generative models have grown into the workhorse of many state-of-the oof-the-an art ma- chine learning methods. However, their vulnerability under poisoning attacks has been largely understudied. In this work, we investigate this issue in the context of continual learning, where generative replayers are utilized to tackle catastrophic forgetting. By developing novel customization of dirty-label input-aware back- doors to the online setting, our attacker manages to stealthily promote forgetting while retaining high accuracy the current task and sustaining strong defenders. Our approach taps into an intriguing property of generative models, namely that they cannot well capture input-dependent triggers. Experiments on four standard datasets corroborate the poisoner's effectiveness.
INTRODUCTION
The vulnerability of machine learning systems must be scrutinized before they can be deployed to security-criie crittcal applications. The common evasion attack assumes that clean target instances can be manipulated at test time, which can be unrealistic in many scenarios. In contrast, poisoning attacks only make malicious and imperceptible modifications to the training set, so that the prediction on test examples can be mistaken. The threat models may insert poison examples (Chen et al., 2017), flip the training labels (Xiao et al., 2012; Levine & Feizi, 2021), or modify the training example inputs (Biggio et al., 2012; Shafahi et al., 2018).
Although poisoning attacks have been extensively studied under discriminative learning, their potential risk in generative learning has been largely understudied. Ding al. (2019) poisons the training examples so that the learned generator covertly changes some important part of the output image, e.g., turning a red light into green. Salem et al. (2020) enables the adversary to control the output image by planting a trigger in the input image or noise. Both are backdoor attacks requiring write access to test data, and work in batched learning scenarios.
The increasing penetration of generative models in machine learning urges the investigation of poisoning attacks in a broader range of learning paradigms In this work, we focus on continual learning, a prominent setting where tasks arrive in streams and each of them corresponds to diseriminative learning problem such as classification (Chen & Liu, 2018). Since the tasks are streamed and cannot be stored, the running classifier often suffers from catastrophic forgetting, where the performance on older tasks gradually deteriorates (McCloskey & Cohen, 1989).
Deep generative replay (DGR) is natural tool to bring back the memory of the previous tasks by learning generative model to fit the data of these tasks (Shin et al., 2017; Cong et al., 2020). Despite their effectiivnness, new vulnerabilities to the training data D, for the current task t, so that catastrophic forgetting can be promoted when such poisoned DD is used for training both the replayer G2 and the classifier. In this work, we seek practical and stealthy poisoning attacks on DGR that achieve three objectives:
also opened up where misleading examples can be injected
01 After moving past task I, the classifier will soon forget what was learned from (i.e., perform poorly on clean test examples drawn from it) despite using replayer for all the tasks seen far.
02 During task the classifier trained from the poisoned data does not suffer degradation of test accuracy on task itself. This is important because poor performance on the current task can raise significant and immediate suspicion. In contrast, by promoting forgetting, the harm will

---

Under review as a conference paper at ICLR 2023
manifest itself only after the victim has moved on to the next task, by which time it will have become too late because the access to the samples of task is already lost.
03 The poisoned data should be robust to solid defenses deployed by both the classifier and the replayer.
The main difficultyy lies in two folds. Firstly, although both O1 and 02 are straightforward to fulfill individually, they are at odds with each other and are hard to fulfill simultaneously. Secondly, the transiency of data streams compels the adversary to make irrevocable attacks task before future tasks arrive and before catastrophic forgetting can start to occur. Due to this difficulty, poisoning attacks have been much less studied in an online setting. Mladenovic al. (2022) addressed the online decision problem of selecting k examples for evasion attack. Zhang et al. (2020) assumed the instances are drawn i.i.d. from a time-invariant distribution, which not the case in continual learning because tasks may even have disjoint classes. Other works require multiple passes of the data stream (Gong et al., 2019; Lin et al., 2017; Sun et al., 2020), or clairvoyant knowledge of future data (Burkard & Lagesse, 2017; Wang & Chaudhuri, 2018).
Our contribution, therefore, is to overcome these challenges and to reveal the vulnerability of generative models in the sense that their training data can be poisoned stealthily such that task can be learned well at present but forgotten soon in the future. Noting that simple label-flpping poisoning can be easily detected, we resort to dirty-label backdoor/Trojan attack (Liu et al., 2018) to attain 02: the trained classifier performs correctly on clean examples, but errs if the example is planted with trigger. To further achieve 01 and 03, we capitalize on the input-aware backdoor (Nguyen & Tran, 2020), which allows the trigger to vary depending on the image. As a result, it can not only withstand stronger defense ($4), but also enjoys higher variation and stealthiness, hence much harder for generative model to capture. So the replayed images do not well preserve the trigger (we call it trigger-discarding property in $3.3) while retaining the incorrect label, leading naturally to forgetting ($3). The problem is set up in $2 and experiments are provided in $5 to show the effectiveness of the attack. Our innovations are summarized as follows:
Proposing the first poisoning attack that promotes catastrophic forgetting in continual learning. Achieving poisoning (no trigger is needed test time) through a novel way of leveraging backdoor attack that is particularly effective for exacerbating catastrophic forgetting.
Identifying a triggr-ddsscrddng property of generative models that is intriguing for backdoor attack.
Related work Generative models have been pervasive in machine learning (Murphy, 2023, Part IV), reaching far beyond the original role of density estimation and serving as a key infrastructure in supervised, unsupervised, and reinforcement learning. We contend that their vulnerability needs to be examined in the context of their use. In the vanilla density estimation, Condessa Kolter (2020) learned robust variational auto-encoder (VAEs) that retain high likelihood for the data points under adversarial perturbation. The underlying threat is evasion attack, and along similar lines, Tabacof et al. (2016) and Kos et al. (2018) studied attacks that promote reconstruction error of the decoder in a VAE, Some recent works address attacks on membership inference (Hayes et al., 2019; Chen et al., 2020; Hilprecht et al., 2019), model extraction (Hu & Pang, 2021), and attribute inference (Stadler et al., 2022). However, poisoning attacks on generative models are still understudied.
Our aim is to poison generative model instead of learning a generative model to produce poisons for another (discriminative) model (Yang et al., 2017; 10 Muñoo-GGnnáleez et al., 2019). We also leave it future work to defend the proposed attack, noting that (certifiable) defense and detection have been well studied for poisoning attack on batch discriminative models (Peri et al., 2019; Steinhardt et al., 2017; Levine& & Feizi, 2021; Jagielski et al., 2018).
2 ATTACKING GENERATIVE MODELS IN CONTINUAL LEARNING
We consider the continual learning setting, where tasks arrive sequentially. The goal is to keep updating a classifier that predicts accurately not only on the current task, but also on the previous tasks. Each task t is indeed joint distribution P,(X, Y). where X E X' is the input from feature space X. and Y E Ve is the label whose domain Vi may change with the task. For example, Vi consists of digits 0 and 1, and )22 encompasses 2 and 3. Even in the case where the domains remain constant, the distribution P. can shift. The goal of continual learning is to find a classifier C,, such that the overall risk across all tasks seen so far is minimized:
2

---

Under review as a conference paper at ICLR 2023
C R arg min i=1 Ex [()
Σ
(I)
C
3

---

Under review as a conference paper at ICLR 2023
Here E is a loss function, and will focus on multi-class classification with cross- loss. A major challenge in continual learning is that at task t, only the samples from the current P; are available, while those from the previous tasks are longer accessible. Although most algorithms would only adapt to the new task instead of completely retraining from scratch, the performance on previous tasks may still deteriorate significantly, a phenomenon known as catastrophic forgetting.
To alleviate this issue, DGR-based approaches resort to learning a DGR model G, that approximately replicates Pt. Due to the constraints in computation and storage, it has to be lossy approximation because otherwise one might as well store all the past examples. Then at each task i, samples of (X, pairs are drawn from not only the current Pt, but also from the replayers for the previous tasks G == K= Their union subsequently used to update the classifier into C.. The whole process of vanilla DGR bbsed continual learning is illustrated in Algorithm 1.
DRG models can be simplified into single replayer G that is updated over time, as opposed one replayer per task. However, our contribution is the poisoner, not the replayer. Employing multiple replayers only makes attacks even more challenging, because instead of just poisoning one running replayer, We are now tasked to poison many of them, each of which can only be poisoned once at its current task. If we simply keep a single running replayer, then we enjoy many opportunities of poisoning it at any time and our objectives will become much easier to achieve. To conclude, multiple replayers set up a more stringent benchmark for testing our poisoner. A natural choice of the replayer is a conditional generative model such conditional GAN (cGAN), which first samples the label Y from a discrete distribution, and then generates the feature X via the cGAN.
2.1 ATTACKERS AND LEARNERS
Deep generative replay (DGR) used by continual learning to combat catastrophic forgetting Input: Tasks 1, 1,2,... represented PI, Pz....
Algorithm 1:
Two parties participate in the process, and we first set forth our assumptions on them.
1 Initialize classifier Co
The victim learner/user consists of clas-
2 for 1, 1... do
sifier, replayer, and a defender. We as- sume none of them has access to the orig-
3 for E ft - 1] := 21. 13 do S. +- SampleFrommi romDGR (X, Y pairs)
inal clean data, and can only access the poisoned data. Further, the replayer must perform well, i.e., the generated samples 7 match the distribution of data presented to it for training. Otherwise, the replayer would not be adopted by the user in the first place, and can spare any need of attack by, e.g., generating random images with random labels. We also assume that the learner cannot store any data beyond its current task, which is standard in continual learning.
Sample DD from task P.
to be poisoned C2 + "TriinClassiferr(C... Dr D
G. (raaTrneeellayyr()))). e.g. conditional GAN
A defender is an algorithm that the learner employs to scrutinize and prune the possible poisons in the training data. In DGR, it means examining the data collected from the current task t, well as the replayed samples from previous G; in step 4. This is known as pre-training defense, whose defense-patches up the learned model ( (Wang et al., 2019).
The attacker (our threat model) is only allowed to poison (modify) the samples Dr in step 5 of Algorithm 1. The attacker has access to the internal mechanism of classifier, replayer, or defender. It can read the gradient of the classificr's 's training objective with respect to the input, but not its structure or weights. This is moderate mid- midd-grunnd between full access (c.g., training by the attacker itself on the cloud) and no access (independently manipulating the training examples). Such an assumption has been commonly adopted, e.g., by dynamic backdoors (Nguyen & Tran, 2020), Witches' brew (Geiping et al., 2021), and by implicit differentiation based poisoning (Muñoz- González et al., 2017). We will pursue a dirty-label backdoor attack, i.e., a small portion ps of D. will flip their label, along with trigger of size Pa planted to their input image (more details in $3.2).
Achieving poisoning through backdoor. Although our method will leverage backdoor attacks, our overall goal is poisoning attack, not backdoor. In backdoor attacks, classifier predicts poorly only on backdoored examples with triggers, while remaining well on clean test examples. In contrast, poisoning attacks (excluding backdoor) aim to predict poorly all examples, irrelevant of "trigger".
In continual learning, an attacker generally does not have the liberty of planting a trigger on test examples. So we will address in 01 and 02 much more challenging setting (from attacker's s perspective) where such an access is not available. It is important to note that our approach only utilizes backdoor attacks means of achieving the goal of poisoning the generator/repaayer.
4

---

Under review as a conference paper at ICLR 2023
2.2 DIFFICULTIES IN THE ATTACK
Since most of the existing poisoning algorithms are in the batch setting, the extension to continual learning brings about new and significant challenges.
Firstly, the poison cannot compromise the current classifier, but should sufficiently poison the DGR so that the samples drawn from it during the later tasks will be detrimental enough to forget the previously learned tasks. This rules out simply flipping the label of some examples in D,, because it is easy to detect (Levine & Feizi, 2021) and the resulting classifier C, will perform poorly on task t.
Secondly, since the future tasks have not been witnessed yet at task and the training of C++1 has not started, it is infeasible to optimize the forget-inducing distortion on Dr via back-pree soock--paagationn based optimization the context and objective are not yet available for future forgetting.
THE ATTACK ALGORITHM
Our poisoning attack proposes evading the defense by leveraging the input-aware backdoor attack (Nguyen & Tran, 2020), so that mislabeled data points carrying a trigger can be injected to 'D in a small amount. In particular, our approach achieves 01 O3 through the following effects:
1. Since the triggers depend on (hence vary across) the input data, the defender can hardly detect it.
Since the mislabeled examples for the current task all carry an input-aware trigger, the learned backdoored classifier for task f makes mistakes only for backdoored examples. As such, it predicts accurately on pristine test examples for task which carry no trigger.
As generative models essentially represent lossy compression, it is generally unable capture the triggers that change with the input. When replayed later for a future classifier at task 1+1,,4 1,.,,,......... 2.... the triggers go absent while the incorrect label is retained. So the classifier will be trained on mislabeled examples of task f with no backdoor, hence misclassifying clean test examples.
3.1 BACKDOOR ATTACKS
Our solution is based backdoor attacks, which despite the marked resemblance to poisoning attacks, do not misclassify test example unless a pre-designed trigger is inserted to it. Such a flexibility of modulation proves essential. Backdoor attacks such as BadNets (Gu et al., 2019) plant pre-selected or learned trigger into some training images at pre-selected or varying location. A number of variations are available such as soft blending and multi-channel. Adding the resulting image to the training data along with a flipped label (randomly selected for an untargeted attack, and pre-specified for a targeted attack), a classifier can trained that enjoys two important properties: io
P1 Once test image is also backdoored with trigger, the predicted label will change to the pre-specified (or random) one in a targeted (or untargetted) attack.
P2 However, the test accuracy pristine images (without trigger) can remain very high.
Our inspiration originates from this trigger-bases modulation. Suppose we plant the trigger on D; in step 5 of Algorithm 1, and denote the resulting training set as D. Then the resulting C, will perform well pristine test examples of task t, because they do not carry the trigger. After moving to task t 1, the replayer will (approximately) reproduce D,, which point two cases can be considered: the replayer works purely by rote, then D. will exactly replayed and the resulting classifier C++1 will be backdoored in the same way C. As result, it will still predict accurately clean samples from task t, i.c., the attack fails in promoting forgetting.
If the replayer is lossy and is unable capture or reproduce the trigger, then the replayed examples might no longer carry the trigger. However, they still carry the flipped label. As a result, C++11 will now be trained on mislabeled examples without a trigger, and will therefore perform poorly on task t. In this case, the attacker successfully promoted forgetting.
The requirement on the replayer the second case may appear unrealistic, because firstly the replayer is supposed to faithfully preserve the salient information in the inputs to address catastrophic forgetting. Secondly, a user (who constructs and trains the replayer) obviously has no motivation to collaborate with attacker. Therefore, the key challenge for the attacker is design delicate triggers that are as likely to be overlooked and disregarded as possible under generative modeling, while retaining the good performance on clean test data during task (property P2).
5

---

Under review as a conference paper at ICLR 2023
Algorithm 2: InputAwareBBcckoorr
Input: Data generation distribution P(X, Y), which will the invoked with P = Pe at task t
(D, Laiv) + -nputttwaarBBaccoor---fffee B(y))
return arg mint [Cat + Advo J. where La (<ew> ((C()).)) is the classif. risk
Algorithm 3: InputAvaareBBacktoor-Oi
Input: P as in InputAwareBackkoor, and backdoor generators Bry] == [By : E
Initialize D = which will contain clean and poisoned examples. Set Laio (diversity loss).
for (x, 3) sampled from P for task do
Sample dl (0, 1), sample (i, 1) from P excluding (z, v). sample v/ from VMrh
Laie += + =|| By (x) By (+) // Accrue the diversity loss
5
if d < Pb then x' + x 3 (z),, D+= (x',3)) // make a backdoor example else if d < Ps + Pe then x' + x By (.). D+= (x', // make cross example
else += (x,y) // clean example
return D and Lase
This is indeed challenging as we experimented. Static backdoor (BadNet) can be easily replayed by a generative model. Trojan attack requires access to the victim model's structure and weights. Neither can survive the defense of neural cleansing. We also tested static backdoor with changing location, which again, turned out easily detected and fixed by neural cleansing. Witches' Brew (Geiping et al., 2021) and other gradient matching based methods need to know the target before deploying the attack, while future tasks are unknown in continual learning. Eventually, it turns out the inpul-aware backdoor satisfies our need, where a trigger is customized for each example through a learnable generative model, hence exhibiting much less regularity for the replayer to capture.
3.2 INPUT-AWARE BACKDOOR
We first recap the input-aware backdoor (IAB, Nguyen & Tran, 2020) as shown in Algorithm 2 under given data distribution P, and then detail how utilize it for our purpose. In line 2, the classifier C and class-wise backdoor generating networks By are jointly optimized over an objective constructed Algorithm 3, where each (x, y) sampled from P contributes in one of the following modes:
As a backdoor example with probability Pa: a wrong label y' is randomly picked, and then trigger that depends on x is generated by By (I) and injected to x in line via elementwise product O.
As a cross- cross-trigger example with probability Pe. To ensure that trigger synthesized for one example is not effective for another, another à is sampled from P with label ÿ. Then x is injected with the trigger generated from è for wrong label y', and the result is paired with the clean label y (line 6).
As clean example otherwise (line 7).
In Algorithm 2, Bly is explicitized line to stress that both the diversity loss Lavo and classification risk Cet (through D) are functions of Bpy, which is then optimized in line along with the classifier C. To see that the attacker fits in our threat model, note C' and B are jointly optimized in line 2 of Algorithm 2, and the attacker only requires reading the gradient with respect to the input of C. As observed in our experiment, IAB proffers the following property (Nguyen Tran, 2020):
P3 The backdoor in IAB can be hardly detected by state-of the-the-art art methods, e.g., neural cleansing.
To summarize, we fulfilleed 02 by P2, and 03 by P3. To meet 01, we require a trigger-discarding property as follows, which plays key role in our method and will be discussed in the next subsection:
P4 The replayer cannot well capture the trigger generation network of IAB, in the sense that the replayed examples do not well preserve the triggers.
3.3 TRIGGER-DISCARDINE GENERATIVE MODELS
Property P4 depends on both the replayer and the trigger. If the trigger is a constant small white square at the image center, most generative models will preserve it. Same is true if the replayer only replicates the training set. In general, is supposed to capture the salient features of the input, and one might presume that triggers are likely to be discarded if they vary a lot across examples. It turns

---

Under review as a conference paper at ICLR 2023
out not true. For example, we placed square/trianglc/ronnd in random colors at random positions of the images, and a WGAN easily reproduced them with these (interpolated) color, shape, and position.
Formally, let Ps be the data distribution, and suppose given an input I, the trigger is generated by a Iearnable network fo(æ) and is added to x by a pre-specified operation g(z, fo fo(z)). It induces a push-forward of Pz: O == (xr + g(z, fo)))) # P.. Let
distribution of backdoored examples a generative learning algorithm A map a set of examples {x; ); to distribution. Then the trigger is intended demote some divergence (e.g., KL and Wasserstein) between
Pz and E Bei A(2x
(2)
Directly computing the gradient in 0 is both expensive and infeasible, because A is assumed inacces- sible in $2.1. Fortunately, our experiments show that P4 is well (but not perfectly) achieved when IAB is applied in conjunction with several SOTA generative models such as conditional Wasserstein GAN (cWGAN, Engelmann & Lessmann, 2021) and conditional VAE (Sohn et al., 2015). This is no surprise because these models have limited capacity, and the trigger's 's dependency on the input, which is more involved than just random, significantly raises the sample complexity for generative learning. A theoretical analysis is left for future work, and $5.2 empirically illustrates this intriguing property.
3.4 POISONING THE REPLAYER VIA IAB
We are now ready to apply IAB to poison the DGR used by continual learning against forgetting. We will call our method Continual Input-Aware Poisoning (CIAP). It does not backdoor test images.
Algorithm 4 demonstrates the opera- tion of all participants (user, attacker, and defender) during task t, which cor- responds to the loop under a given t in Algorithm 1 (line 3 to 7 therein). The red colored steps are reserved for de- fense, which will be detailed in Sec- tion 4. In line 7, the poisoned data D. for task is joined with the replayed data S to construct the classifica- tion risk Lat. Both the attacker and the user are trained in the same way as in Algorithm 2, while the only difference is that our optimization here is based on mini-batches, and the replayer G, is additionally learned in line 10.
Algorithm 4: Operation of the user, attacker, and de- fender during task t (in place of line 3 to 7 of Algorithm 1)
Input: P. for task /. Input: Classifier C₂-1, and backdoor generators B(y} Initialize C with C.-1.
2 for E [t 1] do
S, t- SumpleerooRRelaver((C, (X, Y pairs)
4 Defender: Apply v-SVM on the replayed data S
for number of iteration (run in mini-batches/ do
De, Ldiv InputtwareBBckooooOObbii
B(y))
// Both Dj and Like are functions of B|y
Les + sum of E(C(x), y) over (a,, v) D, U U // La is a function of C' and Bp1
User: update C to reduce Cet
Attacker: update B to reduce La + Adiv
Update the replayer G, using D, and the latest B 10
4 THE DEFENDER
Defender: Neural Cleansing on C2 +- C
We consider two defenses against the CIAP attack. The first is neural cleans-
return Ge, Bpy) and C.
ing (Wang et al., 2019), which has been inserted in line 11 of Algorithm 4. If it were successful, then the backdoor planted in C2 would be detected and removed, thereby defeating objective 02 with immediate poor accuracy on clean test examples at task t.
Our second defense is aimed at objective 01. To this end, we apply an outlier detector =-SVM to S1a-1, which is in line 4 of Algorithm 4. If it managed to filter out mislabeled replayed samples, then the attacker would fail to bolster catastrophic forgetting. Here v a hyperparameter controlling the fraction of outliers. Since its value is unknown in practice, our experiment will enumerate a range of v values, and demonstrate the extent to which the learner's performance can be saved respectively.
It is crucial to recognize that the replayed examples are not simply label-flipped poisons (i.e., clean images with wrong label), although the replayer is poisoned with label-fliped and backdoored examples. This is for two reasons. Firstly, since the examples of a class y' is fed to the replayer to train for the class y, the generation of the features/images for class y is contaminated. Secondly, the -eput-dependent triggers introduce additional complications to the generative model. Indeed, we tested by directly generating label-flipped examples based on clean images, and v-SV easily filtered them out. However, this is not the case when v-SVM is applied to our replayed images ($5.3).
7

---

Under review as a conference paper at ICLR 2023
5 EXPERIMENTAL RESULTS
We next experiment CIAP verify: it attains the two objectives 01 and O2; ii) the trigger- discarding property introduced in $3.3 holds true for commonly used generative models; iii) CTAP remains effective under strong defenders (03). The code is available at Online Supplementary.
5.1 EFFECTIVENESS OF THE ATTACK FOR OBJECTIVES 01 AND O2
We tested CIAP on five datasets: split-MNIST (Ciresan et al., 2011), split-CIFAR-10 (Krizhevsky Hinton, 2009), FashionMNNISTMMIISST (Xiao al., 2017), permuted-MNISTT (Goodfellow et al., 2014), and split-EMNISS (Cohen et al., 2017). We used Spinal VGG the victim classifier (Kabir et al., 2020) for the four MNIST datasets, and ResNet (He et al., 2016) for the split-CIFAR 10 dataset. The results shown here eWGAN with gradient penalty the replayer, and more results for cVAE are deferred to Appendix E. The poison ratio = 0.25, and the cross ratio Pe = 0.15. Each trigger allowed to change 10% of the pixels of selected image (mask density), and Figure will show that the triggers are quite inconspicuous.
split-MNIST We separated the entire dataset of MNIST into five tasks, each consisting of images from two disjoint classes in MNIST the first task includes classes 0 and 1; the second task includes 2 and 3; and so on. The victim model was trained for 100 epoch in each task.
1.0
1.0
Tookig Aoo
Testigg
Tosing
100
Nambar of Epooho
400
500
100
Namber Epochs
400
800
100
Natibor Esoche
400
500
(a) Bascline: no replayer
(b) Clean with replay
(c) Attacked with replay
Figure 1: Test accuracy on clean test images for split-MNIST
Figure la shows the baseline result without a replayer, where the blue line represents the test accuracy of the first task, orange line for the second, etc. As expected, the test accuracy for each task drops rapidly to 0% after the victim model proceeds to new task. It is 0% because the new task has no overlap with the previous ones in the label space. Figure 1b shows the result of faclliiil training, where the forgetting is significantly mitigated, and the test accuracy remains high on all trained tasks. This confirms the effectiveness of DGR and the sufficient capacity of the cWGAN.
Figure Ic shows the result after our attack CIAP is enacted. The test accuracy of each current task can still achieve nearly 100%, corroborating the achievement of objective 02. When the learner moves to the next task, the accuracy on the previous tasks falls sig- nificantly to around 20% despite some fluc- tuations. This confirms that the objective O1 (forgetting) has also been attained.
D
D
(a) Clean (h) Poisoned 0 (c) Clean
(d) Poisoned
Figure 2: The clean and poisoned images used train the replayer for split-MNIST.
Finally, we plot in Figure some example clean images of class 0 and from the first task of Mplit-MNIST, along with their corresponding poisoned images constructed by IAB (before label flipping). Clearly the poisons are quite inconspicuous.
split- CBiit-CIFRR-10 on CIFAR-10, with the same setup of five disjoint tasks. Here the victim model was trained for 50 epochs each task.
To illustrate the effectiveness of CIAP in colored space, we repeated the experiment
Similar to the MNISS-MNIST, the victim model completely forgets the earlier trained tasks after a new task starts, shown in Figure 3a. After DGR is introduced in Figure 3b, although forgetting is not as
8

---

Under review as a conference paper at ICLR 2023
Testing Accuracy
Accuracy
Texting Accurscy
60
60)
60-
40
40
80
Mor
20
8
20
50
100
150
200
250
0
50
100
150
200
250
0
50
100
155
200
256
Number Epochs
Number Epochs
Number of Fanchs
(a) Baseline: no replayer
(h) Clean with replay
(c) Attacked with replay
Figure 3: Test accuracy on clean test images for SpLit-CIFAR-10
well mitigated as in split-MNIST, solid improvement is still made on the test accuracy for all trained tasks. However, the improvements brought by DGR were completely eliminated by the CIAP attack. As Figure shows, the test aceuracy on past tasks drops down to 0% after being poisoned. During the current task, however, the accuracy can still achieve the same level as in Figure 3a.
We also tested on the split-EMNIST dataset with 10 tasks, and the results are similar; see Appendix C. The results of FashionMNIIT-MNIIST and permuted-MNIST are in Appendix A and B, respectively.
5.2 INVESTIGATION OF THE TRIGGER-DISCARDING PROPERTY
To hetter illustrate property P4, we set up two experiments on split-MNIST using eWGAN. The first one studies the percentage of backdoored images (images with a trigger) generated by the replayer, when a varying portion of the training images are backdoored. Our goal is to show that such a percentage is much lower for IAB than for a static backdoor, i.e., the triggers of IAB are much less likely to survive the generative learning. A static backdoor refers to a white square on the image's ' top left corner. To this end, we trained binary poison detector based on a training set that is backdoored with the IAB network learned from the first task. Another detector was trained analogously for static backdoor. This allows us to measure the percentage backdoored images from the replayer. The two detectors achieve 97% accuracy, and similar ideas have been used in evaluating generative models such as inception score (Salimans et al., 2016).
As shown in Figure 4, the static backdoor maintains almost the same percentage of backdoored images used for training, while that for IAB grows much more slowly, producing only 40% backdoored images when 90% of the training images are backdoored. This shows that IAB is far more likely to be discarded by the replayer.
100
IAB
1.00
80
static
RTB0K Tatt
ATREC Sapcses Batta
80:
60
Backdoor
40
40
20
10
60
10
20
30
40
50
60
Backdoor Injected
Figure 4: Poison rate of replayed images
(a) Static backdoor
(b) Input-aware backdoor (IAB)
Figure 5: ASR for static backdoor and IAB
Our second experiment examines the chance of replaying backdoored image by using the attack success rate (ASR). In the same setting as the above experiment, we backdoored 25% images in the first 30 epochs with flipped labels, and used them to train a replayer. Then we generated examples from it in the second 30 epochs, and used them incrementally train new classifier. This classifier is tested on backdoored images, and the proportion of misclassified images is calculated as the ASR.
If the triggers are preserved by the replayer, then classifier learned from the replayed images should permit a high ASR. This is confirmed in Figure Sa where the ASR remains at 100% for static backdoor. The drop in the middle is because the classifier trained from scratch. In contrast. the ASR drops to zero for IAB in Figure 5b, confirming that the newly trained classifier is not backdoored, i.c., the replayed images do not preserve triggers sufficiently well for training backdoored classifier.

---

Under review as a conference paper at ICLR 2023
5.3 ASSESSING CIAP ATTACK UNDER DEFENSE (OBJECTIVE O3)
We next study how well our attack withstands defenses. To this end, a v-SVM with a radial basis kernel was applied to the output of the convolutional layer of SpinalVGG. This allows a portion of replayed samples to be filtered out, and the proportion is controlled by E (),
1.0
1.0
1.0
Auni
Myra
Tooiing Assurayy
Teelng Ausagayy
Teeing hocugscy
Moa
3.4
92
0
100
2000
200
400
500
1.00
2000
300
400
500
100
200
300
400
500
(a) /
0.15
(b) //
0.25
(c)
0.35
Figure 6: Test accuracy after defense with different & values for split-MNIST
As shown in Figure 6 where is varied in {0.15, 0.25, 0.35) the split-MNIST dataset, the filtering by v-SVM does help a little, especially when v is set around the poison ratio (pp = 0.25). However, it remains unable well remove the impact of the attack, and the test accuracy on past tasks still falls below 50%. Similar results on the other datasets are relegated to Appendix D.
0 O
3
S
5
8
1
6
9
5
q
7
3
U
3
O 3
3
0
0 S /
A
3
2
v i 3
1
1
l
/ 3
4
7
1
3
1
7
(a) Clean replay
(b) Poisoned replay
(c) Poisoned replay after SVM
Figure 7: Replayed images on split-MNIST with label "1" from (a) clean replayer, (b) poisoned replayer, and (c) poisoned replayer after filtered with =-SVM (v = 0.25).
The limited improvement could be partially ascribed to the compromised image quality due to the backdoors. To better visualize the consequence poisoning on the replayer, we compare in Figure the replayed images before and after the attack. Figure 7a presents example images generated by replayer that is trained on clean images only. In contrast, Figure 7b shows that the poisoned replayer often generate images from incorrect class, i.e., another class that is incorrectly labeled as "1". Although the replayer cannot reproduce the input-aware backdoor, it tends to turn the backdoors into some random noise, making it harder for the filter to identify those poisons. As result, the remaining replayed images are only slightly improved by the D-SVM filtering as shown in Figure 7c.
We finally investigated the proportion of mislabeled example pairs generated by the replayer. The resulting confusion matrix is shown in Table in Appendix F, using the replayers after completing task 5 split-MNIST. The total "wrong pair ratio" turns out not high.
CONCLUSION AND FUTURE WORK
We proposed a novel poisoning attack on the generative replayer in continual learning, so that catastrophic forgetting can be promoted while the accuracy the current task is not hurt. Our approach takes advantage of input-aware aware backdoor attacks, whose triggers cannot be well captured by normal generative models thanks their input dependency. In future work, will delve more into the theoretical analysis of the -ieger-discadding property. We will also extend the approach to continual learning without known task boundaries.
10

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
Anonymous downloadable source code for reproducing experiment results is available at Online Supplementary. Please refer to the included rea file for details.
REFERENCES
B. Biggio, B. Nelson, and P. Laskov. Poisoning attacks against support vector machines. In International Conference Machine Learning (ICML), 2012.
Cody Burkard and Brent Lagesse. Analysis of causative attacks against svms learning from data streams, In Proceedings of the 3rd ACM on International Workshop on Security And Privacy Analytics, pp. 31-36, 2017.
Dingfan Chen, Ning Yu, Yang Zhang, and Mario Fritz. GAN-leaks: taxonomy of membership inference attacks against generative models. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security (CCS), 2020.
X. Chen, Chang Liu, Bo Li, Kimberly Lu, and D. Song. Targeted backdoor attacks on deep learning systems using data poisoning. ArXiv, 1b1.0022., 2017.
Zhiyuan Chen and Bing Liu. Lifelong Machine Learning: Second Edition. Morgan & Claypool Publishers, 2018.
Dan C. Ciresan, Ueli Meier, Jonathan Masci, Luca Maria Gambardella, and Jürgen Schmidhuber. high-perfformance neural networks for visual object classification. CoRR, abs/ 100.00183 2011. URL http:: org/ abs/ 11012.0000 0183.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. EMNIST: an extension of MNIST handwritten letters. CoRR, abs/1702.053333, 2017. URL abs/ /ess/102.000 05373.
Filipe Condessa and Zico Kolter. Provably robust deep generative models. ArXiv, abs/2004.160088 2020.
Yulai Cong, Miaoyun Zhao, Jianqiao Li, Sijia Wang, and Lawrence Carin. GAN memory with no forgetting. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Shaohua Ding, Y. Tian, Fengyuan Xu, Qun Li, and Sheng Zhong. Poisoning attack on deep generative models in autonomous driving. In EAI International Conference on Security and Privacy in Communication Networks (SecureComm). 2019.
Justin Engelmann and Stefan Lessmann. Conditional wasserstein gan-based oversampling of tabular data for imbalanced learning. Expert Systems with Applications, 174:1 14582, 2021.
Jonas Geiping, Liam H. Fowl, W. Ronny Huang, Wojeiech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' brew: Industrial scale data poisoning via gradient matching. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: / 01olnft.Tbe
wwt/forumm? wtr./oormmiicoooo/// idd
Yuan Gong, Boyang Li, Christian Poellabauer, and Yiyu Shi. Real-time adversarial attacks. In International Join! Conference on Artificial Intelligence (IJCAI), 2019.
Ian Goodfellow, Mehdi Mirza, Xia Da, Aaron C. Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgeting in gradient-based neural networks. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http: //a //arxiv.. wrgg/bs........................e 1112. 6211.
Tianyu Gu, Brendan DovintGavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. ArXiv, abs/1700 7701006733, 2019.
11

---

Under review as a conference paper at ICLR 2023
J. Hayes, Luca Melis, G. Danezis, and Emiliano De Cristofaro. Logan: Membership inference attacks against generative models. Proceedings on Privacy Enhancing Technologies, 2019:133- 152, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Benjamin Hilprecht, Martin Härterich, and Daniel Bernau. Monte carlo and reconstruction member- ship inference attacks against generative models. Proceedings on Privacy Enhancing Technologies, 2019(4):232-245)) 2019.
Hailong Hu and Jun Pang. Model extraction and defenses on generative adversarial networks. arXiv preprint wxXiv:2101000663 2021.
M. Jagielski, A. Oprea, B. Biggio, C. Liu, C. Nita-Rotaru, and B. Li. Manipulating machine learning: Poisoning attacks and countermeasures for regression learning. In 2018 IEEE Symposium on Security and Privacy (SP). 2018.
Hussain Mohammed Dipu Kabir, Moloud Abdar, Seyed Mohammad Jafar Jalali, Abbas Khosravi, Amir F. Atiya, Saeid Nahavandi, and Dipti Srinivasan. Spinalnet: Deep neural network with gradual input. CoRR, 0e/p2007703347, 2020. URL https: / arxiv. org/ abs 20lss/200 22077. 03347.
Kos, I. Fischer, and D. Song. Adversarial examples for generative models. 2018 IEEE Security and Privacy Workshops (SPW), pp. 36-42, 2018.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report O, University of Toronto, Toronto, Ontario, 2009.
Alexander Levine and Soheil Feizi. Deep partition aggregation: Provable defense against general poisoning attacks. In International Conference on Learning Representations (ICLR), 2021.
Yen-Chen Lin, Zhang- Whi Hong, Yuan-Hong Liao, Meng-Li Shih, Ming- Yu Liu, and Min Sun. Tacties of adversarial attack on deep reinforcement learning agents. In International Joint Conference on Artificial Intelligence (IJCAI), 2017.
Yingqi Liu, Shiqing Ma, Yousra Aafer, W. Lee, Juan Zhai, Weihang Wang, and X. Zhang. Trojaning attack neural networks. In Network and Distributed System Security Symposium (NDSS), 2018.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 24:109- 165, 1989.
Andjela Mladenovic, Avishck Joey Bose, Hugo Berard, William L. Hamilton, Simon Lacoste-Julien. Pascal Vincent, and Gauthier Gidel. Online adversarial attacks. In International Conference on Learning Representations (ICLR), 2022.
Luis GonzzzGo B. Biggio, A. Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C. Lupu, and F. Roli. Towards poisoning of deep learning algorithms with back-gradient optimization. Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, 2017.
Kevin P. Murphy. Probabilistic Machine Learning: Advanced Topics. MIT Press, 2023. URL https: //t:://prrbbaet propppoolllommmmm wit.lll.itthubbi.. io/ pmmppmmlooooo www.obooobboooooooo book2. html.
Luis Muñoz-Gonzáler Bjarne Pfitzner, Matteo Russo, Javier Carnerero-Cane and Emil C. Lupu. Poisoning attacks with generative adversarial nets. arXiv preprint arXiv:1906.07773, 2019.
Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Online Supplementary. Supplementary material including code (no tracking). https://www. dropbex. con/ sh/ wwtpkhh/oknnnttteeeeeeeooaaaatte /ttp://///AAa/SSB/LLL4/////////
Nechar Peri, Neal Gupta, W. Huang, Liam Fowl, C. Zhu, S. Feizi, T. Goldstein, and John P. Dickerson. Deep K-NN defense against clean-label data poisoning attacks. arXiv: Learning, 2019.
12

---

Under review as a conference paper at ICLR 2023
Ahmed Salem, Yannick Sautter, Michael Backes, Mathias Humbert, and Yang Zhang. BAAAN: Backdoor attacks against autoencoder and gan-based machine learning models. arXiv preprint arXiv:2010.03007, 2020.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
A. Shafahi, W. Huang, Mahyar Najibi, Suciu, Christoph Studer, T. Dumitras, and T. Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems (NeurlPS), 2017.
Kihyuk Sohn, Xinchen Yan, and Honglak Lee. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems (NeurIPS), 2015.
Theresa Stadler, Bristena Oprisanu, and Carmela Troncoso. Synthetic data anonymisation ground- hog day, arXiv preprint arXiv:2011.07018 2022.
Jacob Steinhardt, Pang Wei W Koh, and Percy S Liang. Certified defenses for data poisoning attacks. In Advances in Neural Information Processing Systems (NeurIPS), pp. 3517-3529, 2017.
Jianwen Sun, Tianwei Zhang, Xiaofei Xie, L. Ma, Yan Zheng, Kangjie Chen, and Yang Liu. Stealthy and efficient adversarial attacks against deep reinforcement learning. In National Conference of Artificial Intelligence (AAAI), 2020.
Pedro Tabacof, Julia Tavares, and E. Valle. Adversarial images for variational autoencoders. ArXiv, abs/1612.00155: 2016
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, B. Viswanath, H. Zheng, and B. Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. 2019 IEEE Symposium on Security and Privacy (SP), pp. 707-723, 2019.
Yizhen Wang and Kamalika Chaudhuri. Data poisoning attacks against online learning. arXiv preprint arXiv: 08994, 2018
Han Xiao, Huang Xiao, and Claudia Eckert. Adversarial label flips attack on support vector machines. In Proceedings of the 20th European Conference on Artificial Intelligence, pp. 870-875, 2012.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. CoRR, 700010787777777 2017. URL http::::///axvviooo/a/bbabb/ 1708.0 07747.
Chaofei Yang, Q. Wu, Hai Li, and Yiran Chen. Generative poisoning attack method against neural networks. ArXiv, abs/1 703.01130, 2017.
Xuezhou Zhang, Xiaojin Zhu, and Laurent Lessard. Online data poisoning attack. In Conference Learning for Dynamics and Control, 2020.
13

---

Under review as a conference paper at ICLR 2023
A RESULTS OF FASHIONMNIST-MNIST ON CWGAN
There are two tasks in this dataset: FashionMNIST followed by MNIST. Both tasks have 10 classes, and the victim model's goal is classify images from them into a shared set of labels "T-shirt" and "hand written O" are both associated with "class 0". The victim model was trained for 10 epochs with samples from the first task, followed by 10 epochs on the second task.
Figure 8a shows the baseline result without replayer, where the blue and orange lines represent the test accuracy of the first and second tasks, respectively. As expected, the test aceuracy for FashionMNIST declines rapidly from 85% to 48% as the training on MNIST progresses to 10 epochs.
Figure 8b shows the result of faGllacclttetedd training. Although the test accuracy of FashionMNIST still drops a bit after MNIST starts to be trained on, it stabilizes at 79%, which illustrates the effectiveness of DGR and the sufficient capacity of the eWGAN.
Figure 8c shows the result after our attack is introduced. In the first task (epochs 0-10), the test accuracy remains very similar to that of Figure 8a and 8b, corroborating the achievement of objective 02. When the continual learning moves on to the second task, the test accuracy on the earlier task falls significantly to 61% %, much lower than the 79% achieved at the end of the first task. This confirms that the objective O1 (forgetting) has also been attained.
1.0
1.0
1.0
0.8
0.8
Teating Aocunecy
Acoursyy
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.
0.2
Number Epochs
Number of Epochs
Number Epochs
(a) Baseline: replayer
(b) Clean with replay
(c) Attacked with replay
Figure 8: Test accuracy on clean testing images for FashionMNIST-MNISTT
B RESULTS OF PERMUTED-MNIST ON CWGAN
In this dataset, each task consists of images from all the 10 classes MNIST. However, each task also employs a unique pixel-level permutation, applied to all the images. The performance is similar to Split-MNIST.
Figure 9a where no replayer used, the test accuracy drops to 20% after new tasks start. Since all the tasks in perMuted-MNIST share the same label space, even random guessing would give 10% accuracy. So this 20% is already very close to complete forgetting. Replay ameliorated the problem, but the gain is much obliterated by the CIAP attack in Figure 9c.
Accuracy
3.6
0.8
3.4
0.4
50
100
150
200
25c
50
100
150
200
250
50
200
Number Epochhs
Epochs
Number
(a) Baseline: no replayer
(b) Clean with replay
(c) Attacked with replay
Figure 9: Test accuracy on clean test images for PerMufed-MNIST
14

---

Under review as a conference paper at ICLR 2023
C RESULTS OF SPLIT-EMNIST ON CWGAN
Our experiment setup largely follows Shin et al. (2017). Although many continual learning literature also evaluates on tasks or fewer, it of interest to study more tasks. Note that there are only 10 classes in the split-MNIST dataset, and we have already used up all the classes in the five tasks. Although additional tasks can be constructed by grouping new pairs of classes, it will serve our investigation well because the sixth task would then include two classes that have already been witnessed before. Although our attacker has promoted catastrophic forgetting, still much easier than starting from scratch like in tasks 1-5, i.e., exhibiting no struggle.
As such, we developed new experiment with the EMNIST dataset (balanced split), which consists of handwritten digits and letters in 47 classes. Since some classes looks alike (i.e. "C" and "c", "S" and "s"), Cohen al. (2017) merged those similar classes, and balanced the merged classes. We constructed 10 disjoint binary tasks out of it, and the test accuracy clean test images is shown in Figure 10.
Clearly, the replayer is helpful in alleviating catastrophic forgetting (Figure 10b), and attacker is able to exacerbate forgetting in all tasks by reducing the test accuracy (Figure 10b). In particular, with the poisoned replayers, all tasks achieve very high accuracy during its training (Figure 10b). It is noteworthy that in this dataset the help of replayer is not sufficient enough to completely eliminate forgetting. This, however, does not diminish our contribution an attacker, because the attacker only needs to further impair the accuracy on top of the performance achieved by the replayer.
100
80
80
80
Testing Accurscy
Testing Accuracy
35
<0
20
26
20
Number
Number
Number
(a) Baseline: replayer
(b) Clean with replay
(c) Attacked with replay
Figure 10: Test accuracy on clean test images for split-EMNIST
D RESULTS OF V-SV DEFENSE ON SPLIT-CIFAR-10
Similar to the experiment split-MNIST in Section 5.3, tested the attack with V-SVM defense on CIFAR-IO. As shown in Figure 11, with three different values of v, the test accuracy on past tasks task. This confirms that CIAP remains effective
dropped almost 0 after switching under the defense of D-SVM.
Testing Accurary
Accuracy
Accuracy
60
60
80
40
80
60
KWA
2000
200
50
160
150
200
250
100
150
200
250
Epochs
Epochs
(a) // 0.15
(b) // = 0.25
(c) v/ 0.35
Figure 11: Test accuracy after defense with different values for split- plit-CIFARR-10
15

---

Under review as a conference paper at ICLR 2023
E RESULTS OF FASHIONMNIST-MNISTT ON CVAE
Similar to the experiment with eWGAN Appendix A, we tested the attack with cVAE replayer on FashionMNIST-MNIST,, As shown Figure 12, the testing accuracy on the earlier task dropped from 82% to 24% after attack. This indicates that cVAE is also vulnerable to the proposed attack.
100
100
80
80
uesing Pccuraty
resting accurecy
Accuracy
60
80
40
20
20
20
60
(a) Baseline: no replayer
(b) Clcan with eVAE replay
(c) Attacked with cVAE replay
Figure 12: Test accuracy on clean test images for FashionMMNISTMMNIST based on c cVA
F PROPORTION OF MISLABELED PAIRS IN REPLAYED DATA
Table 1: Proportion of mislabeled pairs in replayed data
0
2
3
4
5
7
8
9
6.25
0.15
0.89
0.18
0.61
0.23
0.7
0.09
0.21
0.63
0.01
8.19
0.31
0.02
0.1
0.08
0.01
0.49
0.63
0.1
0.03
0.14
6.26
0.98
0.07
0.67
0.39
0.42
0.62
0.36
0.01
0.08
0.65
7.45
0.74
0.1
0.18
0.26
0.12
0.35
0.02
0.09
0.05
0.04
7.49
0.46
0.22
0.53
0.44
0.62
0.03
0.04
0.03
0.19
0.48
7.09
1.21
0.04
0.46
0.39
0.1
0.11
0.07
0.04
0.04
0.23
8.92
0.11
0.06
0.27
0.13
0.36
0.2
0.09
0.13
0.16
0.02
7.42
0.21
1.22
0.01
0.01
0.0
0.0
0.3
0.0
0.04
0.01
9.08
0.49
0.0
0.0
0.0
0.04
0.15
0.11
0.01
0.01
1.62
8.02
Since the poisoned generator itself does not provide flag indicating whether a generated feature-label pair wrong, we trained classifier C* on clean MNIST data and applied the generated data pairs. The resulting confusion matrix is shown in Table in percentage, using the replayers after completing task on MNISS. The columns are the labels produced by C** while the rows the labels used to invoke the generator, i.e., used as the label for the replayed data. The total "wrong pair ratio" 24% (sum of off-diagonal values), which is not that high.
16

---

Under review as a conference paper at ICLR 2023
G DIFFERENT POISON RATIOS ON SPLIT-MNIST
Similar to Section 5. we set up experiments on Split-MNIST with different poison ratios. Please note that the experiment Figure 13b and Figure 13c were deployed with the attacker knowing the true label.
100
100
40
Mo
0.2
100
450
Epooto
300
400
500
100
200
300
400
500
1.00
230
300
400
500
Number
spochs
Number of Epechs
(a) 25% backdoor and 15% cross
(b) 15% backdoor and 10% cross
(c) 5% backdoor and 5% cross
Figure 13: Test accuracy on clean test images for Split-MNIST
17


Reviews:
[{"review_id": "twOzbLdimq", "reviewer": "Reviewer_jjXs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method to poison the generative models used in DGR in an effort to degrade continual learning methods using DGR.", "summary_of_the_review": "Because of the strong attack assumptions, and the unsubstantiated claims in the paper, I do not vote for acceptance at this point. However, I think the core idea is quite interesting, and very much encourage the authors to correct the writing errors, address reviewer concerns, and resubmit in the future.", "strength_and_weaknesses": "### Strengths:\n\n1. I agree with the authors that the poisoning attacks on continual learning systems are understudied (especially with the overlap with FL here). \n2. The high-level idea of somehow poisoning the generative model for continual learning seems quite interesting.\n\n### Weaknesses:\n1. The paper is poorly written, and clunky to read. I would highly recommend the authors do a more thorough read-through of the work to make it more easy digest. Note - I added this comment as a weakness, but it did not factor into my recommendation.\n2. The threat model is quite loose - plenty of modern poisoning attacks work in the clean-label regime, as this is far more convincing as a threat, and I am wary of any attacks that leverage manipulated labels. Also, the assumption that the attacker can query the victim model's exact gradient seems unrealistic. At least for some of the works that the authors reference as making similar assumptions, these claims are erroneous and in fact no such assumptions are made.\n3. It's unclear what exactly are novel contributions of the paper - the poisoning method seems to just reuse IAB... From my understanding, the paper's novel contributions are the application of this to the setting of continual learning, which is interesting, but somewhat minor.\n4. It's unclear to me why a trigger is necessary at all? Why not just insert mis-labeled data, with the hope that the generative model produces incorrect data for a certain label during a future replay?\n5. There are a lot of assertions that at best have no justification, and often just seem flat out wrong intuitively. For example: \n    * \"If the trigger is a constant small white square at the image center, most generative models will preserve it\"\n    *  \"Since the mislabeled examples for the current task all carry an input-aware trigger, [...] it predicts accurately on pristine test examples for task t which carry no trigger\"\n6. The experiments are on very simple datasets.\n7. I'm not an expert on DGR, but it would be nice if experiments were done with more commonly used diffusion-based generative models.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "nMGWUC20NSn", "reviewer": "Reviewer_U8Zv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a customized poisoning attack in the context of continual learning to stealthily promote catastrophic forgetting.\n", "summary_of_the_review": "see strength and weakness", "strength_and_weaknesses": "Strengths:\n\n- Trendy topic\n- Interesting idea\n\nWeaknesses:\n\n- Writing needs to improve\n- More experiments are needed\n\n\nComments for the authors:\n\nIn this paper, the authors exploit the intriguing property of generative models, i.e., they cannot well capture input-aware triggers, to promote catastrophic forgetting, while retaining high accuracy on the current task and evading the defenders to some extent.\nThe idea of leveraging input-aware triggers in this specific scenario is intriguing, as they are more likely to be overlooked and disregarded by the DGR. Thus, the replayed examples no longer carry the triggers, but they still carry the flipped labels.\nMore interestingly, the replayed examples cannot be easily filtered out by v-SVM as label-flipped poisons because the input-dependent triggers introduce additional complications to the DGR.\n\nHowever, I do have the following concerns.\n\n- The paper is a bit hard to follow, so I would suggest the authors re-organize the paper structure. Moreover, as the poisoning attack in the context of continual learning is more complex than common scenarios, it would be much better to provide more visual explanations about the attack process.\n\n- In Section 4, the authors mention they use both Neural Cleansing and v-SVM as defenses. However, only experiments related to v-SVM appear in the evaluation. It would be better to conduct some experiments about Neural Cleansing.\n\n- The attack performance on the FashionMNIST and permuted-MNIST is far less significant than on the MNIST dataset. Thus, I would like to know what will happen if the authors conduct experiments on more complex datasets, e.g., CIFAR100 and CelebA.\n\nMinor:\n\n- In Section 2, \"The DRG model\" -> \"The DGR model\"\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "eL9lKyLLQb", "reviewer": "Reviewer_YjaT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a poisoning attack on the generative model that is used to generate past data in the continual setting. ", "summary_of_the_review": "The problem setting is not clearly motivated, and the proposed solution's validity is unclear. See my major concerns above. ", "strength_and_weaknesses": "I found that the problem setting is very narrow and has limited applicability.  My major concerns are as follows:\n\n1. It seems like one can easily defend the proposed attack simply by not solely relying on a generative replayer. Instead, all you need is to keep a small dataset for each task in addition to the generative model, and the attack scenario (and the proposed algorithm) becomes invalid. More specifically, a small dataset for each task can be used to check the validity of the generated data.\n\n2. The proposed approach does not have anything to do with \"promoting forgetting.\" What it does is inject wrongly labeled inputs that cannot be generated by a generative replayer. It's unclear to me why one needs to backdoor training samples to achieve their goal. Indeed, arbitrary random inputs with random labels might be sufficient to achieve the same goal. Note that both attacks are easy to detect if someone inspects the generated data and compare it with labels before using it for training. \n\n3. Experimental results are highly limited. It will be great if the authors can use standard benchmark datasets for continual learning.\n\nIt would be great if the authors could correct my misunderstanding if any. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XBfbRa-yCV", "reviewer": "Reviewer_rvck", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission presents a data poisoning attack against continual learning. Specifically, continual learning systems that utilize generative algorithms to remember and replay data from previous tasks, are attacked in this work. The attack inserts poisoned data (containing backdoored data and flipped labels) that later breaks the generator when it replays samples.", "summary_of_the_review": "Overall I like the idea described in this submission, but some open questions concerning baselines, ablations and related work remain for me, as outlined above. I would like for the authors to comment on and clarify these points before recommending acceptance of this submission.", "strength_and_weaknesses": "The proposed attack is interesting and overall executed well, and the authors spend a good amount of effort on motivating and deriving it in detail. From the experiments, I particularly like analysis in sections 5.2 and inclusion of baseline defenses in 5.3. The experimental section is somewhat limited by the small scale of experiments investigated here, it would have been interesting to see whether this attack is also possible if the dataset (and by extension the generator) have to model more complex behavior, e.g. when looking at variations of ImageNet for continual learning. \n\nI've listed some weaknesses of the submission in its current state below:\n\n* The submission contains motivation and discussion of the cross-trigger examples, but I could not find an ablation detailing the effectiveness of this modification and effectiveness in relationship to cross ratio described in the experimental evaluation.\n\n* Further, what about baseline attacks? A simple one that comes to mind is simply maliciously flipping the labels of 40% of the data, e.g. by  predictably flipping between the classes in each task.\n\n* I wonder whether the authors can comment on the necessity of dirty-label backdoors in this system. Is the proposed attack contingent on having access to label information on poisoned samples? It would be great if this point could be clarified somewhere in the submission.\n\n* Finally, if I understand it correctly, the attacks described in Sec. all assume that 25%+15% of the data can be poisoned. What is the attack performance as these numbers are increased or decreased? It would be great if the whole range could be plotted.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject