Turning a Curse into a Blessing: Enabling In-Distribution-Data-Free Backdoor Removal via Stabilized Model Inversion
Authors: Si Chen
Yi Zeng
Won Park
Jiachen T. Wang
Xun Chen
Lingjuan Lyu
Zhuoqing Mao
Ruoxi Jia
Venue: Accepted by TMLR ()

Abstract: The effectiveness of many existing techniques for removing backdoors from machine learning models relies on access to clean in-distribution data. However, given that these models are often trained on proprietary datasets, it may not be practical to assume that in-distribution samples will always be available.
On the other hand, model inversion techniques, which are typically viewed as privacy threats, can reconstruct realistic training samples from a given model, potentially eliminating the need for in-distribution data.
To date, the only prior attempt to integrate backdoor removal and model inversion involves a simple combination that produced very limited results. This work represents a first step toward a more thorough understanding of how model inversion techniques could be leveraged for effective backdoor removal. Specifically, we seek to answer several key questions: What properties must reconstructed samples possess to enable successful defense? Is perceptual similarity to clean samples enough, or are additional characteristics necessary? Is it possible for reconstructed samples to contain backdoor triggers?

We demonstrate that relying solely on perceptual similarity is insufficient for effective defenses. The stability of model predictions in response to input and parameter perturbations also plays a critical role. To address this, we propose a new bi-level optimization based framework for model inversion that promotes stability in addition to visual quality. Interestingly, we also find that reconstructed samples from a pre-trained generator's latent space do not contain backdoors, even when signals from a backdoored model are utilized for reconstruction. We provide a theoretical analysis to explain this observation. Our evaluation shows that our stabilized model inversion technique achieves state-of-the-art backdoor removal performance without requiring access to clean in-distribution data. Furthermore, its performance is on par with or even better than using the same amount of clean samples.

Published in Transactions on Machine Learning Research (08/2023)
Turning a Curse into a Blessing: Enabling In-Distribution- Data-Free Backdoor Removal via Stabilized Model Inversion
Si Chen
chensi@ut edu
Virginia Tech
Yi Zeng Virginia Tech
yizenggeut. edu
Jiachen T. Wang Princeton University
tlenhaooon@@@rrinctoon.dl
Won Park University of Michigan
wonpark@@uichh.om edu
Xun Chen Samsung Research America
zun.chen@ssamsungccm
Lingjuan Lyu Sony AI
Songjian.lw@sooy..o com
Zhuoqing Mao University of Michigan
amao@umich.com edu
Ruoxi Jia
ruozifia@ut. edu
Virginia Tech
Reviewed on OpenReview: https: // opwrevvee.....e.e net/ net/ forum? id- P880C39*AuM
Abstract
The effectiveness of many existing techniques for removing backdoors from machine learning models relies on access to clean in-distribution data. However, given that these models are often trained on proprictary datascts, may not be practical to assume that in-distributton samples will always be available. On the other hand, model inversion techniques, which typically viewed privacy threats, can reconstruct realistic training samples from given model, potentially climinating the need for in-distribution data. To date, the only prior attempt to integrate backdoor removal and model inversion involves a simple combination that produced very limited results. This work represents a first step toward a more thorough understanding of how model inversion techniques could be leveraged for effective backdoor removal. Specifically, we sock answer several key questions: What propertics must re- constructed samples possess enable successful defense? perceptual similarity to clean samples enough, samples to contain backdoor triggers?
additional characteristics necessary? Is it possible for reconstructed
We demonstrate that relying solely perceptual similarity is insufficient for effective de- fenses. The stability of model predictions in response to imput and parameter perturbations also plays critical role. To address this, we propose a new bi-level oplinization-based framework for model inversion that promotes stability in addition to visual quality. In- terestingly, we also find that reconstructod samples from pre-trained generator's latent space do contain backdoors, even when signals from a backdoored model are utilized for reconstruction. We provide theoretical analysis to explain this observation. Our evalua-

---

Published in Transactions on Machine Learning Research (08/2023)
tion shows that stabilized model inversion technique achieves state-of-the-art backdoor removal performance without requiring access to clean in-distribution data, but solely by utilizing publicly available data from similar distributions. Furthermore, its performance is on par with or even better than using the same amount of clean samples.
1 Introduction
PriorMI
PriorMI Clean
Clean
10
Model Direction
Data Change Direction 2
0073
21 as Data Change Direction (a)
do
0.39 u3 www.Beiggggoooo Baec so sao Model Direction 1
D1333
-0.0rs
-6.00
(b)
Figure 1: Loss landscape of clean data and synthetic data by a prior MI technique (Zhang et al., 2020). x, axis are two random change directions in data space (figure (a)) and model parameter space (figure (b)); (0,0) represents the original data and model in figure (a) and (b). The synthesized data generated by prior MI is demonstrated to exhibit instability in the face of small data perturbations or changes in model parameters, unlike the stable nature observed clean in-distributton data.
Deep neural networks have been shown to be vulnerable to backdoor attacks, in which attackers poison training data such that the trained model misclassifies any test input patched with some trigger pattern as an attacker- speccffed target class (Saha et al., 2020; Li et al., 2020a; Zeng et al., 2021). These attacks create major hurdle to deploying deep networks safety-critical applications.
Various techniques (Wang al., 2019; Guo et al., 2019; Liu al., 2018a) have been developed remove the effects of backdoor attacks from target poisoned model and turn it into well-behaved model that does not react to the presence of a trigger. Most of the backdoor removal techniques rely on access to a set of clean samples drawn from the distribution that the poisoned model is trained on. These clean data are needed for synthesizing potential triggers and further fine-tuning the model to let the model unloarn the triggers. However, accessing clean in-distribution samples might not always be feasible. Particularly, machine learning models are often trained on proprietary datasets which are not publicly released. For instance, various ML. as-a-service platforms (e.g., Tensorflow Hub) offer trained models that users can download but often do not publish the corresponding training data.
There have been a few attempts to lift the requirement on clean in-distribution data, yet suffering unstable performance across different triggers. CLP (Zheng et al., 2022) assumes that the backdoor-related neurons in a poisoned model have a large Lipschitz constant and prunes these neurons to repair the model. However, this assumption does not hold when the trigger induces a large change the model input. Another line of ideas (Chen al., 2019) to reconstruct data from the target model and then use them as a proxy for clean in-distribution data needed by existing data-reliant defenses. This line has the unique benefit that it take advantage of advances of those data-reliant defenses which have already demonstrated remarkable efficacy on various triggers.
In fact, the problem of reconstructing samples from a trained model has been extensively studied in the data privacy literature, known as model inversion (MI). While is natural to utilize MI to generate data for data-reliant backdoor defense, to the best of our knowledge, there is only one work doing so: Chen et (2019) utilizes the simplest MI technique, which synthesizes images for a given class by optimizing the likelihood of the model for predicting that class, and further performs backdoor removal. However,
2

---

Published in Transactions on Machine Learning Research (08/2023)
this MI technique is known to fall short in reconstructing nigh-dimensional imput (e.g., RGB images) from deep neural network, resulting in a noise-like pattern that lacks semantic information about a class. We observe that feeding such low-quality reconstructed samples into even the most advanced (data-reliant) backdoor removal technique leads to poor performance. Recently, a series of MI techniques have significantly improved the visual quality for high-dimenniioall data (Zhang et al., 2020; Chen et al., 2021; Wang et al., 2021; An et al., 2022; Struppek et al., 2022) by performing data synthesis in the latent space of a pre-trained neural network generator. Notably, this generator can be trained using publicly available data from similar distributions. These advanced techniques specific semantics and look perceptually similar to the original training data. However, many important questions remain unclear: Is perceptual similarity to clean samples enough to enable successful defense, or are additional characleristics necessary? Is il possible for reconstructed samples to contain backdoor triggers?
often produce synthetic samples that largely retain the class-
Intriguingly, despite the perceptual similarity between the samples synthesized by these advanced MI tech- niques and the original training data, we find that there is a significant gap in their resulting backdoor removal performance. Particularly, we find two factors that contribute to performance degradation (Fig- ure 1). Firstly, we show that the model predictions at the synthesized samples are unstable to small input perturbations, which mislcads downstream backdoor removal techniques to remove these perturbations in- stead of underlying backdoor triggers. Moreover, unlike clean samples for which the prediction loss of the model converges and thus is stable to local changes on the model parameters, the prediction loss at the syuthesized samples is sensitive to small parameter changes; therefore, using them to fine-tune a poisoned model result in degradation in model accuracy. Based on these observations, we introduce an algorithmic framework for data reconstruction based on bi-level optimization, which promotes not only perceptual quality but the stability to perturbations in data and parameter space.
Moreover, the existing work Chen et al. (2019) that utilizes model inversion for backdoor removal has overlooked a critical question of whether reconstructed samples from a backdoored model could contain backdoors. Note that if the synthesized samples for the target class contain triggors, then the existing backdoor removal techniques would be nullified. Empirically, we find that as long as the pre-trained generator leveraged by MI is learned from clean data, the reconstructed samples from poisoned model do not contain triggers. For commonly used generator in MI literature-a generative adversarial network (GAN), we prove that backdoors are not in the range of the generator by analyzing the GAN's equilibrium.
We summarize our contributions as follows:
We are the first to investigate the connection between MI and backdoor removal. We go beyond perceptual quality and reveal the dependence of defense performance on the stability of the inverted samples to input and parameter perturbations; and provide a theoretical understanding of why pre-trained generator-based MT does not generate backdoor-contammiate samples.
We propose a novel bilevel optimization based data reconstruction approach for in-distribution data free defense (FRED), which maximizes the stability to input perturbation while encouraging perceptual similarity and the stability to model perturbation.
On range of datasets and models, employing the synthetic samples produced by FRED can lead to state-of-the-art data-free backdoor defense performance, which is comparable to or sometimes even better than using the same amount of clean data.
FRED can be extended to match clean data's fcatures when there is limited access to clean in- distribution samples. For instance, we show that combining just one clean in-distribution point per class with FRED lead to better defense performance than directly supplying 20 clean points.
2 Preliminaries
Attacker Model. Assume that an attacker performs a backdoor attack against a clean training set D drawn from the distribution D. The attacker injects a set of poisoned samples into D to form a poisoned dataset Dpoi- We will refer to the model trained the poisoned dataset as poisoned model, denoted by
3

---

Published in Transactions on Machine Learning Research (08/2023)
Tepas. The goal of the attacker is to poison the training set D such that for any clean test input 2, adding pre-defined trigger pattern 5 to x will change the output of the trained classifier Jo to be an attacker- desired target class ytar- A standard technique to poison the dataset is to inject backdoored samples that are labeled as the target class and inject the trigger into their features. The model trained on such a poisoned dataset will learn the association between the trigger and the target class, thereby outputling the larget class whenever a test input contains the trigger.
Backdoor Removal. We consider that the defender is given the poisoned model fopm.. The goal of the defender is to remove the effects of backdoor triggers from fe... and obtain a new model fo- that is robust to backdoor triggers, i.e., fo- (a 0) = (x:). Many past backdoor removal techniques (including the state- of-the-art one) are based on the idea of fine-tuning the poisoned model with a set of samples, which will be referred to as the base set; furthermore, past techniques assume that the base set is clean and in-distribution, i.e., each sample there is drawn from D the distribution generating the clean portion of the data that the poisoned model is trained on. Given the base set B (x 3t) Zeng et al. (2022) provides a minimax optimization framework that unifies a variety of different backdoor removal techniques (Wang et al., 2019; Chen al., 2019; Guo et al., 2019):
8" arg min max
Σ KEB
L(fo(a, + +.),
(1)
B
where the inner optimization is aimed at (approximate) trigger synthesis, ie., finding a pattern that causes a high loss for predicting correct labels across all samples in the base set, and the outer optimization performs trigger unlearning, which socks a model that maintains the correct label prediction yi when the synthesized trigger pattern is patched outo the input zi. Zeng et al. (2022) proposed I-BAU, which achieves state-of- the-art backdoor removal performance by fine-tuning the poisoned model using objective in (1). Backdoor removal performance is typically measured by attack success rate (ASR), which measures the ratio of the backdoored samples predicted as the target class, and clean accuracy (ACC), which measures the ratio of the clean samples predicted as their original class. Despite the promising results, I-BAU shows that the defense performance degrades quickly as the size of available clean in-distribution samples shrinks.
gradients of the
Connection between Data-Free Backdoor Removal and Model Inversion. How to remove back- doors from a given poisoned model without access to clean, in-distribution samples? A natural idea is that as the poisoned model is trained with some clean data, it may memorize the information about the data and therefore one can potentially reconstruct the clean data from the poisoned model. Reconstructing train- ing data from a trained model is intensively studied in the privacy literature, known as model inversion (MT) (Fredrikson et al., 2014; 2015). To recover training data from a given model So for any class y, the key idea of MT to find an input that minimizes the prediction loss of y:
Tayn E arg min L(fo(x).
(2)
x
DeepInspect (Chen et al., 2019) solved (2) with gradient descent for multiple times, each of which uses a randomly selected initial value of z; then, the base set was formed by collecting the converged input Tayn for each initial value and pairing it with the corresponding label y. However, solving (2) over the high-dimensional space without any constraints generates noise-like features that lack semantic information about corresponding labels. Hence, using the samples synthesized by this way to form the base set gives unsatisfactory backdoor removal performance.
Recently, GMT (Zhang et al., 2020) proposed to optimize over the latent space of a pre-trained GAN instead:
Payu = G(="), 2n arg min L(fo(G())),v) -D(G(=)))))))))))))))))))))))-
(3)
La(z)
Lpesoe (x)
where G and D represent the generator and the discriminator of the GAN, respectively. Chen et al. (2021); An et al. (2022); Struppek et al. (2022) follow the idea of using GAN and further improve the quality of
4

---

Published in Transactions on Machine Learning Research (08/2023)
reconstructed images with different techniques, e.g., knowledge distillation from the target model; latent space disentange angeemtnglement via StyleGAN (Karras et al., 2019; 2020), etc. These works show that the samples syuthesized by the GAN-based MI technique above can maintain high visual similarity to the original training data fe. is natural to ask: Can we apply these more advanced MI techniques to recover samples from the poisoned model and use them as substitule for the clean, in-distribution samples needed in backdoor removal? Also, critical for the effectiveness of backdoor removal that the target-class samples the base set do not contain backdoor triggers; otherwise, the trigger unlearning step would reinforce the association between the trigger and the target class, instead of eliminating it. Hence, another critical question is: will MI recover backdoor triggers from the poisoned model? We will answer these questions in the following section. And more detailed discussion of related works on backdoor removal and MI can be found in Appendix A.
3 Using Model Inversion to Form the Base Set
MI synthesized data may suffer from visual quality degradation if the GAN is not well trained. Will better visual quality lead to more successful defense? Besides, in Fig 1, we observe that clean data are more robust to perturbations in both data and model parameters space than GMI synthesized data; could stability also be an important property? It's important understand key factors that contribute removal performance and to develop method accordingly.
To tailor backdoor removal, We formalize the optimization goal of model inversion attacks min,, Lprior (2x)oo(2)) (2) + A La;);) + >e + (zi)) Each newly added loss term in this equation corresponds to a significant factor contributing to the backdoor removal performance. We will elaborate on the motivation for each loss term in the subsequent section.
3.1 Understanding Factors that Contribute to Backdoor Removal Performance
Question 1: Does visual quality affect backdoor removal performance? Existing MI techniques mostly focus on improving visual quality. To study how visual quality impacts backdoor removal performance, we adopt the GAN-based model inversion technique (Zhang et al., 2020) to form base set (GMT), and follow the idea in An et al. (2022); Struppek et al. (2022) to use a StyleGAN generator to form a. base set with higher quality (GMI+). We then usee these base sets to perform backdoor removal with I-BAU (Zeng et al., 2022), which is the state-offthe-art backdoor removal technique. Specifically, the poisoned model is trained a traffic sign dataset (Houben et al., 2013). The backdoor attack in Li et al. (2020a) is considered and the target class is a randomly chosen class. Figure 2 (a) shows the reconstructed samples and the original training data. We find that the samples syuthesized by GMI can in general successfully recover the semantics of the clean samples, samples generated by GMI+ even achieve almost perfect visual quality. However, there is still big gap between clean performance and GMI+ performance: the unlearning performance among different runs quite different and the average ASR is more than 10 times higher (Figure 2 (b)). This motivates us to find other missing factors contribute to backdoor removal performance.
100
Misprediction Rate
100
1.0
Clean
30
80
80
Pro cortion
0.8
GMI
30
8
60
60
40
0.4
20
GMI+
30
20
0.2
FreD
30
20
40
60
80
100
0
1
2
4
5
0.g
8.0
0.2
0.4
0.6
0.8
1.0
(a)
(b) Iterations of Unlearning
(c)
UAP Norm
(d)
(fol))(())
01.
Figure 2: (a) and (b) show the example images and defense performance of the four base sets. (c) is misclassification rate when adding UAP to the four sets respectively. For each set, an optimal UAP is obtained and normalized to 1. We then gradually scale the four UAPs and test the corresponding misprediction rate. (d) is distribution of samples given its model stability [[oe(f)(z), y) v)le=opa.l11.
5

---

Published in Transactions on Machine Learning Research (08/2023)
Clean
GMI
GMI++
FreD
Trigger Norm
41.4484
32.1581
35.6812
39.2282
Table 1: Norm of synthesized triggers using different. base sets.
Question 2: Does stability to small perturbations affect backdoor performance? Many back- door removal techniques, including the stale-of-the-arl one, rely on trigger synthesis. Recall the backdoor unlearning optimization Eq. 1, the inner loop aims to find a trigger causing the highest loss, and the trig- ger initialized to be all zeros. However, if synthetic samples are sensitive small universal adversarial perturbation (UAP), then these UAPs could be synthesized instead of the actual trigger, leading to poor unstable defense performance. To validate our hypothesis, we use synthesizer hesized trigger norm a metric to evaluate stability of different base sets a smaller synthesized trigger implies that a smaller perturbation is required to flip the prediction on a batch of samples. As shown in Table 1, clean base set leads to the largest norm. Figure ((c) also shows that clean data need to be perturbed with a stronger UAP to reach the same misprediction rate as GMI-synthesized data, implying that clean data more robust to UAPs.
To lest whether improving stability to small pertuabtions could help decrease ASR, we introduce an addi- tional loss in Equation 3, named pettubpettnratiin loss: Lap(z) -CosSim( (G())) (G(z)), Sa (G(z) 8)), where CosSim stands for cosine similarity. This loss calculates the change of the model output logits when a synthesized sample G(z) is perturbed by 8. Here d is randomly sampled from a normal Gaussian distribution. From Table we can see that ASR drops when increasing weight of Ldp, which validates our hypothesis. However, Lap alone does not have much impact on ACC, How can we further improve the accuracy during unlcarning?
Ldp
Lemp
X 10
x 100
x 1000
x 10
x 100
X 1000
ACC
0.75
0.82
0.72
0.94
0.96
0.97
ASR
0.27
0.08
0.02
0.42
0.49
0.37
Table 2: Backdoor unlcarning performance with varied weights of data-perturbation (Lap) and model- perturbation (Lmp) loss.
Question 3: Does stability to model parameters affect backdoor performance? The poisoned model is usually trained to be optimal on the original training data, meaning that the gradient with respect to the model parameter on the data should be close to zero: [[Vo)) ()) N0==0,. 1I R O, as shown Figure 1(b). However, Figure 2(d) shows that, while 90% of the clean sample has [[Vo(fo(z)) v)lo= l1 S 0.001, GMI/GMI+ reconstructed samples distribute more diversely, and the gradient norm based GMI generated samples are relatively higher. We observe that ACC quickly drops when performing unlearning on these samples, which is consistent with catastrophic forgetting phenomenon (Kirkpatrick et al., 2017). Intuitively, we can maintain a high ACC by regularizing the model gradient during inversion.
To test this hypothesis, we introduce an additional loss in Equation 3, named model-perturbation loss: Imp(2) [fo ())))) Ih,, which mcasures the stability of the prediction for the synthesizod sample G(2) to small changes the parameters of the poisoned model. Table 2 shows that this loss helps improve ACC during unlearning.
3.2 Proposed Approach
Given the findings above, we propose FRED, an approach to reconstructing the training data from trained model. FRED differs from recent MI techniques in that its synthesis goal not only considers synthetic data quality and recovery of class-specific semantics, but also addresses the specific challenges non-converging prediction and small universal perturbation that hinder successful application to backdoor removal. We introduce following new loss terms critical enable the downstream task of backdoor removal:
The model-perturbation loss Tamp (2) = [[Vo 1))

---

Published in Transactions on Machine Learning Research (08/2023)
The data-perturbation loss (d) CosSimm(S (G(z)), Mo (G(z) 0]).
(Optional) The feature consistency loss Leon(z) E (a' d Dejomm g0 (G(z)) g0 (2))ll2-
po represents the leature extractor of the poisoned model 1ppii L.e, the outpul of the penultimate layer. The loss is only used when extend approach to defense setting where set of clean in-distribution samples Delean is available. It mcasures the feature distance between the synthesized sample and the available clean samples.
ALGORITHM Algorithm of FRED.
Input GGneraaor G, target model T, batch size B, clean data (optional), max iterations N, learning rate 01,02-
for each class E (1, K) do
Initialize 2: =(1) ~ N(O, I).
Initialize d: g(1) 0 01xd where d is the dimension of synthesized sample G(z))
for each ileration i E N) do
Temporary Update 2: ( =(()
- -a0 Ei VoLap(z(1) (10))()),00")))
Update 5: 5(++1) = 5(6) + B E V& (3) 8(1)).
Update 2: 2(++1) = 20) - a1 B E V (2(1), 101
7
b=
end
Zy (N)
10 end
return 21:....26
Instead of using simple Gaussian noise in Ldp; we propose bilevel-optimization algorithm 1 to find the most potent universal perturbation for B synthesized samples:
8" arg Σ Tap (1 (6),
(1)
s.t. 27 (8) arg min (prior(2s) Aa + deLmp((z) + Ag + La
(1. B}
However, it could be computationally expensive the inner optimization at any 8 requires synthesizing batch of samples. To tackle this challenge, propose online approximation algorithm 1 to update and 8 alternatively through a single optimization loop. This lype of algorithm is often used in meta learning (Shu et al., 2019; Madaan et al., 2021). The output of I provides the optimal perturbation and a batch of synthesized samples robustified against the perturbation.
Hyperparameter Tuning: X to 0 are the weights associated with each loss in Equation 4. In our experiments, we choose ê 1000 following the prior works on MI. We find A = 1000, A 10 work well across different datasets and models. The best value for A is Lask-dependant, chosen by grid search that yields the smallest value of Lirior Lel + Lmp + Ldp- Table provides sensitive analysis of x2 and X4, which are the weights of our proposed loss terms. Detailed choice of other hyperparameters are provided in Appendix C.
4
MI does not recover backdoors
4.1 Empirical Study
We perform experiments on the GTSRB dataset (Houben et al., 2013) to study whether the synthesized samples would contain backdoors. Specifically, we train poisoned models under various attacks e.g., invisible attack (Li al., 2020a), smooth (Zeng et al., 2021), trojan square (Liu al., 2018b), WaNet (Nguyen & Tran, 2020b); and apply FRED to reconstruct set of samples from this model. We generate 100 images

---

Published in Transactions on Machine Learning Research (08/2023)
for each class. To detect whether or not synthesized data contain the backdoor trigger, train binary trigger detection classifier on clean GTSRB training set, a GAN-generated set, and their poisoned correspondence. The trained trigger classifier has 100% accuracy on held-out test set. Applying this trigger classifier to our synthesized data, we get that no images are detected to contain the trigger. However, it is still possible that backdoored data are within the support of the GAN generated images but just not discovered by our synthesis technique as the underlying optimization does not directly lead the synthesized data recover the backdoored samples. To empirically verify whether the backdoored data are not found by our optimization or they not in the support of the GAN, after regular optimization shown in Algorithm 1, we continue to optimize these synthesized data to minimize the mean square error (MSE) between them and their poisoned version. However, even after the optimization, backdoor detection rate is still O. The intuition here is to encourage GAN to generate trigger paliern by maximizing the pixel-level similarity between GAN generated images and poison images. We adopted MSE as our loss function because it's commonly used for pixel-level recovery in the literature. While cosine similarity can also be used mcasure the pixcl-level similarity, it is not easy optimize MSE.
Above experiments are all based on the assumption that the auxiliary dataset Daux used for training the GAN is clean. What if some poisoned data is mixed into the GAN's training data? We further poison Daux with different ratio and test the trigger detection rate respectively. As shown in Figure 3, detection rate remains 0 when Daux is poisoned by invisible attack (Li et al., 2020a) using 1% and 2% poison rate, which is typically used for backdoor attacks. Even when the poison rate increased to an uncommonly large ratio (i.e., 50%), the detection rate remains low at 0.02. Bau et al. (2019) shows similar findings of the wholesale omission of GAN generating some objects. These objects are either complex patterns that include many pixels (c.g., large human figure), findings, even if backdoors exist in GAN's training dala, it would be hard for them to take effect unless using a substantial poison rate, which becomes unrealistic. Also, recent backdoor data detection methods (Ma et al., 2022) have achieved great efficacy. They can serve as pre-processing steps to screen out poisoned samples in GAN's training set.
under-representedd in the GAN training distribution. Based on these
1e-2
Detection Rate
2.0
x1
1.5
x2
1.0
x5
0.5
x10
0.0
x10
1%
2%
5%
10% 20% 50%
(a)
Single-Class Poison Rate of Daux (b) x10
Figure 3: (a) Trigger detection rate when increasing data poison rate of samples from a single class in auxiliary dataset Dauz- Trigger detection rate here refers to the percentage of images that were detected containing triggers among all generated images, while the poison rate refers to the percentage of poison images with the target-class images only. (b) Example of images generated from latent code h with different scales of norm.
4.2 Theoretical Justification based on Equilibrium in GAN
Here, we provide the theoretical justification of why reconstructing samples from poisoned model based on (clean) pre-trained GAN does not recover backdoored samples. We require the following two assumptions. A ssummptio 1. The generator G is L-Lipschitz in latent vector input h E IRP.
Assumption 2. For all R2, |M]] B for some B > 0, then G(h) has semantic meaning.
The Assumption 2 is justified by Figure 3 (b): We observe an apparent quality degradation of the generated images when increasing the norm of the latent vector h. When the norm is 106 larger, the generated images
8

---

Published in Transactions on Machine Learning Research (08/2023)
do not contain semantic meaning. Note that both clean and backdoor data considered to have semantic meaning.
We are interested in whether the range of the generator range(G) contains backdoored data. Since the backdoored images still have semantic meaning by definition, if range(G) contains backdoored data they will be within the high-density region where the corresponding latent vector h has || S B. By the Lipschitzness (which implies continuity) of G, it means the density of the distribution induced by the generator (G(h), I)) the backdoored data points O.' Thus, reduce the question of "whether the range of G contains backdoored data" to "whether the generator distribution has non-zero density backdoored data points."
We proceed by formulating GAN training as a two-player game between the generator and discriminator. The game terminates only when the two players reach a min-max solution where neither party has the incentive to deviate from the current state. Such a min-max solution called pure Nash equilibrium. Basod the game-theoretic framework, we show the following result, and the proof is outlined in Appendix B.
Theorem (Informal). When the generalor learns a distribution with non-negligible densily on backdoored data, the generator and discriminator connot achieve pure strategy Nash equilibrium.
This result implies that, no backdoored data point can appear in range( when the GAN trained properly where the generator and discriminator reach equilibrium. Thus, no matter how we search over G(h) for different latent vector h during the MI step, it is impossible to find h such that G(h) is backdoored image.
5 Evaluation
Our evaluation focuses on the following aspects:
Assess the effectiveness of FRED to enable data-free backdoor removal, and the benefit of FRED
with limited amount of clean samples available (Sec. 5.2, Sec. D);
Study application of FRED to adversarial fine-tuning (Jeddi et al., 2020), which aims to robustify pre-trained model against adversarial examples (Goodfellow et al., 2014) (Sec. 5.2, Sec. D);
Ablation study on several design choices of FRED, including different loss terms and the number of synthesized samples (Sec. 5.2);
Study performance of FRED with varing distribution shifts OOD datasets (Sec. C.1);
Study the effectiveness of FRED supplying other existing backdoor cleansing methods (Sec. E).
Study the effectiveness of FRED against adaptive atlacks (Sec. F).
Our evaluation setting reflects real-world scenarios, where the training data of pre-trained GANs often differs from the target model's training set in terms of distribution (e.g., pre-trained GAN is usually built with large, open data, yet the target model could be built by some stakcholder with proprictary datascts). We also considered aut-of-distribution data with various distributional shifts from the target-model training data enable comprehensive evaluation (refer to Section C.1).
5.1 Experimental Setup
Data. We evaluate datasets built for different prediction tasks, including face recognition, traffic sign classification, and general object recognition. For each task, we choose two datascts, one used for training the poisoned model and another for learning pre-trained GAN. Detailed usage of the datasets shown in Table 3.
Backdoor Attacks. We evaluate nine different. kinds of backdoor attacks in all-to-one settings (the target model will misclassify all other classes' samples patched with the trigger as the target class), including the
Note that the latent with semantic meaning (h: [|/b|| B3) an open set.

---

Published in Transactions on Machine Learning Research (08/2023)
Face Recognition
Traffic Sign
General Object
Classification
Detection
Poisoned Model
PubFig(Pinto et al., 2011) CelebA (Liu et al., 2015)
GTSRB (Houben et al., 2013)
CIFAR-10 (Krizhevsky et al.)
Pre-trained GAN
TSRD(Huang)
STL-10 (Coates et al., 2011)
Table 3: Datasets.
hidden trigger backdoor attack (Hidden) (Saha et al., 2020), input-aware backdoor (TAB) attack (Nguyen & Tran, 2020a), WaNet (Nguyen & Tran, 2020b), Lo invisible (Lo inv) (Li et al., 2020a), L2 invisible (L2 inv) (Li et al., 2020a), the frequeney invisible smooth (Smooth) attack (Zeng et al., 2021), Irojan watermark (Troj-WM) (Liu et al., 2018b), trojan square (Troj-SQ) (Liu et al., 2018b), and blend attack (Chen et al., 2017). The implementation details of the attacks are deferred to Appendix C.
Baselines. We compare FRED with five baselines, where the first four baselines differ kinds of samples contained in the base set and share the same downstream backdoor removal technique, namely I-BAU, which achieves the state-of- of-tte-rrtt backdoor removal performance given a clean base set. 1) Clean: The base set is formed by clean samples drawn from the original training data of the poisoned model. 2) Out-of- the-distribution (OOD): The base set consists of ALL of the OOD samples that are used for learning the pre-trained GAN. 3) Naive: The base set contains samples synthesized by the MI adopted in Chen et al. (2019) which directly optimizes in the pixel space. 4) GMI: The base set is formed by the synthetic samples from GMI (Zhang et al., 2020). The comparison between FRED and GMI will demonstrate the effectiveness of our designed loss terms. 5) CLP (Zheng et al., 2022): The last baseline a recent data-free backdoor removal technique that does not utilize the idea of data synthesis. Instead, it prunes the neurons directly based corresponding Lipschitz constants.
Our baselines provide comparison from different perspectives: 1) Clean is the original backdoor removal method with an extra clean in-distribution set provided, serving as the upper bound for defense performance. 2) OOD, Naive, GMI share the same assumption as our defense that the defender can leverage clean publicly available data from a related domain. This setting is practical in real-world scenarios considering the large amount of public data available online. 3) CLP is the state-of-the-arl backdoor removal defense that is not base-set-reliant compared to this type of baselines, our method offers a unique plug-and and-lla benefit.
Protocol. For all baselines except OOD, we draw/ generate 20 samples per class for PubFig and GTSRB; 40 for CIFAR-10. A detailed study of choosing the number of samples to be generated for each class is shown in Section 5.2. For the hyperparameters, we fix \ 1000, aa 1000, A 10, set λ = 10 for PubFig and GTSRB, and X2 100 for CIFAR-10. The defense performance is averaged over three runs of I-BAU.
Lo inv
La inv
Initial 0.97
Clean 0.98
OOD
Naive
CMI
FreD
CLP
Initial 0.998
Clean
OOD
Naive
GMI
FreD
CLP
ACC ASR
0.88
0.88
0.93
0.95
0.94
0.98
0.9330
0.95
0.94
0.94
0.94
1.0
0.03
0.02
0.08
0.09
0.02
0.02
0.06
0.832
0.06
0.48
0.01
0.01
Smooth
Wanet
Initial 0.97
Clean 0.98
OOD
Naive
GMT
FreD
CLP
Initial 0.98
Clean 0.94
OOD
Naive
GMT
FreD
CLP
ACC ASR
0.83
0.82
0.96
0.97
0.18
0.26
0.17
0.81
0.94
0.01
0.998
0.1
0.40
0.05
0.03
0.02
0.02
0.99
0.05
0.34
0.97
0.16
0.05
0.99
IAB
Troj-SQ
Initial 0.94
Clean 0.02
OOD
Naive
GMI
FreD
CLP
Initial
Clean 0.96
OOD
Naive
GMI
FreD 0.94
CLP
ACC ASR
0.81
0.45
0.89
0.91
0.92
0.98
0.40
0.45
0.86
0.81
1.0
0.10
0.11
0.11
0.10
0.10
1.0
0.01
0.06
0.16
0.10
0.06
0.23
Troj-WAN
Blend
Initial 0.98
Clean 0.96
OOD
Naive
GMI
FreD
CLP
Initial
Clean 0.96
OOD
Naive
GMI
FreD
CLP
ACC ASR
0.47
0.62
0.84
0.87
0.87
0.98
0.47
0.68
0.82
0.92
0.75
1.0
0.01
0.30
0.09
0.30
0.09
0.12
1.0
0.08
0.51
0.55
0.48
0.22
0.85
Table 4: Results of FRED boosted backdoor unlearning on GTSRB.
10

---

Published in Transactions on Machine Learning Research (08/2023)
5.2 Results
Data-Free Backdoor Defense. Table 4 shows that FRED outperforms naive MI, OOD, GMI and CLP against various backdoor attacks on GTSRB. Results for the other datasets can be found in Appendix D and FRED remains the best. Figure 7 visualizes the samples synthesized by Naive, GMI, and FRED. GMI, and FRED can generate samples with better visual quality, whereas Naive generates merely noise-like samples. This visualization explains the significant. defense performance improvement achieved by GMT and FRED upon Naive. The performance of FRED mostly on par with Clean. Interestingly, FRED achieves a higher ACC and comparable ASR than baseline utilizing clean data when defending against the LAB attack performed on the CIFAR-10 dataset. This may be because the model is overfitted to the clean training samples, and samples generated by FRED reduce the degree of overfitting by providing more abundant features. Note that CLP fails in defending against Smooth, Wanet, and Blend attack: Either the ACC drops to close to zero (Smooth and Wanet), or ASR remains high (Wanet and Blend), or both occurs (Wanet). As the above triggers have large norm and hence induce large changes the model input, CLP's assumption that backdoor-related neurons in the poisoned model have large Lipschitz constant does not hold. To better interpret the data synthesis process of FRED, we show series of samples generated at different iterations in Figure 6. We observe that the appearance of the generated images varies significantly over the first ten optimization iterations and stabilizes afterwards.
Data-Limited Backdoor Defense. Here, we evaluate the benefits of FRED when there exists a tiny amount of clean samples. Particularly, we consider a stress test with I sample. With a single sample, even the stale-of-the-art data-reliant backdoor removal technique works poorly as shown in the CIFAR-10 and PubFig83 results in Table 5). To evaluate FRED, we use FRED with the proposed feature consistency loss Leon to generate 20 additional samples for each class, and the final result (FRED-Booster) is obtained by using the combination of both clean sample and 20 generated samples for each class. Table 5 shows that FRED can significantly boast the defense performance compared to solely using the available clean sample(s). Moreover, using 20 samples from FRED plus one clean sample gives better defense performance than 20 clean samples. As a final note, compared to FRED, CLP, the data-free backdoor removal bascline based on model pruning, cannot. be benefited from additional clean samples.
Clcan
Clean(1)
FreD-Booster
GTSRB
ACC
0.98
0.93
0.98
Smooth
ASR
0.01
CIFAR-10
ACC
0.82
0.52
0.85
IAB
ASR
0.03
0.18
0.01
PubFig
ACC
0.86
0.44
0.86
Troj-WM
ASR
0.03
0.35
Table 5: Results of backdoor unlearning performance with small amount of clean data and generated samples.
Initial
GMI
Lmp
Ldp
Lmp + Ldp
ACC(%)
98.8
86.52
92.70
89.32
94.49
GTSRB
ASR(%)
100.0
10.06
8.63
8.25
5.90
ACC(%)
92.21
70.52
72.32
70.61
83.53
PubFig
ASR(%)
100.0
26.06
3.25
1.63
2.88
Table 6: Ablation study of proposed potel-pertur loss Lmp and data-perturbation loss Ldp-
A Ablation of Loss Terms. We proposed two loss terms 1) modellpertuubattion loss Lmp and 2) data- perturbation loss Lap to improve the utility the synthesized samples in the data-free backdoor defense setting. Table presents an ablation study of the two losses on poisoned model trained on GTSRB under the trojan square attack as well as a model trained on Celeba under the trojan watermark attack. We observe that lap improves the ASR more than hmp while hmp is a more critical driver of maintaining the ACC
11

---

Published in Transactions on Machine Learning Research (08/2023)
compared to ldp. This observation aligns with our design objectives. Recall that lap is designed enable effective synthesis of backdoor trigger and thus directly related to the reduction of ASR. On the other hand, hmp encourages the stability of prediction to small parameter changes, which in turn mitigates catestrophic forgetting during unlearning; hence, is directly related to maintaining the clean accuracy.
Ablation of Base Set Sizes. We study the impact of the number the synthesized samples contained in the base set. We choose the number of samples for each class be [1, 5, 10, 15, 20, 25, 30) and evaluate the defense performance by averaging over 3 runs of the defense. To better interpret the performance of FRED, also compare with Clean, Naive and GMI using the same amount of samples. Note that this experiment excludes the OOD baseline: because the label space of the OOD samples and that of the poisoned model may not overlap, the labeled OOD samples are insufficient or even none for some classes. Figure shows that the defense performance keeps increasing as the number of samples increases and converges to optimum when the number of samples for each class above FRED, GMI, and Clean can maintain a high ACC when larger number of generated samples are used, but Naive suffers a significant ACC drop. We also observe during the experiments that performance of Naive has large variance when evaluated could induce from the inconsistent quality among generated samples, instability of the samples against input perturbation, leading to synthesizing inaccurate triggers. On the other hand, the variance of FRED is similar to the variance of Clean, indicating good generalizability of FRED-enabled defenses. Another interesting finding is that when performing unlearning with a small amount of samples (i.e., 1 or 5 per class), FRED even achieves higher ACC than Clean.
use the poisoned model to generate pseudo-labels for the OOD samples but
base set with different size. The variance
100
95
80
FreD
ACC
ASR
90
Ozsn Nabe
20
GR
ReO
15
5
(a)
# of seeds per class
(b)
#
of seeds per class
Figure 4: Ablation study of the number of samples used on backdoor defenses the GTSRB with Lo inv attack.
6 Conclusion
In this paper, we investigate the connection between model inversion and backdoor defense, and present FRED to generate synthetic samples that can be used as a substitute for clean in-distribution data to sup- port backdoor removal. FRED can also be used to boost defense performance when only limited clean in-distribution data are available. This work sets foundation towards developing highly effective in- distribution-data-free backdoor defenses. In particular, one potentially supply synthetic data to other future defenses to enable their data-free mode of usage or improve their performance in the limited data setting.
7 Acknowledgement
RJ and the ReDS lab acknowledge support through grants from the Amazon- Virgini Tech Initiative for Efficient and Robust Machine Learning, the National Science Foundation under Grant No. ITS-2312794, NSF IIS-2313130, NSF OAC-2239622, and the Commonwealth Cyber Initiative. JTW is supported by Princeton's 's Gordon Y. S. Wu Fellowship.
12

---

Published in Transactions on Machine Learning Research (08/2023)
References
David Alvarez-Melis and Nicolo Fusi. Geometric dataset distances via optimal transport. NewrIPS, 33:
21428-21439, 2020. 19
Shengwei An, Guanhong Tao, Qiuling Xu, Yingqi Liu, Guangyu Shen, Yuan Yao, Jingwei Xu, and Xiangyu Zhang. Mirror: Model inversion for deep learning network with high fidelity. In Proceedings of the Network and Distrihuled Systems Securily Symposium (NDSS 2022). 2022. 3, 4,
Eugene Bagdasaryan and Vitaly Shmatikov. Blind backdoors in deep learning models. In USENIX Security. pp. 1505-1521, 2021. 21, 22
David Bau, Jun- Yan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, and Antonio Tor- ralba. Seeing what a gan cannot generate. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4502-4511, 2019.
Eitan Borgnia, Valeriia Cherepanova, Liam Fowl, Amin Ghiasi, Jonas Geiping, Micah Goldblum, Tom Goldstein, and Arjun Gupta. Strong data augmentation sanitizes poisoning and backdoor attacks without an accuracy tradeoff, 2020. 16
Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Mol- loy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. CEUR Workshop Proceedings, 2018. 16
Huili Chen, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar. Deepinspect: A black-box trojan detection and mitigation framework for deep neural networks. In IJCAI, volume 2, pp. 8, 2019. 2, 3, 4, 10, 16
Si Chen, Mostafa Kahla, Ruoxi Jia, and Guo-Jun Qi. Knowledgg-enniched distributional model inversion attacks. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 16178-16187, 2021. 3, 4, 16
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning, 2017. 10, 20
Edward Chou, Florian Tramer, and Giancarlo Pellegrino. Sentinet: Detecting localized universal attacks against deep learning systems. In 2020 IEEE Security and Privacy Workshops (SPW), pp. 48-54. IEEE, 2020. 16
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, 2011. 10
Bao Gia Doan, Ehsan Abbasnejad, and Damith C Ranasinghe. Februus: Input purification defense against trojan attacks on deep neural network systems. In Annual Computer Security Applications Conference, pp. 897 912, 2020. 16
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications securily, pp. 1122,1333, 2015. 4, 16
Matthew Fredrikson, Eric Lantz, Somesh Jha, Simon Lin, David Page, and Thomas Ristenpart. Privacy in pharmacogenetics: An (Fad-to-End} case study of personalized warfarin dosing. In 23rd USENIX Security Symposium (USENIX Security 14), pp. 17 32, 2014. 4
Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C. RRaasinnhhe, and Surya Nepal. Strip: A defence against trojan attacks deep neural networks. In ACM ACSAC, pp. 113 125, 2019. ISBN 16
7781466376280. doi:
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint or Xio: 6572, 2014. 9
13

---

Published in Transactions on Machine Learning Research (08/2023)
Wenbo Guo, Lun Wang, Xinyu Xing, Min Du, and Dawn Song. Tabor: A highly accurate approach to inspecting and restoring trojan backdoors in ai systems. arxiv preprint 1 1901.998.00 01 01763, 2019. 2, 4, 16, 22
Sebastian Houben, Johannes Stallkamp, Jan Salmen, Mare Schlipsing, and Christian Igel. Detection of traffic signs real-world images: The German Traffic Sign Detection Benchmark. In International Joint Conference on Newral Nelworks, number 1288, 2013. 5, 7, 10
LinLin Huang. Chinese traffic sign database. recogmition.hte html. 10
URL http://www.nl/ ia. aa..a.aaa......................a wcc.c/aaaatraaffccataaa
Ahmadreza Jeddi, Mohammad Javad Shafiee, and Alexander Wong. A simple fine-tuning is all you need: Towards robust deep learning via adversarial fino-tuning. arXiv preprint arXiv:2012. 13628, 2020.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVE conference on compuler vision and pollern recognilion, pp. 4401 4410, 2019. 5
Tero Karras, Miia Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33:12104-12114 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaune Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barasskaa-Briikka,, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521 3526, 2017. 6
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1885 1894, International Convention Centre, Sydney, Australia, 06-11 11 Aug 2017. PMLR. URL http: http://proceedings mlr press/v70//ko177a http:///gg......rrrrr/////////aa html. 16
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). URL http:// /ttp:://wwww... cs. toronto... httu//krii/cifeee html. 10
Shaofeng Li, Minhui Xue, Benjamin Zhao, Haojin Zhu, and Xinpeng Zhang. Invisible backdoor attacks deep neural networks via steganography and regularization. IEEE Transactions Dependable and Secure Computing, 2020a. 2, 5, 7, 8, 10, 20
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distilla- tion: Erasing backdoor triggers from deep neural networks. In International Conference on Learning Representations, 2020b. 16
Kang Liu, Brendan Davin,Gavitt, and Siddharth Garg, Fine-pruning: Defending against backdooring atlacks on deep neural networks. In International Symposium on Research in Attacks, Intrusions, and Defenses, pp. 273-294. Springer, 2018a. 2
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Loo, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In 25nd Annual Network and Distributed System Securily Symposium, NDSS. The Internet Society, 2018b. 7, 10, 20
Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang. Abs: Scanning neural networks for back-doors by artificial brain stimulation. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pp. 1265-1282, 2019. 16
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEFE international conference on compuler vision, pp. 3730-3738, 3373,, 2015. 10
14

---

Published in Transactions on Machine Learning Research (08/2023)
Wanlun Ma, Derui Wang, Ruoxi Sun, Minhui Xue, Sheng Wen, and Yang Xiang. The" beatrix reauritrresurrrcctio::: Robust backdoor detection via gram matrices. NDSS, 2022. 8
Divyam Madaan, Jinwoo Shin, and Sung Ju Hwang. Learning to generate noise for multi-attack robustness. In International Conference on Machine Learning, pp.
PMLR, 2021. 7
Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. NewrIPS, 33:3454 3464, 2020a.
10, 20
Tuan Anh Nguyen and Anh Than Tran. Wanct-imperceptible warping-basch backdoor attack. In ICLR, 2020b. 7, 10, 20
Nicolas Pinto, Zak Stone, Todd Zickler, and David Cox. Scaling up biologically-ispiree computer vision: A case study in unconstrained face recognition on facebook. In CVPR 2011 WORKSHOPS, pp. 35-42, IEEE, 2011. 10
Han Qiu, Yi Zeng, Shangwei Guo, Tianwei Zhang, Meikang Qiu, and Bhavani Thuraisingham. Deepsweep: An evaluation framework for mitigating dnn backdoor attacks using data augmentation. In Proceedings of the 2021 ACM Asia Conference on Computer and Communications Securily, pp. 363-377, 2021. 16
Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 11957-11965, 2020. 2, 10, 20
Guangyu Shen, Yingqi Liu, Guanhong Tao, Shengwei An, Qiuling Xu, Siyuan Cheng, Shiqing Ma, and Xiangyu Zhang. Backdoor scanning for deep neural networks through k-arm optimization. In International Conference on Machine Learning, pp. 9525-9536. PMLR, 2021. 16
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-weight-ned Learning explicit mapping for sample weighting. Advances in neural information processing systems, 32, 2019. 7
Hossein Souri, Liam Fowl, Rama Chellappa, Micah Goldblum, and Tom Goldstein. Sleeper agent: Scalable hidden trigger backdoors for neural networks trained from scratch. Advances in Newral Information Processing Systems, 11-19165--9178, 2022. 21, 22
Lukas Struppek, Dominik Hintersdorf, Antonio De Almeida Correia, Antonia Adler, and Kristian Ker- sting. Plug & play attacks: Towards robust and flexible model inversion attacks. arXiv preprint 2rXiv:2201.12199 2022. 3, 4, 5, 19
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor atlacks. In NeurIPS, pp. 8000 8010, 2018 16
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Tdentifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 707-723. IEEE, 2019. 2, 4, 16, 22
Kuan-Chich Wang, Yan Fu, Ke Li, Ashish Khisti, Richard Zemel, and Alireza Makhzani. Variational model inversion attacks. Advances in Neural Information Processing Systems, 34, 2021. 3
Yi Zeng, Won Park, Z Morley Mao, and Ruoxi Jia. Rethinking the backdoor attacks' triggers: A frequency perspective. In ICCV, 2021. 2, 7, 10, 16, 20
Yi Zeng, Si Chen, Won Park, Zhuoqing Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. In International Conference on Learning Representations, 2022. 4, 5, 16
Yuheng Zhang, Ruoxi Jia, Hengzhi Pei, Wenxiao Wang, Bo Li, and Dawn Song. The secret revealer: Gener- alive model-inversion altacks against deep neural networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 253 261, 2020. 2, 3, 4, 5, 10, 16
Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Data-free backdoor removal based on channel lipschitz- ness. arXiv preprint or Xrxi:22008.00 03111, 2022. 2, 10, 16
15

---

Published in Transactions on Machine Learning Research (08/2023)
A Related Works
Backdoor Defenses. Backdoor defenses normally can be performed on two levels: data-level and model- level. For data-level detection or cleaning, the defender aims to identify (Gao et al., 2019; Chen et al., 2018; Tran et al., 2018; Koh & Liang, 2017; Chou et al., 2020; Zeng et al., 2021) or purify (Doan et al., 2020) the poison input in the training set, thus requiring the access to training data. Model-level detection (Liu et al., 2019; Shen et al., 2021) or cleaning, instead, aims to detect if a pre-trained model poisoned or mitigate vulnerabilities of the models. In this paper, we focus on model-level cleaning. Most of the prior works this line (Wang et al, 2019; Chen et al., 2019; Guo et al., 2019; Zeng et al., 2022; Li et al., 2020b; Borgnia ct al., 2020; Qiu et al., 2021) require a small set of clean data to synthesize triggers and further perform unlearning, Among them, I-BAU (Zeng et al., 2022) has achieved the state-of-the-ar defense performance against a wide range of existing attacks. However, the performance of I-BAU degrades as the number of clean samples reduces. We aim to enable I-BAU to function effectively without any clean data. A recent work (Zheng et al., 2022) proposes a method to perform backdoor removal without using clean data. They identify model channels with high Lipschitz constants, which are directly calculated from the weight matrices, backdoor related channels; and do simple pruning to repair the model. However, their method only applies to backdoor scenarios and cannol benefit from clean samples if available. Our method, by contrast, also applies to evasion attacks and is able to leverage available clean samples to further boost defense performance. Above all, the performance of our method is more favorable.
Model Inversion. The goal of model inversion (MI) is similar to ours. But from an attack perspective, MI aims to divulge scnsitive attributes in the training data, and to achieve this goal, the generated data should have good visual quality. Fredrikson et al. (Fredrikson al., 2015) follows the maximum likelihood principle and performs MI by searching over the image space for sample with highest likelihood under the given target model. DeepInspect (Chen et al., 2019) employs this simple MI to generate surrogate training set for backdoor unlearning and achieves good results on MNIST and GTSRB. However, we find that samples generated by this naive MI approach have bad visual quality and usually fail in downstream defenses on high-dimensional datasets (e.g., PubFig and C1FAR-10). In this paper, we build upon the idea of recent MI works (Zhang et al., 2020; Chen et al., 2021) that scarch for a synthetic sample in the latent space of a pre-trained GAN instead of the image space. Even when the GAN is not trained on the in-distribution data, this idea, can greatly help improve the visual quality of synthesized samples. The key innovations that set our work apart from the MI attacks is that we go beyond the traditional thigh-likelihood assumption made in all existing MI works about clean data and further formalize other plausible assumptions, especially those related to data- and model-stability. We show that enforcing the synthetic data to satisfy these assumptions can significantly improve their utility for defenses.
B Why is backdoored data not on GAN's range?
Throughout the section, we use d for the dimension of samples, and p for the dimension of the latent vector. We denote discriminator D : R2d - [0, 1), generator G R2 - Rdd We use Prea to denote the real distribution the GAN aims to learn. The generator G defines a distribution Pa as follows: generate latent vector h ~ (O, from p-dimensional spherical standard Gaussian distribution, and then apply G on h and generate sample generators G = [G]. Tdeally, D is the class of all 1-Lipschitz functions. For distribution P, we use P(xx) to denote the density of P on x, and P(S) denotes the Lebesgue integration f [[x E S)dP(x). We use Ep[D] as an abbreviation for [Dx)] We use supp(-) to denote the support of distribution. We use da(s)) to denote 1- Wasserstein distance with l2- fo-norm, i.., the Earth Mover distance.
Notations.
G(h). We denote the class of discriminators as D }{D and the class of
The two required assumptions are demonstrated in previous section 4.2. To formally state our theorem, we formulate the training of GAN a game between generator and discriminator.
Definition 4 (Payoff). For class of generators G = {G} and a class of discriminators D = {D}, we define the payoff FDD.G) of the game between generator and discriminator as
F(D,G)
E P
[D(x)]
E I~PO
[D(x)]
(5)
16

---

Published in Transactions on Machine Learning Research (08/2023)
The generator and discriminator aims at reaching min-max solution, i.., the pure Nash equilibrium, where neither party has the incentive to deviate from the current state..
Definition 5 (pure equilibrium). A pair of strategy (D", G*) is a pure equilibrium if for some value V,,
e D, FDD, G*) SV
AG E F,F(D" G VV
However, such an equilibrium may not be achievable for a pure strategy setting. We introduce a natural relaxation for quantifying the extent of equilibrium between a pair of Definition 6 (s-approximate pure equilibrium). A pair of strategy (D", G*) is an e-approzimate pure equi- librium if for some value V,
VD D. AG E F, G)) VV-
We are now ready to state our main results.
General Idea: game-thcoretic interpretation of GAN. From a game theory perspective, the generator and discriminator in GAN are engaged in a two-player zero-sum game. In game theory, a Nash Equilibrium is a state in which neither player can improve their outcome by changing their strategy, assuming the other player keeps their strategy fixed. In the context of GANs, Nash Equilibrium is achieved when the Generator produces data that is indistinguishable from real data, and the Discriminator is no better than random guessing at telling the difference between real and generated data. In Theorem (or Theorem 7), we show that when the generator's learned distribution contains a significant probability mass on backdoorod images, which is out-of-distrrbution (OOD) data, the equilibrium cannot be established between the generalor and discriminator, which forces the generator to reduce the probability mass on the backdoored images.
Theorem 7 (Formal version of Theorem 3). Given any two distributions P1.P2 s.t. for the set SOOD = {ar E supp(P22) minggssupp(P, (Uspp((P || - y/ 2 1}, we have P2(SooD) 2 21 for some o' E [0, 1). Let D = arg max Ep [D] Ep, [D] and D E D If induce a minture distribution Pg = (1 Tqp for some q E (0, 1), then there erists no E D s.t. (D,G) is E-appronimate pure equilibrium for any < [q(p,[D")
Moreover, when Preat = Pl, we have ε < (1 d)) Further more, given Assumption and =exp(=)B") T(/2++1)
2, we can lower bound q if rannee(G) contains backdoored data, which leads to < (1-0') LV2 where I is the Gamma function.
Interpretation. To interpret the above theorem statement, one can regard Pi as clean distribution (not necessarily Preat), and P2 as a distribution that contains backdoor data on its support. Since backdoored images are separated from clean images (i.e., out-of-distribution (OOD) data), we can assume that all backdoored images are within the set SOoD {i supp(P2) : (ainyyssp(PP)..ssnnp The above theorem thus states that no equilibrium could be achieved if Pe has non-negligible density SOOD- The theorem essentially reveals the tension between generator's goal to trick the discriminator and its propensity to generate backdoored (OOD) data. In simple terms, think of P1 as a distribution representing normal, clean images, and P2 representing malicious, backdoored images. The generator mixes those distributions to create Pe, which it uses to try to fool the discriminator. Now, if the Po mixture has a significant probability sampling backdoored (OOD) images from Pa, il is much easier for the discriminator spot the differences between the generated and real data. Hence, there exists a better move for the generator to increase its "payoff" by sampling less from P2, hence equilibrium becomes unattainable under such Pa (captured by the upper bound of e). In essence, the generator is in quandary: including backdoored data in the mixture makes it easier for the discriminator to win, which is contrary to what the generator wants.
||x 1|x=-ll yll 2 1}.
Remark. It also possible that P1 also supports on {a min supp((aaa)))) I| vl 2 1). but this leads to vacuous results.
17

---

Published in Transactions on Machine Learning Research (08/2023)
B.1 Proof of The Formal Theorem
Proof Overview. In Lemma 8, we derive the upper bound of the degree of equilibriun when Pa P) + gP2 in terms of the best discriminator in distinguishing between Preal and P1, ie.. D DED
arg nen Ep..al [D] - Ep, [D]. Then in Lemma 9 and 10, we further bound different terms in the upper bound of E derived Lemma under Assumption 1 and 2 stated in the maintext.
Lemma 8. Given any two distributions P2, let D arg naX Ep [Dp [D]. For any D E D", if G induce distribution Pg = (1 P) qP2, then there crisls D E D s.l. (D,G) is S-upprorimate pure equilibrium for any < (Fp, [7)] Ep,(D))
Proof. We define an alternative generator G s.t. Pg. P1. Given any discriminator D, the payoff gain of G by switching strategy to G is
- F
(1 (1- g) [D] Ep, [D]) (Bpr, [D] [p](D] (Bpp,n.) [D] Ep, (D])
(6)
= -Bp
Bp, }
Given any discriminator D, the payoff gain of D by switching strategy D is
-FF,,
= Ep [D'] (1 - 9) - pEp,
(Rp [D] - - p1 [D] gBp, [D]
Eca [D"] Ep, [D") + (Ep, - (D"))
(Ep [D] - Ep, [D] qEE,, [D] Ep, [D])
du(Proal, P1) (Bp, [D'] (p'"))
- -IEp,.. [D] - Ep, D]) - ((Ep, -Ep
By Definition 6, (D,G) cannot be e-approximate equilibrium for
max (F(D,G) F(D,
F(D,G))
since otherwise at least one of D and G will gain more than by changing its strategy to D or G". Therefore, we are interested in lower bounding minp max (F(D, G) - F(D,G*) F(D"",,,,,,,F F(D, Note that the minimum can only be achieved when F(D,G) = - whore we have
LHS qEp, [D] Ep,[D]
(Preal P1) + (Ep, [D) Ep, (D*)) (E D Ep, [D]) ((pp, [D] Er, (D))
(9)
RHS
and we have
2LHS du Pı) + (EEp, (D*] Ep, (D') -(E (EP...! Ep, [D])
(10)
qBp, D Ep, (D')
where the last inequality is due to
Eprcal [D] - [D] < sup Ep [D] Ep, [D]
DED
(11)
= du (Preall Pi)
Therefore,
min max
-F(D,))
(12)
> 2 (Ep, [D'] - (D((D'))
18

---

Published in Transactions on Machine Learning Research (08/2023)
Remark. This result may be of independent interest.
Lemma 9. Consider the set SOOD = fI E supp(P2). pippyeswp(PP)) || - 2 21.. Pf 1-d' o' for some q' E 1], then we have Ep, [D*] g
Proof. The value of D on supp(P2) (supp(Preat) U supp(Pi)) does not affect Er [D] - -Ep, [D], therefore we only need to ensure that D satisfies the Lipschitz assumption. It is easy to see that D (2) can be 0 for all x E SOOD- Since P2(SOOD) 2 1 - g', we know that Ep, [D*] s
Therefore, we know that (G, D) is s-approximate pure equilibrium only for 2 $ (Ep, [D*] -g'). Moreover, when Pre Preal P1, it is impossible to distinguish between Preal and Pi and thus D contains function D that output (xx) = for all E supp(Prea)). Thus 2 _ (1 z²).
Now we lower bound based on Assumption 1 and 2.
exp(-)B²)
Lemma 10. q
LV2
T(p/211)
Proof. Suppose for some h E IP we have G(h) E SOOD, then for all ||/1 - All we have [|G(h") (Gh))) < L ||'' - hll I 1. Therefore, G(h') E supp(P2). Therefore, within the ball centered at with radius 1/L will all have G(N') supp(P2), and thus
TP/2
2
(2 2/2 exp( exp(-B2/24 T(p/2 1) (1/L)
(13)
P
1
T(p/2+1
Plugging this lower bound back to the original bound for leads to the final result in Theorem 7.
C Experimental Settings
C.1 Choice of OOD
Clean in-distribution data may not always accessible in real-world applications, however, given the access of the target model, one may make inference about the type of data the model trained on, c.g,, whether it is a face recognition model or digit classification model, etc. This offers us the chance to make use of public available data (OOD) of the same type. FRED does not assume the overlap of the label space between OOD and the private data, as the OOD is only accessed during the GAN training stage.
Performance of FRED given different OOD is shown in Table 7, and the distance between OOD and the private distribution is calculated in Optimal Transport Dataset Distances (OTDD) (Alvarez-Melis & Fusi, 2020), which gives a meassurement of dataset distance even if the label sets are completely disjoint. As we can see, by using the in-distribution data, which in this case CIFAR-10, FRED achieves ASR=0.01 while ACC 0.87. Even when using OOD data which has a much larger distance, FRED can still achieve a comparable ASR and a even better ACC. The only exception is using Caltoch-256 as OOD, the ASR is relatively higher compared with other settings. One possible reason is that Caltech-256 only has 30,609 samples, which is smallest dataset among the four datasets. GAN may not be well-trained on this dataset.
Given the fact that State-of-tho-art MI provides could provide more advanced techniques, which relax the dependency between the target model and OOD (Struppek et al., 2022), FRED can also be benefit from it and leads to a even better performance.
19

---

Published in Transactions on Machine Learning Research (08/2023)
C-
STL-10
Tiny-ImageNet CIFA R-10
Caltech-256
CIFAR-10
CIFAR R-1A
CIFAR-10
# Classes
10
200
OTDD
324.3
3486.48
4068 28
3844.61
ACC*
0.96
0.33
0.74
0.57
ACC
0.87
0.85
0.89
0.80
ASR
0.01
0.05
0.02
0.24
Table 7: Results of FRED when using different OOD datasets. ACC* gives corresponding transfer accuracy respectively, and OTDD gives the distribution shift. The target model is trained CIFAR-10 under TAB attack.
C.2 Backdoor Attacks Implementation Details
We evaluate nine different kinds of backdoor attacks in all-to-onc settings (the target model will misclassify other classes' samples patched with the trigger as the target class); including the hidden trigger backdoor attack (Hidden) (Saha et al., 2020), input-aware backdoor (LAB) attack (Nguyen & Tran, 2020a), WaNet (Nguyen & Tran, 2020b), Lo invisible (Lo inv) (Li al., 2020a), L2 invisible (L2 inv) (Li al., 2020a), the froquency invisible smooth (Smooth) attack (Zeng et al., 2021), trojan watermark (Troj-WMr WM)) (Liu et al., 2018b), trojan square (Troj-SQ) (Liu et al., 2018b), and blend attack (Chen et al., 2017).
On CIFAR, we test all the nine attacks listed above. Note that initially, Hidden can only work in one-to- one attack settings where the goal is to fool one class with the trigger, thereby resulting in low ASR in all-to-one settings. To address this issue, we manually increase the norm bound to 50/255 with one round of fine-tuning of pre-trained clean model achieve an acceptable ASR. However, the ASR of Hidden on GTSRB is still less than 10%, hence, we exclude it from evaluation on GTSRB. On PubFig, we adopt Trojan watermark (Troj-WM), Trojan square (Troj-SQ), and blend attack. The implementations of each attack follow the original works which propose them. The adopted trigger and the target label on each dataset is visualized in Fig. 5.
Hidden: B
IAB:
Jop
WaNet:
TOL
Lo inv:
La inw:
f
Smooth: T6L
Troj-WM: TOp
Troj-SQ: TOL
Blend:
To
20
20
20
(1)CIFAR-10
(2) GTSRB
(3) PubFig
Figure 5: Datasets and examples of backdoor attacks incorporated. We consider three different datasets in this work: (1) CIFAR-10, (2) GTSRB, and (3) PubFig. Nine different backdoor attack triggers are included in the experimental part as listed. Above, also show the target label used during the evaluated attacks (e.g., Hidden targeting at label 8 of the CIFAR-10 dataset).
C.3 FreD Implementation Details
In our experiments, we choose the batch size B Algorithm be 40, max iterations N 4500, learning rate 02 le We choose lcarning rate an parameters in Equation 4, we choose X₁ 1000 following the prior works on MI. We find As 1000, A 10 work well across different datasets and models. The best value for Xg is task-dependant, and chose using grid search that yields the smallest value of Leprior + Lel + Lmmp + Ldp-
for CIFAR-10 and 01 2 for the rest. For
In our experiments, we use NVIDIA GeForce RTX 2080 Ti. The synthesize base set generation takes approximately hours for private dataset CIFAR-10 and GTSRB, and hours for PubFig, computational time of the backdoor removal depends on the downstream defense method usod. For I-BAU, takes 6.82 on average on CIFAR-10, 7.84 on GTSRB and 13.2s on PubFig for single iteration, and can effectively mitigate backdoor trigger within 100 iterations.
20

---

Published in Transactions on Machine Learning Research (08/2023)
D
More FreD Boosted Defense Results
[O
16
GAN Output
Ground Truth
Figure 6: Examples of model-specific synthesize by FRED at first 10 iterations for Identity 12 in PubFig dataset. Rightmost in the lower row is the real image of this identity from PubFig.
FreD
GMI
Naive
(1) CIFAR-10. Identity: Dog.
(2) GTSRB. Identity: Keep right sign.
(3) PubFig. Identity: Adam Sandler.
Figure 7: Examples of images obtained by FRED and naive MI. Each subplot represents randomly generated
samples for the same class. The upper row shows images generated by FRED and the lower row shows images generated by naive MI.
More results of FRED boosted backdoor defense are given in Table 8,10, which FRED yields the best performance besides the clean baseline. One interesting finding is that FRED achieves a lower ASR and comparable ACC than baseline utilizing clean data when defending against the L2 inv attack performed on the CIFAR-10 dataset. This may because the model is overfitted to the clean training samples, and samples generated by FRED reduce the degree of overfitting with more abundant features.
To provide better understanding of how FRED works, give the images generated by FRED in the first 10 iterations for Identity 12 in PubFig dataset. As shown in Figure 6, appearance of the generated fake person gradually changed to be closer to the real identity.
We also conduct additional experiments to provide results on recent backdoor attacks, namely Blind(Bagdasaryan & Shmatikov, 2021) and SlceperAgent(Suuri et al., 2022), to enhances the comprehen- siveness of our evaluation. Specifically,, we utilized the open-source implementation 2 of these two attacks. It's worth noting that neither of these attacks was originally evaluated on the GTSRB dataset, and their performance was found to be poor. The ASR of SleeperAgent remained consistently low (below 10%), while Blind could not effectively balance ASR and benign ACC 3 where in either case, the ASR very low the ACC was very low. Hence, we excluded the evaluation of these two attacks on the GTSRB dataset.
Clean
Naise
Clean
Naive
Clean
Nalve
0.88
0080
0.07
0.51
0.009
0.35
0.30
0.35
0.57
0.67
1138
0.33
0.36
0.51
0.33
ASR
0.57
0.15
0.07
0.09
0.08
0.06
0.06
0.30
0.07
0.12
0.06
0.07
0.01
0.03
0.56
0.18
0.02
0.54
0.20
0.18
0.56
Wanet
LAB
Hidden
Initil
Chosn
0OD
Nuiv
GMI
FraD
CLP
Initad
Chesn
OOD
Naise
GAU
FraD
CLP
Inttal
Clean
0OD
Kaive
GMI
FraD
CLP
ACC
0.50
0001
088
0.21
0.81
682
6.11
0.31
0S81
0.85
0,70
0.76
0.80
0.11
0.80
0.50
0.88
0.35
ASR
0.01
0.82
0.82
0,03
0,06
0,08
0.36
Trol-SQ
Initaa
Chosn
0OD
Nuive
CMI
Initad Chesn
OOD
Naisu
Initial
Clean
OOD
Naive
GMI
ACC ASR
0.54
0.81
0.51
0.14
0.71
0.71
0.71
0.36
0,90
0.50
0.21
0.74
0.75
0.06
0.54
0.90
0.85
0.63
0.76
0.79
0.46
1.0
0.07
0.57
0.28
0.06
0.05
1.0
6.04
0.11
0.29
0.02
0.08
0.58
0.59
0.05
0.71
0.05
0.00
0.06
0.28
Table 8: Results of FRED boosted backdoor unlearning on CIFAR-10.
21

---

Published in Transactions on Machine Learning Research (08/2023)
Sleeper Agent
Blind
Inicial
Clean
OOD
Naive
GMI
FreD
CLP
Initial
Clean
OOD
Naive
GMI
FreD
CLP
ACC ASR
0.31
0.85
0.79
0.18
0.35
0.44
0.91
0.02
0.31
0.10
0.10
0.02
0.08
0.78
0.09
0.23
0.25
0.10
0.07
0.20
Table 9: Results of FRED boosted backdoor unlearning on CIFAR-10 on recent backdoor attacks SleeperA- gSnt(Souri et al., 2022) and Blind (Bagdasaryan & Shmatikov, 2021).
Tro: WM
Toof-SQ
Blend
Instial Clusn
DOD
Nuive
CNT
FroD
CLP
Initid
Closn
DOD
Nuive
ChU
FroD
CLP
Instal
Clou
OOD
Nnive
GMI
FroD
CLP
ACC
0.86
0.06
0.13
0.83
0.84
0.15
9.74
0.88
0.06
0.32
0.83
0.00
Table 10: Results of FRED boosted backdoor unlearning on PubFig.
E FreD Combined with Other Backdoor Defenses
While FRED shows satisfying results when combined with I-BAU, which is one the most popular backdoor defense framework, this paper we also show the application of FRED combined with other backdoor defense techniques. Note that we focus on model-level cleansing in this paper. Specifically, we supply Neural Cleanse (Wang et al., 2019) and Tabor (Guo et al., 2019) with FRED synthesized data. As shown in Table 11 and 12, FRED outperforms OOD, Naive, GMI; and even achieves better results than Clean baseline under some settings.
TAB
Smooth
Initial
Clean
OOD
Naive
GMI
FreD
Initial
Clean
OOD
Naive
GMI
FreD
ACC ASR
0.95
0.89
0.89
0.88
0.89
0.80
0.93
0.87
0.81
0.56
0.85
0.87
0.99
0.03
0.08
0.06
0.06
0.04
0.95
0.43
0.69
0.89
0.39
0.45
LO inv
1,2 inv
Initial
Clean
OOD
Naive
GMI
FreD
Initial
Clean
0OD
Naive
GMI
FreD
ACC
0.93
0.91
0.80
0.81
0.87
0.89
0.94
0.91
0.82
0.81
0.94
0.88
ASR
0.97
0.03
0.85
0.91
0.05
0.04
1.00
0.31
0.63
0.59
0.88
0.28
Tro
Troj- WM
Initial
Clean
OOD
Naive
GMI
FreD
Initial
Clean
OOD
Naive
GMI
FreD
ACC ASR
0.94
0.88
0.69
0.89
0.72
0.91
0.94
0.86
0.77
0.75
0.85
0.85
1.00
0.08
0.98
0.99
0.10
0.00
1.00
0.00
0.98
0.98
0.16
0.07
Table 11: Results of FRED combined with Neural Cleanse on defending against various attacks on CIFAR-
10.
F FreD against Adaptive Attacks
Another interesting case is the attacker may have knowledge about the pre-trained GAN and its correspond- ing training data, and design an adaptive atlack to surpass our proposed defense. To investigate such setting, we designed the following attack pipeline:
1) Selecting more promising source class: When performing one-to-one attack, the attacker can choose source class (i.e., the class to be poisoned) exhibiting greater similarity to the OOD data. For example, when targeting a GTSRB dataset-trained target model, the attacker may select class 6 ('80 lifted') as the source class, which has a similar pattern to class 7 in the OOD (TSRD) dataset.
2) Designing a trigger pattern closer to the OOD distribution: To achieve this, we utilize G(z) as our trigger, where 2 = arg max, ((G(z)+ source image), where D is the discriminator of the GAN and is the generator. The rationale behind this approach is that. we expect the poisoned source image to exhibit higher confidence under the Discriminator, making it easier to be generated by GAN.
We compare our designed adaptive attack to all the eight non-adaptive attacks we evaluate in this paper. As the baseline. As shown by Figure 8 below, the attack performance (ASR) of the adaptive procedure is lower compared to the non-adaptive ones without any defense. The reason is that this trigger is constrained to be on a spocific embedding space, limiting its attack capability. However, when comparing 'After Defense' ASR of FreD, the adaptive attack indeed demonstrates better attack performance than the non-adaptive baseline, but the improvement is marginal. It is important to note that the current design is a simple and intuitive
22

---

Published in Transactions on Machine Learning Research (08/2023)
IAB
Smooth
Initial
Clean
OOD
Naive
GMI
FreD
Initial
Clean
OOD
Naive
GMI
Fre
ACC
0.93
0.90
0.89
0.89
0.89
0.89
0.93
0.91
0.85
0.55
0.81
0.56
ASR
0.95
0.03
0.10
0.06
0.06
0.04
0.35
0.93
0.84
0.66
0.48
L0 Inv
L2
Inv
Initial
Clean
OOD
Naive
GMI
FreD
Initial
Clean
OOD
Naive
GMI
FreD
ACC ASR
0.91
0.91
0.83
0.80
0.85
0.88
0.94
0.58
0.82
0.77
0.86
0.87
0.07
0.56
0.03
0.11
0.97
0.25
0.47
Troj-SQ
Troj- WM
Initial
Clean
OOD
Naive
GMT
FreD
Initial
Clean
OOD
Naive
GMI
FreD
ACC
0.91
0.89
0.81
0.90
0.91
0.58
0.80
0.51
0.31
0.58
ASR
Table 12: Results of FRED combined with Tabor on defending against various attacks on CIFAR-10.
example. FRED could be vulnerable a more advanced adaptive attack, which is one the limitations of this paper. Further improvements will be explored in future work.
L0 inv
Smooth
IAB
Troj-WM Blend
Adaptive
inv
Wanet
Troj-SQ
1.0
1.0
0.8
0.8
0.6
0.6
ASR
ASR
0.4
0.4
0.2
0.2
0.0
0.0
0.2
0.4
0.8
0.2
0.4 0.6
0.8
ACC
ACC
(a) Initial (without any defenses)
(b)
FreD
Figure 8: Comparison of the attack performance of non-adaptive attacks (In inv, Ta inc, IAB, Troj-WM, SQ, Wanet, Blend) and adaptive attacks (Adaptive). Figure the left gives attack results without any defense, and figure on the right gives results of attack performance after defense with FRED.
23


Reviews:
[{"review_id": "7J4Rg0fgC5", "reviewer": "Reviewer_cAzf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrength: \n- The paper uses new sampling techniques to improve the existing backdoor removal approaches. \n- The experimental results in the evaluated setting are strong\n\nWeaknesses: \n- The paper can be a bit hard to understand, especially in Section 3. I suggest the authors start by formulating their approach and then explaining it. In the current structure the authors keep mentioning the changes they will do to Equation 3 in the text. It might make it easier to follow the text, if instead they write down the final equation and then explain each component. \n\n- Table 2 can be improved by making it a grid so it would be easier to understand how the parameters affect each other. At the moment the numbers in two columns are significantly different and it is not clear to me how they affect each other. \n\n- While the work relaxes the requirements for in-distribution clean data, now it requires a generative model trained on non-backdoored data. This is a significant assumption and it is not very well discussed in the work. \n\n- One of the main limitations of this approach is that it is heavily reliant on the generative model. As the authors show in Theorem 3 and 7, if the distribution of the training data used to train the generative model and the target model is different, then the generative model won’t generate samples from that distribution. This can be a very limiting factor. The authors add a few experiments showcasing when the distribution of the generative model is different from the target model, however, the experiments are fairly limited. It would be interesting to add some additional experiment showing the effect of different datasets (e.g, using MedMNIST datasets)\n\n- Also the author considered an adaptive adversary which is very good and appreciated, however, the analysis is very limited and shallow. The authors should provide reasoning why such attacks are best attacks that an adaptive adversary can perform. For example for the first attack the authors can show the effect of selecting different classes on the attack success rate. The second attack is not clear from the text. What exactly does `z = arg max  (D(z) + source image)` do? Discriminator does not get z as input also how does output of the discriminator is summed with an image ?  Also the authors argue that they use L_0 inv because it is the best attack, but I am not sure how they are claiming that ? Based on Table 5 it seems this is not true. In any case it would be much better to show the results for all of the attacks since actually this is the main interesting setting. \n\n- The authors should expand more on an adaptive adversary, for example assuming the adversary has access to the generative model, then it can try to force the backdoors to be very similar to the distribution of the generative model. While I think this is what the second attack is supposed to be doing, I couldn’t understand their approach. \n\n\n- The number is Tables 12 and 13 are in percentages while the rest of the paper is not, it is better to make the paper consistent.\nOverall it seems the additional text added to the paper based on the reviewer is not well integrated to the text and it makes it very difficult to understand the work. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new idea of using removing backdoors from a poisoned model. The idea is to use a generative model trained on near-distribution data and then use an existing backdoor removal technique to reduce the effect of the backdoors. The authors show that by modifying the sampling procedure they can improve the quality of the model and also reduce the effect of backdooring. \n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Improved the adaptive attack. \nImprove the structure and flow of the text in the paper", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Zyw0VFwu3R", "reviewer": "Reviewer_xTon", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is generally written and well cited. There are places that are rather confusing, like the structure of Section 3, the placement of the GAN poisoning experiments of Section 4, and the theoretical results in the appendix, but otherwise, I enjoyed reading this paper.\n\nThe idea of the paper is intuitive and combines multiple fields of adversarial machine learning. FreD gets strong results.\n\nWeaknesses:\n\nThe reliance on a pretrained, unpoisoned GAN is rather strong. I think their OOD baseline is an appropriate one that addresses this assumption, but there may be other ways to take advantage of this GAN to improve clean performance.\n\nUnless something really clever is happening in training, robustness to a 50% poisoning rate for any model is implausible, as in Figure 3. I think it’s fine that the paper has these results, but they seem to be more a result of the limited work on backdooring GANs in the literature, rather than a fundamental robustness of GAN training. I’d recommend the authors present these results with a bit more caveats attached. In particular, it’s unlikely that the FreD technique would be robust to an adaptive poisoning attack on the GAN.\n\nComments:\nTheorem 3 and Theorem 7 appear to be the informal/formal version of the same theorem, but the paper never says this. The proof also needs more intuitive guidance.\n\nI wonder if an adversary with access to the pretrained GAN could design a more effective backdoor attack on the paper’s proposed approach. For example, the attack could design a backdoor attack which operates on some features that are not included in the in-distribution data, but are included in the OOD distribution. This would allow the GAN to have backdoored outputs.\n\nOverall assessment:\nInterestingness: Given that this work builds off of model inversion attacks in the literature, and there is also a literature on backdoor sanitization, I think these subcommunities would find this paper interesting. There are a couple limitations that impact this interestingness, including the reliance on the GAN and the prior work which has proposed a similar idea.\n\nCorrectness: The results of training the GAN on poisoned data do not seem to be the results of a worst case poisoning attack. The proof of Theorem 3 is also not written clearly enough for me to understand the intuition.  I am also worried about robustness to adaptive attacks. Small changes to improve the clarity of Theorem 3 and attach more caveats to the results of Figure 3, as well as some discussion of adaptive attacks, would address my concerns here.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a technique to remove backdoors from a classifier by using model inversion attacks to generate new un-backdoored data. While prior work has used model inversion attacks for this purpose in the past, this work improves upon that work by taking advantage of developments in the model inversion field, using a model inversion attack that uses on a GAN pretrained on OOD data.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Discussion of the potential for adaptive attacks\n\nImproved clarity of the proof of theorem 7\n\nDiscussion that GAN backdoors might be improvable\n\nAdding discussion on these points would lead me to raise the claims and evidence score and recommend accept", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9gW2vGISJY", "reviewer": "Reviewer_HdcH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, this submission, is quite extensive in the way it approaches the topic at hand. The writing was a bit hard to understand for me, not due to grammar, but because the intent in each section was not always clear to me, and it was hard to follow along with this investigation. \n\nI have a number of questions and comments that will list below, in no particular order:\n* Fundamentally, the experimental evaluation for this work does not convince me. To circumvent the reliance on clean samples for backdoor removal, a clean dataset from a close domain is used - to train a clean GAN model instead. A larger part of the submission is then allocated to discussing why this might still be sensible, but, to me, the only relevant experiment here is the following: Use the same poisoned dataset that is used to train the model to train a GAN. Then use this GAN as a prior for the model inversion. This is also the only experiment that is directly comparable to other backdoor defenses which only have access to this dataset. This setting is covered briefly in Sec. 4.1, but why is the evaluation section not based on it? In the current submission, this feels like an add-on, instead of a core part of the argument for the sensibility of this defense.\n* The paper experiments with a battery of backdoor attacks, but almost all of these are quite old, and most modern defenses perform reasonably well on these attacks. Usually, I would not feel inclined to suggest another comparison, but comparing with a battery of older does not make sense to me from a security perspective. What about newer attacks, like Blind Backdoors or Sleeper Agent?\n* As an aside, it was surprising to me that Hidden Trick Backdoors, the only attack that was not included in Table 4, is also the only attack in the (broken) appendix Table 10 where the proposed approach does not win.\n* Further, I don't think the experiments concerning adversarial finetuning add meaningful insights to this paper. Performance is close to random on CIFAR-10, even after this defense, and even though the strongest attack, which appears to be AutoAttack 16/255 is left out.\n* Figure 1 is explained only quite late paper and quite hard to understand before then. It would be better if the caption and axes descriptions for this figure were self-contained, if it is not described in the text on related pages.\n\n\nMinor:\n* Typos: \"Most of backdoor removal techniques\", \"can lead to the state-of-the-art data-free\", \"Note that CLP fails defensing\"\n* What are the compute requirements for all parts of this backdoor defense pipeline taken together?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission \"Turning a Curse into a Blessing: Enabling In-Distribution-Data-Free Backdoor Removal via Stabilized Model Inversion\" is concerned with backdoor attacks against image classification models. One possibility to defend against backdoor attacks is to use a backdoor-removal defense. These defenses, such as via trigger synthesis, e.g. I-BAU, often require clean data. This submission instead uses a model inversion approach, using a (clean) GAN to generate approximate data that can then be used to run back-removal via trigger synthesis. This approach is evaluated on a number of classical backdoor attacks and datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "* Run the evaluation in a setting where direct comparisons are possible: Train a GAN on the same poisoned dataset and use that model for model inversion. \n* Include AutoAttack with 16/255 for the adversarial finetuning. Another option would be to remove that section entirely.\n* Fix Tables 10,11, Figure 1.", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pvLTVVuCJF", "reviewer": "Reviewer_BweS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. Although the proposed approach combines existing techniques to remove backdoors from a poisoned model, the design is not trivial with some interesting technical challenges and the paper dealt with well. \n2. The empirical performance is strong and consistent, while other attacks typically have a larger variance. \n\nWeakness:\n1. The performance in adversarial fine-tuning is rather poor and it is clear whether this result can be relevant in practice. Mainly, the robustness against adversarial examples are far below the current state-of-the-art. \n3. The comparison to backdoor defenses based on data sanitization defenses (not backdoor removal from a given poisoned model) is missing. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper combines the advances in model inversion techniques and backdoor removal techniques and show that, effective defenses can still benefit from the strong data-reliant defenses in the lack of in-distribution data. Empirical results show that the proposed attack consistently performs best across different tasks compared to the existing baselines that remove backdoors based on real or synthetic datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I do not have major concerns about this paper. A few comments that I think the authors should address is:\n1. explain why the adversarial fine-tuning is interesting in practice, given the poor performance against adversarial examples compared to current state-of-the-art in RobustBench. This part will not diminish the contribution of this paper, as other baselines also perform quite poorly, instead the question is, whether this results are interesting enough to be included in the paper. \n2. The authors said that minimize the euclidean distance between the synthetic examples and the poisoned samples and found that these synthetic samples are still not recognized as containing backdoors. I am wondering why Euclidean distance a useful metric here? Will cosine similarity be more help? A discuss on this will be helpful. \n3. For poisoning GAN model, the authors mentioned that unless the attacker can inject enormously large fraction of samples, the GAN model is free from backdoors. But it seems even with 50% poisoning ratio, the backdoor detection rate is still as low as 2%. I am not sure why GAN can contain backdoors at large ratios such as 50% (2% detection rate is negligible number to me)? I am raising this because I am not sure if there any bugs in the implementation that caused this (seems somewhat intuitive that, when large number of samples are carefully backdoor, the GAN model should contain backdoors).\n   ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers found the experimental results strong, but not entirely convincing. The reason here is that while the proposed method seems to work well against known attacks, there seems no fundamental reason to believe that the method would prevent more carefully designed attacks.

The reviewer's opinion is split on this paper, and a majority of the reviewers would have enjoyed to see a deeper investigation of adaptive attacks. However, the reviewers agree that the claims wrt. to the investigated attacks are clear and justified by evidence.


For the camera-ready version, please:
- avoid overclaims on 'no clean data needed',
- remove the red annotations
- fix formatting issues (e.g. there are whitespaces after equation (2) and (3), etc.)
- the new figures (e.g. the grid) could perhaps be formatted a bit more nicely, and rearranged (currently page 9 is half-empty), so that all fits again in 12 pages
Decision: Accept (poster)