Boundary augment: A data augment method to defend poison attack

IET Image ProcessingVolume 15, Issue 13 p. 3292-3303 ORIGINAL RESEARCH PAPEROpen Access Boundary augment: A data augment method to defend poison attack Xuan Chen, Xuan Chen Department of Base, Air Force Engineer University, Xi'an, ChinaSearch for more papers by this authorYueNa Ma, Corresponding Author YueNa Ma xakgd2020@163.com Department of Base, Air Force Engineer University, Xi'an, China Correspondence YueNa Ma, Department of Base, Air Force Engineer University, Xi'an, China. Email: xakgd2020@163.comSearch for more papers by this authorShiWei Lu, ShiWei Lu Department of Base, Air Force Engineer University, Xi'an, ChinaSearch for more papers by this authorYu Yao, Yu Yao Department of Base, Air Force Engineer University, Xi'an, ChinaSearch for more papers by this author Xuan Chen, Xuan Chen Department of Base, Air Force Engineer University, Xi'an, ChinaSearch for more papers by this authorYueNa Ma, Corresponding Author YueNa Ma xakgd2020@163.com Department of Base, Air Force Engineer University, Xi'an, China Correspondence YueNa Ma, Department of Base, Air Force Engineer University, Xi'an, China. Email: xakgd2020@163.comSearch for more papers by this authorShiWei Lu, ShiWei Lu Department of Base, Air Force Engineer University, Xi'an, ChinaSearch for more papers by this authorYu Yao, Yu Yao Department of Base, Air Force Engineer University, Xi'an, ChinaSearch for more papers by this author First published: 28 August 2021 https://doi.org/10.1049/ipr2.12325AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract In recent years, Deep Neural Networks(DNNs) have been applied in many fields such as computer vision and natural language processing. Many third-party cloud training platforms have been built to facilitate many individual users or small enterprises for training their models, for example, Colab(google) or AWS cloud platform. For these cloud platforms, there exist many potentially fatal risks, including poison attacks. At the same time, as for federated learning, poison attack is also a severe threat to which. In this paper, a novel method to defend against poison attacks by estimating the distribution of poison data and retraining the backdoor model with a few training data is introduced. The estimated distribution under the manifold DeepFool algorithm fits the poison data well, which can be used to search the manifold boundary of the poisoned data and the clean. Unlike empirical defense methods, the authors' approach is attack-agnostic, which means that the approach is robust for the various attack methods. Also, it is proven that the adversarial training approach is a practical approach to defend against the poison attack. The authors' approach is tested on the datasets MNIST, CIFAR-10, GTSRB and ImageNet. The accuracy of the retrained model decreases slightly, but the ASR drops drastically, which proves that our approach has a powerful generalization to defend against the most poison attacks. 1 INTRODUCTION Deep Neural Networks (DNNs) have been applied in many fields such as computer vision [1], machine translating [2] and automatic speech recognition [3]. In particular, DNNs have achieved great success in real life. However, a good performance deep learning model often requires a complex structure and a large amount of training data. It is more difficult to train a deep learning model individually so that cloud platforms come into being. However, this could lead to a series of threats because the training data and model are out of individual control and the users only have the local testing dataset [4]. Gu et al. firstly showed new security risks: the malicious network trained by adversary BadNet. The malicious network could perform well on benign inputs but poorly on poisoned inputs injected with special patterns. Since then, there have been more attack and defense approaches related to data poison. In this paper, we mainly focus on a particular and powerful poison attack, which aims to slightly change the training data by injecting a backdoor pattern. This type of poison attack is also named backdoor attack. With loss of ambiguity, we also call it poison attack. Except for poisoning-based attacks in this paper, some non-poisoning-based attacks are also proposed, such as [54]. The main research direction is the poisoning-based attacks and defense against the poisoning-based attack. Next, we will show some existing multi-types of poison attacks. The visible trigger poison attacks[4, 5] are most common and belong to a straightforward way to infect the target model. How to hide the trigger is a problem in this form of attack. Besides, there also exists some invisible trigger poison attacks [6, 7]. [8] demonstrated that the poisoned data could be the same with correct labels under poison attack, which is more efficient and stealthy. In the next section, we will explain these attack methods in detail. [9] demonstrated that there exist varying types of defense approaches and the mainstream strategies are about trigger-backdoor mismatch [10, 11], backdoor elimination[12, 13] and trigger elimination[14, 15]. A great diversity of defending methods is proposed to defend these several attacks, including empirical backdoor defenses and certified backdoor defenses. The empirical backdoor defenses often work and perform better in practice, although lacking knowledge-based and theoretical evidence. In contrast, the certified defenses are more reliable. More than that, the empirical defenses are more easily bypassed through strong attacks, which means that empirical defenses have a poor general capacity to defend some attacks past the defenses. Compared with the empirical defenses, the certified defenses would work better and perform excellent general capacity. We observe that there exist increasing poisoned attacks whose poisoned patterns are global to hide stealthily. [5, 16, 17] pointed that fixed local trigger patterns could be easy to be found through manual work. Besides, [4] demonstrated that backdoors are clearly shown in the heatmap of convolutional filters, and [18] use optimization method and reverse engineering to get modified triggers effectively. However, the reverse engineering triggers are still shown in the heatmap of convolutional filters, although their magnitude may be small. In Section 3, we introduce the global pattern trigger and local pattern trigger in detail. Furthermore, different attack methods, including local trigger pattern methods and global trigger pattern methods, will be compared in this paper. Figure 1 shows the whole process of our approach. The user entrusts the third-party platform to train the model, and the malicious third-party platform will return a malicious model with a backdoor. Then the user can retrain the model with little data, and the retrain approaches followthe approaches in our paper. We use PGD adversarial training [19] and the Boundary Augment approach. The Boundary Augment approach uses a modified DeepFool algorithm [20] to do adversarial training and then run data augment. We explain these two methods in Section 4 in more detail. Our contributions are following as: We propose a novel defense method belonging to certified defense, and this approach could defend most of the poison attacks. We implement our defense technology on various datasets, including large-scale datasets, and this defense technology is easy to apply to different neural network architectures. We modify the DeepFool algorithm[20] to search for the distribution of clean and adversarial examples to estimate the manifold of poisoned data. FIGURE 1Open in figure viewerPowerPoint This figure shows the whole process of our approach. The user submits the training datasets to the cloud platform. However, the malicious cloud platform would poison the training dataset and train a backdoor model which performs well with clean data but performs poorly with poisoned data. The user would get a backdoor model from a malicious cloud platform and original training dataset. How to mitigate the poison attack? In our approach, the user would choose a few shot data from the training dataset. The chosen data are random chosen from the shuffled training dataset. Next, we use PGD adversarial training and Boundary Augment to retrain the backdoor model, and then, we will get a mitigated model 2 RELATED WORK [23] proposed an attack method to attack SVM by injecting specially crafted training data to increase the test error of SVM. [4] claimed a similar attack method, which is different from adversarial attack [24] named evasion attack. This approach added some special patterns to the training data and demonstrated that the BadNets attack could change the meaning of traffic signs by installing a backdoor to the target model. In addition, [25] used reverse engineering to generate a general trigger even if there is no knowledge of the datasets. However, the triggers generated by such an approach are local trigger patterns and lacking stealthiness, although this attack strategy is powerful. [16] proposed a clean label attack to modify the BadNets poison attack. Clean label attacks have two approaches, one is utilizing GAN[26] to generate poisoned data, and the other is to add adversarial perturbations to target objects, which could make the poisoned data seem benign to human vision. Besides, [21] raised a stronger attack compared to BadNets, and [27] came up with a method via steganography and regularization to apply invisible triggers. In recent times, [8] proposed a hidden trigger poison attack that hides the trigger most of the time and reveals it at the test time to be late to defend. [7] proposed a novel backdoor method by corrupting only target samples, which benefits the stealthiness of the poisoned data. However, this method is also easy to detect because the pattern is fixed although it has global patterns. [5] utilized physical reflection models to implement poison attack. Compared to the local trigger pattern, this attack approach is much stronger and more stealthy. Recently, the research on the poison attack is numerous. [28] proposed to use a small and smooth warping field to generate poisoned examples, which can make the modification unnoticeable for the human being. [29] proposed how to attack poison attack federated learning with semantic attack approaches. [30] also explored the related idea of semantic poison attacks. The semantic poison attacks do not need to modify the inputs and utilize the high-level semantic information of inputs as triggers to attack. Because the real physical world is complex and full of uncertainty, which is constricted to poison attack, [31] proposed a novel attack method to increase the attack success rate in the physical world through physical transformations. [32] firstly introduced the poison attack for self-supervised learning. In a word, with the deepening of people's understanding of the poison attack, there will appear many more poison attacks, but at the same time, there will also be many defense methods against these attack methods. Figure 2 shows the results of some of these attacks. FIGURE 2Open in figure viewerPowerPoint The first image is the original one, and the next one follows [4] of becoming a poisoned data through adding a small trigger pattern. The third one installs trigger by adversarial perturbing, then adding a small slightly visible trigger pattern [16]. The fourth one follows Embedding backdoor [21], which looks the same as BadNets attack which mainly modifies the target model. Next, the following one develops a ramp signal as a global pattern to reduce the target accuracy[22]. The last one uses physical reflection models to change the target objection slightly. Compared to some others, we can find that the trigger pattern of the Refool attack is exactly not easy to be seen [5] In the earlier research, [4] proposed a detecting method that showed the convolutional filters of the first layer and some neuroses belonging to the backdoor model are activated in comparison to the benign model. [33] explored the spectrum of covariance of feature representation between poisoned data and clean data and used SVD decomposition to compute the outlier score for each example, then chose which to be removed and retrained the backdoor model. Similar to [33, 34] proposed an activation clustering approach to detect poisoned data, whose intuitive idea is to cluster the activation of examples into two clusters, one is a poisoned cluster and one is a benign cluster. However, [35] and [36] successfully attack the activation clustering approach to claim that this detecting method is not robust enough to detect some particular poisoned data. Recently, [37] proposed a detection method that utilizes adversarial perturbations to detect poisoned examples. [38] claimed that using the frequency domain detection will be a feasible way. Without access to the training dataset, [39] proposed an unsupervised anomaly detection to defend against the poison attacks. And more, [40] proposed a black box backdoor detection method to identify poison attacks with the limited query access to the model, which is also making use of reverse engineer to restore the triggers for each class, similar to NeuralCleanse[18]. In addition to these detecting methods, there also exist many active defending methods. [41] proposed a defense scheme combining fine-tuned technology and pruning defense. Initially, the user prunes the model to the adversary and then fine-tunes the pruned model for accuracy. Besides, [18] introduced the Neural Cleanse method, including identifying backdoors and reconstructing possible triggers. Three mitigation schemes: input filters, neuron pruning and unlearning, are proposed to effectively improve the robustness against the poison attack. [42] studied transformation-based preprocessing approaches that are practical to defend many SOTA poison attacks. Whereas, [18] tends to assume that the defender knows the attack method coming from the adversary, which restricts the application of such a defending scheme. [43] perturbed the input by superimposing various image patterns and observing the randomness of predicted classes for perturbed inputs. The input is more likely to be a malicious input if the entropy is low. Otherwise, the input is benign. [44] claimed that adversarial training could mitigate the risk of the poisoned data, which showed that minimizing adversarial risk on the poisoned data is equivalent to optimizing an upper bound of natural risk on the original data. [45] also proposed to use an adversarial training approach to train the model and improve the robustness of the model. [46] showed that the tight hyperparameters could help mitigate poisoning attacks. They encouraged to use L2 regularization to defend against poison attacks. To be fairer to evaluate some attack methods defense, [47] release TROJANZOO, a framework including 12 attack approaches, 15 defense approaches, 6 attack performance metrics and 10 defense utility metrics. There is no doubt that this will be a handy tool for researchers. At the moment, many defense approaches are empirical defenses with no detailed explanation, but in this paper, we provide a certified defense, and we will give clear instruction in the next section. 3 PRELIMINARY 3.1 Poison attack We assume that the user uploads the training dataset to a cloud platform, and the training process is primary on a cloud platform. We denote D b e n i g n = { ( x , y ) } is training dataset, where x denotes training image and y denotes training label. The malicious adversary would generate poison data as follows: x p o i s o n e d = c l i p ( x + α · x t r i g g e r , x m i n , x m a x ) , where α ∈ [ 0 , 1 ] is a trade-off hyper-parameter, x t r i g g e r is the trigger pattern. x m i n , x m a x are the minimum and maximum values of images x. Next, we denote the training process with poison dataset D p o i s o n = x p o i s o n , y follows as: f b a c k d o o r = a r g m i n f E ( x , y ∼ D b e n i g n ∪ D p o i s o n ) [ m a x L ( x , y ) ] , where L ( x , y ) is the loss function. 3.2 Attack success rate To evaluate the performance of poison attacks in the image classification tasks, attack success rate(ASR) is an indicator to measure attack performance. We denote ASR as following: A S R ( P ) = ∑ x , y ∈ D t e s t [ f ( P ( x ) ) = y t a r g e t | y ≠ y t a r g e t ] N D t e s t , where f ( · ) is the backdoor classifier model, P is the poison attack method, D t e s t is the testing dataset and y t a r g e t is the targeted class. 3.3 Global pattern triggers and local pattern triggers There exist many kinds of triggers in the data poison field, and the most common triggers are the local pattern and global pattern. Semantic triggers are also one kind of triggers. There are some similarities between the local pattern triggers and global pattern triggers, but also exist some differences. Next, we introduce the similarities and differences between them. 3.3.1 Similarities The local pattern triggers and global pattern triggers are designed to make the poisoned data more stealthy and less visually distinct from the clean data. The local pattern triggers and global pattern triggers modify the clean inputs by inserting the trigger and then training the model with the poisoned data to run the attack. Sometimes local triggers can help global triggers to attack and achieve a high attack success rate [48]. 3.3.2 Differences Local pattern triggers often do not need a careful design that can have a high attack success rate. However, the adversary needs to make the global pattern triggers elaborately to have better attack capacity and stealthiness. There are many defenses against local pattern triggers, but few against global pattern triggers. For example, [18] propose the Neural Cleanse approach to defend against poison attacks. One existing approach from Neural Cleanse is to use reverse engineer to restore the triggers, which is effective for small-size triggers such as BadNets [4]. However, it would be ill-performance when the size of triggers becomes larger. Also is ConFoc [11] Compared with local pattern triggers, the global pattern triggers have much powerful attack ability and stealthiness. [42] found that the local pattern triggers are vulnerable for the simple transforms. However, the global pattern triggers are robust [5, 28]. 4 APPROACH 4.1 PGD adversarial training In this section, we propose an approach through PGD adversarial training to defend poison attack. Adversarial training is a common defense method to improve the robustness of models, which retrains models by injecting adversarial examples into the training dataset. [19] proposed projected gradient descent (PGD) attack method shown in Figure 3, which is a powerful attack method utilizing first-order information. To fully improve the robustness of the models, [19] adapted adversarial training technology to adversarial examples generated by PGD attack. PGD attack function is as follows: FIGURE 3Open in figure viewerPowerPoint An illustration of the PGD adversarial training x t + 1 = Π x + S ( x t + α · s i g n ( ∇ x L ( θ , x , y ) ) ) , where t is the number of iteration, x is input, y is label, θ is model parameters. Besides, as for PGD adversarial training, the optimization objection is as follows: a r g m i n θ E ( x , y ∼ D ) [ m a x L ( θ , x ′ , y ) ] . ALGORITHM 1. DeepFool Require: Image x, Classifier f. Ensure: Perturbation r ̂ 1: Initialize x 0 ← x , i ← 0 . 2: while k ̂ ( x i ) = k ̂ ( x 0 ) do 3:for k ≠ k ̂ ( x 0 ) do 4: w k ′ ← ∇ f k ( x i ) − ∇ f k ̂ ( x 0 ) ( x i ) 5: f k ′ ← f k ( x i ) − f k ̂ ( x 0 ) ( x i ) 6:end for 7: l ̂ ← arg min k ≠ k ( x 0 ) | f k ′ | ∥ w k ′ ∥ 2 8: r i ← | f i ′ | ∥ w l ̂ ′ ∥ 2 2 w l ̂ ′ 9: x i + 1 ← x i + r i 10: i ← i + 1 11: end while 12: return r ̂ = ∑ i r i [49] considered that adversarial examples are features, and the models are ill-performed because they are not generalized enough for these adversarial examples. Thus, PGD adversarial training technology could be used to find more features of inputs and force the models to learn. The intuitive idea of our paper is that the model could learn more features, which could be easier to distinguish benign or poisoned data from inputs. For poisoned data, adding triggers will change some of their features regardless of whether the trigger pattern is local or global. Figure 4(a) shows the distribution of PGD adversarial example features and poisoned data features. It is clear that the PGD adversarial example features are near to the poisoned data features. The immediate conclusion is that we can retrain the models with PGD adversarial training approach to improve the models' generalization and improve the robustness of models. [44] proposed that adversarial training can be a principled defense method against delusive poisoning, which supports our wonder, and we will show the results of adversarial training in the later section. Whereas, [44] only proved adversarial training could defend against poison attack and did not demonstrate which training approach is more effective. In the next part, we introduce a powerful adversarial training approach. FIGURE 4Open in figure viewerPowerPoint We use the T-SNE approach to plot this figure with features of clean data, adversarial examples and poison data from the MNIST dataset. The red stars represent the clean data, green circles represent the adversarial examples through perturbing the clean data by different approaches and blue triangles represent the poisoned data. We can see that the clean data all gather in a small area, the poisoned data, and adversarial examples all shatter at the whole space. We set the targeted class of adversarial attack and poison attack as 9 4.2 Boundary augment In this section, we propose a novel training approach. PGD adversarial training approach would force the models to learn more features and improve their generalization ability. However, there exist somewhere that the adversarial examples are not overlapped enough, so that is less effective. Another intuitive idea is to search the decision boundary and deploy more data located at the boundary to enhance generalization. We assume that poisoned data only slightly changes the original input, and the poison data would be easy to detect if the inserted triggers are apparent. Thus, we can choose the data augment method to force the decision boundary to move in the direction of the poison data. Figure 5 shows the principle of this process: Figure 5(a) shows the decision boundary of clean data, Figure 5(b) shows the decision boundary of backdoor classifier after data poison, and Figure 5(c) shows that we could find the decision boundary. Then we retrain the model by using data augment approaches, and the decision boundary under poison attacking would move, the new decision boundary would be more generalized, and the poisoned data points would be contained into the original class. FIGURE 5Open in figure viewerPowerPoint In this figure, the left one shows the decision boundary (solid gray line) of a benign model with clean data consisting of three classes. The middle one shows the poison attack process. The red stars and green triangles are the poison data from class 2 and class 3. We can find that the decision boundary changes and is different from the original. On the right, we try to search the boundary and use data augment methods to adjust the boundary. The solid gray line is the decision boundary under poison attack, and the pink dash line is the decision boundary under the retraining approach DeepFool is an attack method in adversarial attack [24, 49], which is an attack algorithm based on hyperplane classification. The key idea of DeepFool is to compute the hyperplane and move the input x to the boundary of the hyperplane, which the classification results will change. DeepFool is shown as Algorithm 1. We suppose the poisoning data only slightly changes the original input; the manifold distribution of the poisoned data is close to the benign data. The decision boundary of the model is between the manifold of the poisoned data and benign data because the classification result of the poisoned data changes. Inspiring from the DeepFool algorithm [20], if we computed the hyperplane, the decision boundary is also found so that we can move the decision boundary and have a correct classification result for the poisoned data. Therefore, our approach can be interpreted as following: We use the DeepFool algorithm to calculate the decision boundary. When we find the decision boundary adjacent to the target class, we can push the data points to move to the decision boundary. The decision boundary can be generalized to make the poisoned data in the correction side of hyperplane. To prevent the data points from greatly surpassing the decision boundary for training and decreasing the classification accuracy, we choose to adjust the step length to confirm that the perturbed data stay around the decision boundary, which does not change the classification results. We Initialize the step λ as 1, and threshold t as 10−4. For the threshold t, if it is too large, the algorithm will end too early, resulting in the large error of the decision boundary, and if it is too small, the algorithm is too slow to run the whole experiment. Figure 6 provides a simple schematic diagram. The DeepFool algorithm is only used to compute decision boundary; our approach computes the more accurate boundary and introduces adversarial training to improve the robustness of model to mitigate the attack. Our algorithm follows Algorithm 2. We call it Boundary Augment algorithm. FIGURE 6Open in figure viewerPowerPoint The difference between DeepFool algorithm with the Boundary Search, and one part of the Boundary Augment algorithm. The green point is the clean example, the red circles and red triangle are the inputs adding perturbations. the first image of the Boundary Augment is that the perturbation is λ i · r i (red arrow), but k ( x i + λ i · r i ) ≠ k ̂ ( x 0 ) , so we adjust the perturbation into λ i ← λ i / 2 (black arrow) until k ( x i + λ i · r i ) ≠ k ̂ ( x 0 ) . and the second image is that k ( x i + 1 + λ i + 1 · r i + 1 ) = k ̂ ( x 0 ) in the next step. The final image is that the inputs with perturbations will converge near the decision boundary. Thus, step λ can guarantee that the inputs with perturbations will not cross the decision boundary and be closer to the decision boundary ALGORITHM 2. Boundary Augment Require: Image x, Classifier f, Step λ=1, Threshold t = 10 − 4 Ensure: New classifier f ̂ 1: Initialize x 0 ← x , λ 0 ← λ , i ← 0 . 2: while λ ≤ t do 3:for k ≠ k ̂ ( x 0 ) do 4: w k ′ ← ∇ f k ( x i ) − ∇ f k ̂ ( x 0 ) ( x i ) 5: f k ′ ← f k ( x i ) − f k ̂ ( x 0 ) ( x i ) 6:end for 7: l ̂ ← arg min k ≠ k ( x 0 ) | f k ′ | ∥ w ′ k ∥ 2 8: r i ← | f i ′ | ∥ w l ̂ ′ ∥ 2 2 w l ̂ ′ 9: x i + 1 ← x i + λ i · r i 10:while k ̂ ( x 0 ) ≠ k ( x i + 1 ) do 11: λ i ← λ i / 2 12: x i + 1 ← x i + λ i · r i 13:end while 14: i ← i + 1 15: end while 16: Data Augment(x) 17: Retrain classifier f 18: returnnew classifier f ̂ In Algorithm 2, where k ( · ) follows the notations from [20]: f ̂ = a r g m a x k ( f k ( x ) ) and f k ( x ) is the output of f ( x ) that corresponds to the k th class. Figure 4(b) shows the feature distribution of the Boundary Augment approach. The augmented data points are much closer to poison data points compared to the PGD adversarial training. In other words, the Boundary Augment approach is more powerful than the PGD adversarial training approach. To evaluate SIG and Refool attack methods, we compute the accuracy of retrained models and ASR of attacks on the GTSRB dataset. We choose high-quality images whose height and width are larger than 100. Figure 7 shows the results between PGD adversarial training and Boundary Augment approach. It is clear that our approach outperforms PGD adversarial training. FIGURE 7Open in figure viewerPowerPoint The figure shows the accuracy of PGD adversarial training approach and boundary augment approach, ASR of five attack methods 4.3 Ensemble training Although the retraining model with PGD adversarial training and boundary search approaches is effective to defend against poison attacks, we find that the attacks under the global trigger patterns are more powerful than the local trigger patterns [5]. [42] claimed that attacks with local trigger patterns are vulnerable when transforming the poisoned inputs with some simple methods. These transforming methods can mismatch the triggers