Robust anomaly detection and backdoor attack detection via differential privacy

Outlier detection and novelty detection are two important topics for anomaly detection . Suppose the majority of a dataset are drawn from a certain distribution , outlier detection and novelty detection both aim to detect data samples that do not fit the distribution . Outliers refer to data samples within this dataset , while novelties refer to new samples . In the meantime , backdoor poisoning attacks for machine learning models are achieved through injecting poisoning samples into the training dataset , which could be regarded as “ outliers ” that are intentionally added by attackers . Differential privacy has been proposed to avoid leaking any individual ’ s information , when aggregated analysis is performed on a given dataset . It is typically achieved by adding random noise , either directly to the input dataset , or to intermediate results of the aggregation mechanism . In this paper , we demonstrate that applying differential privacy can improve the utility of outlier detection and novelty detection , with an extension to detect poisoning samples in backdoor attacks . We first present a theoretical analysis on how differential privacy helps with the detection , and then conduct extensive experiments to validate the effectiveness of differential privacy in improving outlier detection , novelty detection , and backdoor attack detection . 1 INTRODUCTION . Given a dataset where most of the samples are from a certain distribution , outlier detection aims to detect the minorities in the dataset that are far from the distribution , while the goal of novelty detection is to detect newly observed data samples that do not fit the distribution . On the other hand , poisoning examples that are intentionally added by attackers to achieve backdoor attacks could be treated as one type of “ outliers ” in the training dataset . Using machine learning for outlier/novelty detection is typically to train a model that learns the distribution where the training data samples are drawn from , and the final trained model could give a high anomaly score for the outliers/novelties that deviate from the same distribution . In both cases , the machine learning model is not supposed to learn from the outliers in the training dataset . Unfortunately , deep learning models that contain millions of parameters tend to remember too much ( Song et al . [ 2017 ] ) , and can easily overfit to rare training samples ( Carlini et al . [ 2018 ] ) . Protecting data privacy has been a major concern in many applications , because sensitive data are being collected and analyzed . Differential privacy has been proposed to “ hide ” certain input data from the output ; that is , by looking at the statistical results calculated from input data , one can not tell whether the input data contain a certain record or not . The way of applying differential privacy is to add random noise to the input data or the data analysis procedure , such that the output difference caused by the input difference can be hidden by the noise . A known fact is that differential privacy implies stability ( Kasiviswanathan et al . [ 2011 ] ) . Particularly , a differentially private learning algorithm is stable in the sense that the model learned by the algorithm is insensitive to the removal or replacement of an arbitrary point in the training dataset ( Bousquet & Elisseeff [ 2002 ] ) . When the training dataset contains a handful of outliers , the output model of a stable learning algorithm should be close to the one trained on the clean portion of the training set . Intuitively , compared with the model trained on contaminated dataset , the one trained on clean data could be better at distinguishing outliers from normal data . Therefore , differential privacy can potentially be leveraged to improve the identification of outliers . This motivates us to apply differential privacy to anomaly detection and defense against backdoor attacks . Our contribution . First , we present a theoretical explanation on why differential privacy can help to detect outliers from a training and testing dataset , as well as an analysis on the relationship between the number of outliers to detect and the amount of random noise to apply . Second , to demonstrate the effectiveness , we apply differential privacy to an autoencoder network trained on a constructed MNIST dataset with injected outliers , for both outlier detection and novelty detection , to show how much the utility could be improved with different amount of outliers and noise . Third , we apply differential privacy to a real-world task - Hadoop file system log anomaly detection . System log anomaly detection is an important topic in computer security . Our proposed method greatly improves upon the state-of-the-art system in this field . The results indicate that differential privacy is able to eliminate almost all the false negatives , and achieve significantly improved overall utility , compared with the current state-of-the-art work DeepLog ( Du et al . [ 2017 ] ) . Finally , via a proof-of-concept experiment using MNIST dataset with injected poisoning samples , we show that the idea of outlier detection could be extended to backdoor attack detection , and that differential privacy is able to further improve the performance . 2 PRELIMINARY . Given an input dataset and an aggregation mechanism , differential privacy ( Dwork [ 2011 ] ) aims to output the requested aggregation results , which are guaranteed not to reveal the participation of any individual data record . Formally , differential privacy is defined as below : Definition 1 ( Differential privacy ) . A randomized mechanismM applied to data domain D is said to be ( , δ ) -differentially private if for any adjacent datasets d , d′ in D , and any subset of outputs S ⊆ Range ( M ) , it holds that Pr [ M ( d ) ∈ S ] ≤ e Pr [ M ( d′ ) ∈ S ] + δ , where stands for the privacy bound , and δ stands for the probability to break this bound . The adjacent datasets d , d′ could be understood as two databases , where only one record differs , i.e. , ‖d − d′‖1 = 1 . Differential privacy guarantees that the difference between d and d′ are not revealed through inspecting the outputsM ( d ) andM ( d′ ) . Clearly , the closer is to 0 , the more indistinguishableM ( d ) andM ( d′ ) are , and hence the stronger the privacy guarantee is . A common approach to enforcing differential privacy for a function f : D→ R , is to add random Gaussian noise N ( 0 , σ2 ) to perturb the output in R. The intuition is that , for given adjacent datasets d and d′ , one can not tell whether the difference between f ( d ) and f ( d′ ) is incurred by the single record that differs in d and d′ , or by the random noise being applied . The magnitude of Gaussian noise needs to be tailored to the maximum difference between f ( d ) and f ( d′ ) , which is formally defined as L2-sensitivity . Definition 2 ( L2-sensitivity ) . The L2-sensitivity for a function f : D→ R is : ∆ = max d , d′∈D ‖d−d′‖1=1 ‖f ( d ) − f ( d′ ) ‖2 The noise scale σ to apply can be calculated as below ( Dwork et al . [ 2014 ] ) . Theorem 1 . To perturb a function with sensitivity ∆ under ( , δ ) - differential privacy , the minimum noise scale σ of Gaussian mechanism is given by : σ = ∆ · √ 2 ln 1.25δ , where ∈ ( 0 , 1 ) . Deep learning with differential privacy ( Abadi et al . [ 2016 ] ) The procedure of deep learning model training is to minimize the output of a loss function , through numerous stochastic gradient descent ( SGD ) steps . Abadi et al . [ 2016 ] proposed a differentially private SGD algorithm that works as follows . At each SGD step , a fixed number of randomly selected training samples are used as a mini batch . For each mini batch training , the following two operations are performed to enforce differential privacy : 1 ) clip the norm of the gradient for each example , with a clipping bound C , to limit the sensitivity of gradient ; 2 ) sum the clipped per-example gradients and add Gaussian noise N ( 0 , σ2 ) , before updating the model parameters . Abadi et al . [ 2016 ] further proposed a moment accounting mechanism which calculates the aggregate privacy bound when performing SGD for multiple steps . Differential privacy is immune to post-processing . Therefore , the output of the trained model for any queries enjoys the same privacy guarantee as the above SGD-based training process . 3 THE CONNECTION BETWEEN DIFFERENTIAL PRIVACY AND OUTLIER DETECTION . By definition , random noise added into model training for differential privacy hides the influence of a single record on the learned model . Intuitively , if applying differential privacy to the training process , the contribution of rare training examples will be hidden by random noise , resulting in a model that underfits the outliers . Such model will facilitate novelty and outlier detection because it will be less confident in predicting the atypical examples . In this section , we first present a theorem to precisely characterize the above intuition , and then analyze the relationship between the number of outliers in the training dataset and the amount of noise to apply . Notations LetZ be the sample space andH be the hypothesis space . The loss function l : H×Z → R measures how well the hypothesis h ∈ H explains a data instance z ∈ Z . A learning algorithm A : Zn → H learns some hypothesis A ( S ) given a set S of n samples . For instance , in supervised learning problems , Z = X × Y , where X is the feature space and Y is the label space ; H is a collection of models h : X → Y ; and l ( h , z ) measures how well h predicts the feature-label relationship z = ( x , y ) . Let S = { z1 , . . . , zn } be a set of independent samples drawn from an unknown distribution D on Z . For a given distribution D , an oracle hypothesis is the one that minimizes the expected loss : h∗ = arg min h Ez∼D [ l ( h , z ) ] ( 1 ) We define an outlier as a data instance that has significantly different loss from the population under the oracle hypothesis . Definition 3 . We say z̃ is an outlier with regard to distribution D if l ( h∗ , z̃ ) − Ez∼D [ l ( h∗ , z ) ] ≥ T ( 2 ) where T is a constant that depends only on D. We will prove the usefulness of differential privacy to detect outliers for the classes of learning algorithms that produce hypotheses converging to the optimal hypothesis asymptotically pointwise . We define such learning algorithms to be uniformly asymptotic empirical risk minimization ( UAERM ) . Definition 4 . A ( possibly randomized ) learning algorithm A is UAERM with rate ξ ( n , A ) if for any distribution D defined on the domain Z , it holds that ∀z |ES∼DnEh∼A ( S ) l ( h , z ) − l ( h∗ , z ) | ≤ ξ ( n , A ) ( 3 ) In the definition , we make it explicit that the rate ξ ( n , A ) depends on the learning algorithm A . For instance , if A is a differentially private learning algorithm , the rate will depend on the privacy parameters . In that case , with slight abuse of notation , we will denote the rate for a ( , δ ) -differentially private learning algorithm trained on n data instances by ξ ( n , , δ ) . Due to the nonconvexity of their loss functions , neural networks may not enjoy a useful , tight characterization of the learning rate . Thus , we will empirically verify that using noisy SGD to learn differentially private neural networks is UAERM . Moreover , as we will show in the experiment , ξ ( n , , δ ) grows as privacy parameters and δ become smaller . Intuitively , this is because larger noise is required to ensure stronger privacy guarantees , which , on the other hand , slows down the convergence of the learning algorithm . Without loss of generality , we assume that 0 ≤ l ( h , z ) ≤ 1 . The following theorem exhibits how the prediction performance of differentially private models on normal data will differ from outliers and connects the difference to the privacy parameters of the learning algorithm and the amount of outliers in the training data . Theorem 2 . Suppose that a learning algorithm A is ( , δ ) -differentially private and UAERM with the rate ξ ( n , , δ ) . Let S′ = S ∪ U , where S ∼ Dn and U contains c arbitrary outliers . Then Eh∼A ( S′ ) l ( h , z̃ ) − Eh∼A ( S′ ) Ez∼Dl ( h , z ) ≥ T − 2 ( ξ ( n , , δ ) + √ n ( e − 1 + δ ) 2 2 log 2 γ + ec − 1 + cec δ ) ( 4 ) with probability at least 1− γ . The two terms in the left-hand side of ( 4 ) represent the model ’ s prediction loss on outliers and normal test data drawn from D , respectively . Due to the stochasticity of differentially private learning algorithms , the difference is characterized by the expectation taken over the randomness of the learned models . The theorem establishes a lower bound on the prediction performance difference between normal and outlier data . A larger difference indicates that identifying outliers will be easier . The impact of privacy parameters on the lower bound manifests itself in two aspects . On one hand , stronger privacy guarantees ( i.e. , smaller and δ ) will require higher noise to be added into the training process , which increases the learning rate ξ ( n , , δ ) . On the other hand , increasing privacy level will improve the stability of the learning algorithm ; the resulting models tend to ignore the outliers in the training set and become closer to the ones trained on completely clean data , thus making the outlier detection more effective . The second aspect is embodied by the fact that the terms except ξ ( n , , δ ) in the parenthesis of the lower bound grow with and δ . Therefore , the privacy parameters can not be too large or too small in order to ensure optimal anomaly detection performance . Moreover , the relationship between the right-hand side of ( 4 ) and c indicates that the anomaly detection problem will be more difficult with more outliers in the training set . Dissecting the righthand side of ( 4 ) , we further observe that c appears always in tandem with . This implies that for larger number of outliers in the training set ( i.e. , c is larger ) , we will need to tune down and δ to maintain the same novelty detection performance . Last but not least , the definition of outliers in our paper is quite general—it does not make any assumptions about how the outliers are generated . Also , we do not make assumptions about whether these outliers are in training or test data . Therefore , our analysis can shed light on detecting various types of anomalies , including but not limited to outlier/novelty detection , backdoor detection , and noisy label detection . In the following experimental section , we will focus our evaluation on outlier/novelty detection and defense against backdoor attacks .

Summary: This paper proposes the idea of using differential privacy (DP) to improve the performance of outlier and novelty detection. Differential privacy was proposed as a privacy metric which limits the contribution of a single data point in the training set to the output. This property naturally controls how poisoned data would affect the output of the learned model. Under the assumption that a well-trained model would incur a higher loss on the outliers, the paper gives a theoretic bound on how this loss will decrease if there are poisoned samples in the training set.