Detecting adversarial attacks through decoy training

Decoy data is generated from regular data. A deep neural network, which has been trained with the regular data, is trained with the decoy data. The trained deep neural network, responsive to a client request comprising input data, is operated on the input data. Post-processing is performed using at least an output of the operated trained deep neural network to determine whether the input data is regular data or decoy data. One or more actions are performed based on a result of the performed post-processing.

BACKGROUND

This invention relates generally to computer security and, more specifically, relates to detecting adversarial attacks through decoy training.

Abbreviations that may be found in the specification and/or the drawing figures are defined below, at the beginning of the detailed description section.

Deep neural networks (DNNs) have achieved remarkable performance on many tasks, including visual recognition. However, recent research has shown that DNNs are vulnerable to adversarial attacks. In these attacks, and attacker intentionally injects small perturbations (also known as adversarial examples) to a DNN's input data to cause misclassifications. Such attacks are dangerous if the targeted DNN is used in critical applications, such as autonomous driving, robotics, or visual authentications and identification. For instance, a real physical adversarial attack on autonomous DNN models has been shown, which caused the target DNN models to misclassify “stop sign” as “speed limit”. See Eykholt et al., “Robust Physical-World Attacks on Deep Learning Models”, arXiv:1707.08945v5 [cs.CR] 10 Apr. 2018.

SUMMARY

This section is meant to be exemplary and not meant to be limiting.

In an exemplary embodiment, a method is disclosed. The method comprises generating decoy data from regular data and training a deep neural network, which has been trained with the regular data, with the decoy data. The method also includes, responsive to a client request comprising input data, operating the trained deep neural network on the input data, and performing post-processing using at least an output of the operated trained deep neural network to determine whether the input data is regular data or decoy data. The method includes performing one or more actions based on a result of the performed post-processing.

An exemplary apparatus includes one or more processors and one or more memories including computer program code. The one or more memories and the computer program code are configured to, with the one or more processors, cause the apparatus to perform at least the following: generating decoy data from regular data; training a deep neural network, which has been trained with the regular data, with the decoy data; responsive to a client request comprising input data, operating the trained deep neural network on the input data; performing post-processing using at least an output of the operated trained deep neural network to determine whether the input data is regular data or decoy data; and performing one or more actions based on a result of the performed post-processing.

In another exemplary embodiment, a computer program product is disclosed. The computer program product comprises a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a device to cause the device to perform operations comprising: generating decoy data from regular data; training a deep neural network, which has been trained with the regular data, with the decoy data; responsive to a client request comprising input data, operating the trained deep neural network on the input data; performing post-processing using at least an output of the operated trained deep neural network to determine whether the input data is regular data or decoy data; and performing one or more actions based on a result of the performed post-processing.

Another exemplary embodiment is an apparatus, comprising: means for generating decoy data from regular data; means for training a deep neural network, which has been trained with the regular data, with the decoy data; means, responsive to a client request comprising input data, for operating the trained deep neural network on the input data; means for performing post-processing using at least an output of the operated trained deep neural network to determine whether the input data is regular data or decoy data; and means for performing one or more actions based on a result of the performed post-processing.

DETAILED DESCRIPTION

The following abbreviations that may be found in the specification and/or the drawing figures are defined as follows:AI artificial intelligenceAPI application programming interfaceDNN deep neural network, e.g., an artificial neural network (ANN) with multiple hidden layers between the input and outputFGSM fast gradient step methodI/F interfaceN/W networkML machine learning

INTRODUCTION

An introduction to this general area is now presented. As explained above, recent research has shown that DNNs are vulnerable to adversarial attacks. Adversarial attacks intentionally inject small perturbations (also known as adversarial examples) to a DNN's input data to cause misclassifications.FIG. 1is an example of an adversarial attack10on an original DNN using an image to create a misclassification by the original DNN.

FIG. 1shows an original image50of a panda. The original image50has a 66 percent (%) probability of a DNN's selecting the class “panda” for the image. An adversarial attack10injects perturbations c, illustrated by image70, into a data stream with the original image50to create the final image90. The final image90, which has been perturbed by the adversarial attack10, causes the DNN to select the class “dog” with a 99.6% confidence. Thus, the adversarial attack caused a high probability of error in image detection for this example.

Several forms of defense to these adversarial attacks have been proposed. These include adversarial training to reduce error rate in classification, see Goodfellow et al., “Explaining And Harnessing Adversarial Examples”, arXiv:1412.6572 (2014); and Miyato et al., “Virtual Adversarial Training: a Regularization Method for Supervised and Semi-supervised Learning”, arXiv:1704.03976 (2017). Another form of defense is input preprocessing, see Meng et al., “MagNet: a Two-Pronged Defense against Adversarial Example”, CCS '17, Oct. 30-Nov. 3, 2017, Dallas, Tex., USA; and Xu et al., “Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks”, arXiv:1704.01155v2 [cs.CV] 5 Dec. 2017. A further form of defense is different model hardening, see Papernot et al., “Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks”, arXiv:1511.04508v2 [cs.CR] 14 Mar. 2016; and Zantedeschi et al., “Efficient Defenses Against Adversarial Attacks”, Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pages 39-49 (2017).

Although these defenses make it harder for attackers to generate adversarial examples, prior works have shown that those defenses are still vulnerable and they can still generate successful adversarial attacks. See the following: Carlini et al., “Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods”, arXiv: 1705.07263v2 [cs.LG] 1 Nov. 2017; Carlini et al., “MagNet and ‘Efficient Defenses Against Adversarial Attacks’ are Not Robust to Adversarial Examples”, arXiv:1711.08478v1 [cs.LG] 22 Nov. 2017; and Athalye et al., “Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples”, arXiv:1802.00420v2 [cs.LG] 15 Feb. 2018

“Security through obscurity” has become a byword for security practices that rely upon an adversary's ignorance of the system design rather than any fundamental principle of security. Security through obscurity is the belief that a system of any sort can be secure so long as nobody outside of its implementation group is allowed to find out anything about its internal mechanisms. That is, security through obscurity is a security practice which relies on the secrecy of the design or implementation as the main method of providing security for a system. An example of “security through obscurity” is described in Anderson, R., “Why information security is hard—an economic perspective”, in Proceedings of the 17th Annual Computer Security Applications Conference (ACSAC) (2001), pp. 358-365. History has demonstrated that such practices offer very weak security at best, and are dangerously misleading at worst, potentially offering an illusion of security that may encourage poor decision-making. For the latter, see Merkow, M. S. and Breithaupt, J., Information Security: Principles and Practices. Pearson Education, 2014, chapter 2, specifically page 25. Specifically, in DNN contexts, it has been demonstrated that obfuscated gradients, a type of gradient masking, are ineffective to protect against an adaptive attacker, leading to a false sense of security in defenses against adversarial examples. See Athalye et al., “Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples”, arXiv:1802.00420v2 [cs.LG] 15 Feb. 2018.

Security defenses based on deception potentially run the risk of falling into the “security through obscurity” trap. If the defense's deceptiveness hinges on attacker ignorance of the system design—details that defenders should conservatively assume will eventually become known by any suitably persistent threat actor—then any security offered by the defense might be illusory and therefore untrustworthy. Consequently, it i important to carefully examine the underlying basis upon which decoy training can be viewed as a security-enhancing technology.

Like all deception strategies, the effectiveness of decoy training relies upon withholding certain secrets from adversaries (e.g., which decoy representations are used in the training process). But secret-keeping does not in itself disqualify decoy training as obscurity-reliant. For example, modern cryptography is frequently championed as a hallmark of anti-obscurity defense despite its foundational assumption that adversaries lack knowledge of private keys, because disclosing the complete implementation details of crypto algorithms does not aid attackers in breaking cyphertexts derived from undisclosed keys. Juels (see Juels, A., “A bodyguard of lies: the use of honey objects in information security”, in Proceedings of the 19th ACM Symposium on Access Control Models and Technologies, 2014, ACM, pp. 1-4) defines indistinguishability and secrecy as two properties required for successful deployment of honey systems. These properties are formalized as follows.

Consider a simple system in which S={s1, . . . , sn} denotes a set of n objects of which one, s*=sj, for j∈{1, . . . , n} is the true object, while the other n−1 are honey objects. The two properties then are as follows.

1) Indistinguishability: To deceive an attacker, honey objects must be hard to distinguish from real objects. They should, in other words, be drawn from a probability distribution over possible objects similar to that of real objects.

2) Secrecy: In a system with honey objects, j is a secret. Honey objects can, of course, only deceive an attacker that does not know j, so j cannot reside alongside S. Kerckhoffs' principle therefore comes into play: the security of the system must reside in the secret, i.e., the distinction between honey objects and real ones, not in the mere fact of using honey objects.

Overview of Exemplary Embodiments

An overview of some of the exemplary embodiments is now presented. In contrast to existing works that try to harden DNNs to make it impractical to generate adversarial examples, an exemplary embodiment herein introduces decoy training as a novel methodology for misdirecting and detecting adversarial attacks. Decoy training may be thought of to “soften” DNNs to facilitate the generation of adversarial samples that are similar to pre-defined decoys used to train the DNN. Specifically, in an exemplary embodiment, we generate multiple decoys for each class so that those decoys will become different local minima for gradient descent. As is known, gradient descent is a first-order iterative optimization algorithm for finding a minimum of a function. As a result, when attackers attempt to generate adversarial examples based on gradient descent, they are misdirected towards data similar to the training decoy set. This is true because decoy training will make decoys become overfitted in the model. Therefore, it implicitly creates locally optimum paths for gradient decent.

There are several benefits of using decoy training in adversarial contexts:

1) Decoy training is stealthy and transparent to attackers since attackers can still generate adversarial examples, although the generated examples are similar to the training decoys.

2) An approach using decoy training is robust to white-box attacks, where attackers have access to the entire DNN model.

3) Decoy training has low false positives since any input data that are similar to decoys are true adversarial attacks.

4) An approach using decoy training can detect both known and unknown adversarial attacks since such an approach does not depend on any known adversarial examples.

Decoy training as a methodology satisfies indistinguishability and secrecy by design. Indistinguishability derives from the inability of an attacker to determine whether an apparently successful attack is the result of exploiting a DNN model using a derived decoy. Secrecy implies that the decoy training set should be secret. However, full attacker knowledge of the design and implementation details of the machine learning model and algorithms does not disclose which decoys have been selected in the training process. Adapting Kerckhoffs' principle for deception, decoy training is not detectable even if everything about the system, except the training set, is public knowledge.

This argues that decoy training as a paradigm does not derive its security value from obscurity. Rather, its deceptions are based on well-defined secrets. Maintaining this confidentiality distinction between the publicness of the DNN design and implementation details, versus the secrecy of the training set is important for crafting robust, effective deceptions to protect against adversarial attacks.

Exemplary System Overview

More detail regarding these techniques is presented after a system into which the exemplary embodiments may be used is described. InFIG. 2A, a client computer system110is in wired and/or wireless communication with a server computer system170in a communications network100-1. The client computer system110communicates with server computer system170via one or more wired or wireless networks197and wired links176,177or wireless links178,179. The client101may communicate directly with the server computer system170via the one or more client interface elements195or may communicate with the server via the one or more wired or wireless networks197. The client101is illustrated in this example as a human being101-1. However, the client101can be anything that tries to use or uses the AI service(s) provided by the server computer system170and, e.g., its decoy API150. Other examples of clients101are illustrated in more detail in reference toFIGS. 2 and 3.

The server computer system170includes one or more processors152, one or more memories155, one or more network interfaces (N/W I/F(s))161, one or more transceivers160, and client interface circuitry175, interconnected through one or more buses157. Each of the one or more transceivers160includes a receiver, Rx,162and a transmitter, Tx,163. The one or more transceivers160are connected to one or more antennas158. The one or more memories155include computer program code153comprising a first neural network, DNN f(x),280, a second neural network, DNN g(x),290, and a decoy API150-2. Although the DNNs280and290are shown separately from the decoy API150, they may also be part of the decoy API150. The server computer system170includes a decoy API150, comprising one of or both parts150-1and/or150-2, which may be implemented in a number of ways. The decoy API150may be implemented in hardware as decoy API150-1, such as being implemented as part of the one or more processors152. The decoy API150-1may be implemented also as an integrated circuit or through other hardware such as a programmable gate array. In another example, the decoy API150may be implemented as decoy API150-2, which is implemented as computer program code153and is executed by the one or more processors152. For instance, the one or more memories155and the computer program code153are configured to, with the one or more processors152, cause the eNB170to perform one or more of the operations as described herein. It should also be noted that the devices shown in the server computer system170are not limiting and other, different, or fewer devices may be used.

The one or more buses157may be address, data, and/or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, wireless channels, and the like. The client interface circuitry175communicates with one or more client interface elements195, which may be formed integral with the server computer system170or be outside the server computer system170but coupled to the server computer system170. The client interface elements195include one or more of the following: one or more camera(s); on or more radar sensors; one or more audio device(s) (such as microphone(s), speaker(s), and the like); one or more sensor(s) (such as GPS sensor(s), fingerprint sensor(s), orientation sensor(s), and the like); one or more displays; and/or one or more keyboards. This list is not exhaustive or limiting, and other, different, or fewer elements may be used.

In brief, the server computer system170performs training (e.g., under control of the decoy API150) of both the DNN f(x)280and the DNN g(x)290with regular and decoy data. The output of the DNN f(x)280may be a confidence score for each class, while the output for the DNN g(x)290may be a confidence score for either “regular data” or “decoy data”. The decoy API150causes the server computer system170to use both outputs to determine a final output. If the final output indicates “decoy data”, the server computer system170can perform one or more predetermined protective actions. This is described in more detail below.

It is noted that the computer system170is indicated as a “server”. While there may be a client-server relationship between the client computer system110and the server computer system170, this is merely one example. The server computer system170may also simply be a “computer system”, and there could be a peer-to-peer relationship between the computer system170and the client computer system110(or even the client101). The server computer system170may be considered to be a server computer system170-7, as other examples are also possible.

Concerning other possible examples,FIG. 2Billustrates another example of a different communications network100-2. In this example, the client101-2may be a computer system. This example includes one or more camera(s)195-1and/or one or more radar(s)195-2, that also be the client I/F element(s)195and be, e.g., routed through the client I/F circuitry175. For instance, the communications network100-2could be in a self-driving vehicle (such as an automobile or a truck), the “server” computer system170-2could be a processing element within the vehicle, and the client101-2could be another processing element within the vehicle. For example, the client101-2could be the main control system for the vehicle, and could use multiple computer systems, e.g., to analyze road conditions, traffic, weather conditions, and the like, and the server computer system170-2could perform analysis of data from the one or more camera(s)195-1and/or the one or more radar(s)195-2. The client I/F circuitry175and the buses157may include communication elements to communicate using vehicular buses using protocols including one or more of Controller Area Network (CAN), Local Interconnect Network (LIN), and/or others.

FIG. 2Cillustrates another exemplary embodiment. In this example, the communications network100-3includes the server computer system170-3, and may or may not include other networks or elements. The client101-3comprises a program residing in the computer program code153and accessing the decoy API150. The server computer system170-3could also include the client I/F element(s)195, camera(s)195-1, and/or radar(s)195-2, if desired.

For each of the server computer systems170inFIGS. 2A, 2B, and 2C, these are merely exemplary. Such computer systems170may not be true “servers” and may include additional or fewer elements than shown.

Additional Details and Examples

Now that one possible exemplary system has been described, the exemplary embodiments are described in more detail. As previously described, exemplary embodiments herein introduce decoy training as a novel methodology for misdirecting and detecting adversarial attacks. As a result, exemplary implementations of the exemplary embodiments will “soften” DNNs to facilitate the generation of adversarial samples that are similar to pre-defined decoys used to train the DNN.

Refer toFIG. 3, which illustrates examples of a first adversarial attack10(fromFIG. 1) on an original DNN using an image to create a misclassification by an original neural network and a second adversarial attack300using the same image to create a misclassification by a neural network with decoy training. The adversarial attack10has already been described in reference toFIG. 1. The adversarial attack10is only for comparison, as a currently existing reference system. For the second adversarial attack300, the original image50has a 66 percent (%) probability of the DNN's selecting the class “panda” for the image. The second adversarial attack300is performed on the same DNN but that has undergone decoy training in accordance with an exemplary embodiment herein. The second adversarial attack300injects perturbations ε′, illustrated by image370, into a data stream with the original image50to create the final image390. The final image390, which has been perturbed by the adversarial attack300, causes the DNN to select the class “dog” with a 99.8% confidence.

To implement decoy training and make adversarial examples more detectable, one possible exemplary method first generates training decoy samples for each DNN class, where the decoy data is similar to the regular training samples (i.e., data) of each class but may implement specially crafted patterns (e.g., watermarks). Then the method assigns counterfeit labels to the training decoy data (e.g., a decoy resembling the image of a cat is labeled as class “dog”). Next, the DNN is trained on both regular and decoy data. As a result, the regular data will still be classified as their original classes but the adversarial data, which are generated through, e.g., the gradient descent algorithm will resemble decoy data, and will be misclassified to incorrect classes.

In this way, this exemplary approach can easily detect such adversarial examples. This may be performed, for instance, by checking a distribution of the logits layer (by performing a categorical discrimination). As is known, the logits layers may be a penultimate layer in a neural network, which will return raw values for predictions, and the final layer may be a softmax layer that may derive probabilities from the logits layer, e.g., by applying softmax activation. This may also be performed by training a separate DNN classifier (seeFIG. 4A, described below) to distinguish between regular and decoy samples. For example, leveraging this exemplary approach, detection and tracking of adversarial examples can be realized as follows in an exemplary embodiment:

a) a separate DNN model g(x) is trained with an original training set and a corresponding decoy data set (maintaining separate labels for genuine and decoy data);

b) client requests to a front-end ML API are also relayed to the separate DNN g(x) to decide whether the input data is a reconstructed decoy; and

c) the system may flag the request as suspicious and take some predetermined protective action in response to the DNN g(x) outputting a decoy class.

For example, suppose we generate a decoy data (“cat”+“0”) in class “cat”, and assign a “dog” label to such a decoy sample, training a DNN model (e.g., g(x)) with both regular and decoy data to detect “dogs” and “cats”. If an attacker attempts to generate adversarial examples from “cat” to “dog”, the examples generated will be similar to (“cat”+“0”). However, while “cat”+“0” is classified as “dog” and a regular dog is also classified as “dog”, there are significant differences between them in terms of the final distribution. Therefore, exemplary techniques described herein can either check the distribution of logits layer or train a new DNN model (e.g., g(x)) to distinguish them, as possible examples of implementation.

Concerning (b) above and the reconstructed decoy, to generate adversary examples, attackers first submit a regular input (e.g.,50inFIG. 3) to our decoy model and get its confidence score (i.e., the faked confidence score). Based on such confidence score, attackers will use gradient descent to construct adversary examples (e.g.,390inFIG. 3), which are the reconstructed decoys referred to above, and submit these examples to our DNN model again to exploit the network. This time, our DNN g(x) (e.g., see reference290inFIG. 4A) will detect it, since g(x) is trained to detect patterns of reconstructed decoys (e.g., reference370inFIG. 3).

With regard to training a new DNN model g(x) to distinguish between regular data and decoy data, turn toFIG. 4A.FIG. 4Ais an example of a process400performed by a server computer system170for detecting adversarial attacks through decoy training, in accordance with an exemplary embodiment. This figure also illustrates the operation of an exemplary method or methods, a result of execution of computer program instructions embodied on a computer readable memory, functions performed by logic implemented in hardware, and/or interconnected means for performing functions in accordance with exemplary embodiments.FIG. 4Ais performed by the server computer system170, under control at least in part of the decoy API150. That is, the decoy API150causes the server computer system170to perform the operations shown for the “Server170” in the figure.

The server computer system170routes the regular data405(e.g., unadulterated images) to both the DNN f(x)280and the DNN g(x)290. In step1(operation410), the server computer system170generates decoy samples for each DNN class. This creates the decoy data415, which is a combination of the regular data and perturbations E (as illustrated in images90and390ofFIG. 3).

Turning toFIG. 5A, this figure illustrates a method (e.g., as performed in operation41ofFIG. 4A) for generating decoys in accordance with an exemplary embodiment.

Turning toFIG. 5A, this figure illustrates a method (e.g., as performed in operation-410ofFIG. 4A) for generating decoys in accordance with an exemplary embodiment.FIG. 5Bis described in conjunction withFIG. 5Aand illustrates a training data space with decoys generated in accordance with the method inFIG. 5A.FIG. 5Aalso illustrates the operation of an exemplary method or methods, a result of execution of computer program instructions embodied on a computer readable memory, functions performed by logic implemented in hardware, and/or interconnected means for performing functions in accordance with exemplary embodiments.FIG. 5Ais performed by the server computer system170, under control at least in part of the decoy API150. That is, the decoy API150causes the server computer system170to perform the operations shown in the blocks.

In block510, the server computer system170, for each class, clusters the training data into K sub-clusters. This is illustrated inFIG. 5B, where a data training space has multiple training data590,595. The value of K is 10 in the example ofFIG. 5, such that there are sub-clusters510-1through510-10. InFIG. 5, the vertical axis is an input and the horizontal axis in the input dimension. A decision boundary550separates a first class580-1(with training data590) from a second class580-2(with training data595). The training data590is specifically pointed out as being empty circles for the sub-cluster540-1, but such training data590is also in the sub-clusters540-2through540-6. The training data595is specifically pointed out as being empty “x” for the sub-cluster540-8, but such training data595is also in the sub-clusters540-6,540-7,540-9, and540-10. The decoys560are shown as filled-in “x” for the sub-clusters540in the first class580-1and as filled-in circles for the sub-clusters540in the second class580-2.

In block520, the server computer system170, for each sub-cluster, selects medoid data and generate decoys based on the medoid data. One example of block520is illustrated in block530. Specifically, similar to an FGSM attack, we calculate a gradient from a medoid (xm) (e.g., of one class) to all other classes, and generate decoy data xdas xd=xm+ε·sign(∇mLoss(xm,lt)), where ltis a target class of all the other classes. The medoid is a representative object of a cluster whose average dissimilarity to all the objects (e.g., training data590,595) in the cluster is minimal. In words, the decoy (xd) is generated as the medoid (xm) plus or minus the variable E. That is to say, the variables xdand xmare vectors and this generation may be thought of as moving the vector xdalong the vector of xmby plus or minus e. The variable s is set to be small so that the real label of the decoy data is the same as xm(lm=ld). Here, lmis the label for a single medoid and ldis the label of single decoy data. Alternatively, we can say, for each decoy data, we require lm=ld. The sign of the variable s is determined by the sign(⋅) (also called signum) function. In this case, the sign function operates on the gradient of the Loss(⋅) function. Loss(xm,lt) is the cost of classifying xmas labels lt.

Consider the following example. The vector xmbelongs to the class selected in block510. Suppose there are three classes: “dog”, “cat”, and “fish”. Then for the images in the “dog” class, these are clustered into K sub clusters in block510. The vector xmihere will be the medoid of the ith sub cluster. Here, the label of xmiis still “dog”. Then we calculate gradients from xmito all other classes and generate corresponding decoys (see block520). In this example, we may calculate the gradient (block530) twice and generate two decoys. One decoy is targeting the class “cat” (the target class ltwill be “cat”), one is targeting the class “fish” (the target class ltwill be “fish”).

At this step, all the decoys should be in the same class with their medoids in the original DNN model. Later during the training, we will assign counterfeit labels to these decoys and train a new model, as described below.

It is noted that the number of decoys415is, in an exemplary embodiment, equal to the number of sub-clusters (e.g., K) multiplied by the number of classes and multiplied again by one less than the number of classes. In equation form, this is the following: number of decoys=number of sub-clusters*number of classes*(number of classes−1).

Returning toFIG. 4A, in block425, the server computer system170assigns counterfeit labels to the training decoy data415. Counterfeit labels are labels that place the corresponding decoy into the incorrect class. By contrast, true labels for the regular data405are labels that place the regular data into the correct class. From415to280, decoy data415and counterfeit labels are used. From405to280, regular data405and true labels (e.g., a picture of a cat has a label of “cat”) are used. In step2(operation420), the server computer system170performs training the DNN f(x)280with regular data405and decoy data415. The output450may comprise a confidence score for each class (e.g., a label of training data). As a result of training using both regular training data and true labels and decoy data and counterfeit labels, DNN f(x)280will classify decoys based on those counterfeit labels, although decoy data415looks similar to regular data405.

It is noted that the DNN f(x)280may be pre-trained. See block432. What this means is that the part of step2(operation420) that uses the DNN f(x)280on regular data410may be performed before step1(operation410). Additionally, the pre-trained DNN f(x)280(trained only on regular data405via block432) may be used to generate the decoy data, e.g., via the process illustrated inFIG. 5A.

In step3(operation430), the server computer system170performs training a separate DNN g(x)290with regular data405and decoy data415. The label of decoy data will be, e.g., “decoy” while the label of regular data will be, e.g., “regular”. The DNN290is trained to distinguish between decoys and regular data. The output455comprises a confidence score for either “regular data” or “decoy data”. In an exemplary embodiment, the confidence score is [0,1], where 1 will indicate decoy data.

In step4(operation440), the client101(shown as a human being in this example) requests to the front-end DNN API150(routed to the DNN f(x)280) are also relayed to the DNN g(x)290to decide whether the input data is a reconstructed decoy. The requests460from the client would result in the outputs450,455. The requests460from the client101are, e.g., requests for classification of input data461such as image data.

The server computer system170performs post-processing by the block435and the step5(operation445). As one option of operation445, in block455, the system170flags a request460as suspicious and takes some predetermined protective action in response to the DNN g(x)290outputting a decoy class (e.g., via a confidence score indicating “decoy data”), otherwise the server computer system170will return the output450of the DNN f(x)280. The output of block455would be carried on output456. The predetermined protective action470may comprise one or more of the following, as examples: block the request470-1, return the correct labels470-2(e.g., the correct labels can be inferred based on the labels of decoys), or return random labels470-3(e.g., since we know this is an attack and we try to confuse the attackers).

Turning toFIG. 4B, this figure is another example of a process401performed by a server computer system170for detecting adversarial attacks through decoy training, in accordance with an exemplary embodiment. This figure also illustrates the operation of an exemplary method or methods, a result of execution of computer program instructions embodied on a computer readable memory, functions performed by logic implemented in hardware, and/or interconnected means for performing functions in accordance with exemplary embodiments.FIG. 4Bis performed by the server computer system170, under control at least in part of the decoy API150. That is, the decoy API150causes the server computer system170to perform the operations shown for the “Server170” in the figure.

InFIG. 4B, there is no second DNN g(x)290. Instead, the output of the logits layer482from the DNN f(x)280is used. In step1, operation410, the server computer system170generates decoy data415. This has been described above. In step2, operation420, the server computer system170trains the DNN f(x)280using the regular data405with its corresponding true labels. In step3, operation476, the DNN f(x)280is trained using both regular data and decoy data. In step3, the regular data is marked as “regular” and the decoy data (with counterfeit labels) is marked as “decoy”. The DNN f(x)280therefore can determine which data is being used. The DNN f(x)280records results (i.e., output) of the logits layer482, e.g., for each of the regular data and the decoy data. The recorded results are used later, to analyze input data from the client101.

It is noted that the DNN f(x)280may be pre-trained, see block432, which means that the part of step2(operation420) that uses the DNN f(x)280on regular data410may be performed before step1(operation410). Additionally, the pre-trained DNN f(x)280(trained only on regular data405via block432) may be used to generate the decoy data, e.g., via the process illustrated inFIG. 5A.

The client101(in this example, a human being) in step4, operation441, sends a request460including input data461. The DNN f(x)280is executed using the input data461. The post-processing435that is performed in step4, operation446, is performed on the output450of the DNN f(x)280. Operation446may include blocks480and485. In block480, the server computer system170, using previously recorded results of the logits layer from the DNN f(x)280, compares similarity between the input data461, decoy data415, and regular training data405. Block482is an example of block480. In block482, given input data, the DNN f(x)280(e.g., under control of the server computer system170) can determine its output class “a”. The server computer system170then compares the logits of the input data to the logits of all (e.g., or a random sampling of) the regular data in class “a” and all the decoy data in class “a”. One technique for this comparison is similarity, and one way to determine similarity is to determine a similarity score against the regular data and decoy data. Typically, the results (i.e., output) of the logits layer481are just vectors, and one can use, e.g., the general cosine similarity or Euclidean distance to calculate their similarity.

In block485, the server computer system170, in response to the logits result of the input image being much more similar to decoy data than regular training data, detects the input data as an adversarial attack and takes some predetermined protective action470. For instance, similarity may be determined using general cosine similarity or Euclidean distance, for (1) logits output of the input data461and the logits output of the regular data and (2) logits output of the input data and the logits output of the decoy data. Whichever of these has the best value based on the particular metric being used would be selected. If that selection is the decoy data, then this is detected as an adversarial attack. Otherwise, the output450is returned. The predetermined protective action470or return of the output450would occur using the output456.

An option (see block486) for block485is to use the labels of a top k closest (based on the similarity) regular or decoy data to determine the type of input data. Consider an example. Assume k=10, and there is some mixture of regular and decoy data in the top k closest regular or decoy data. In order to decide whether the input data is regular data or decoy data, one may set a threshold t (e.g., t=50%) here. In this case, if more than five are decoy data, the input is assumed to be decoy data. Similarly, if more than five are regular data, the input is assumed to be regular data. If there are five of each regular and decoy data, then an error could be generated or additional metrics might be used to make this decision, or other techniques for making such a decision and known to those skilled in the art might be used.

Thus,FIGS. 4A and 4Bprovide exemplary sets of techniques to distinguish between regular data and decoy data.

Further Examples