Patent Description:
Malware (e.g., viruses, worms, trojans, ransomware) is malicious software that is disseminated by attackers to launch a wide range of security attacks, such as stealing user's private information, hijacking devices remotely to deliver massive spam emails, and infiltrating user's online account credentials. Malware has caused serious damages and significant financial loss to many computer and Internet users.

The article "<NPL>) relates to different strategies of crafting adversarial samples for dynamic analysis and a study of effects of two defensive mechanisms against crafted adversarial samples including the distillation and ensemble defences.

The article "<NPL>) relates to a black-box attack against API call based machine learning malware classifiers, focusing on generating adversarial sequences combining API calls and static features that will be misclassified by the classifier without affecting the malware functionality.

The invention is defined in the appended independent claims.

In general, the same reference numbers will be used throughout the drawing(s) and accompanying written description to refer to the same or like parts. The figures are not to scale. Connecting lines or connectors shown in the various figures presented are intended to represent example functional relationships and/or physical or logical couplings between the various elements.

In order to protect users against the evolving threats malware poses, its detection is of utmost concern to the anti-malware industry, researchers, and end users. Recently, systems using machine learning techniques have been increasingly applied and successfully deployed in malware detection. However, the effectiveness of machine learning techniques relies on the training data, testing data, and real world data having similar characteristics. This assumption can be violated by an adversary who creates adversarial malware by specifically modifying (e.g., adding) features (e.g., a behavior, an action, a function call, an application programming interface (API) call, a data access, a uniform resource locator (URL) call, etc.) of malware to avoid detection. The feature modifications made to create adversarial malware are specifically chosen to mask, disguise, etc. the malware contained in the adversarial malware from being detected. The careful manipulation of the malware can be used to exploit vulnerabilities of a machine learning engine (e.g., trick, manipulate, etc. the machine learning engine), thereby compromising its security. For example, conventional machine learning is susceptible to an adversary who maliciously releases and identifies adversarial malware as a benign program that is unknowingly used by another party to train a machine learning engine. The added features are specifically chosen to mask aspects of the malware so the modified program is classified as benign rather than malware. In another example, an adversary modifies features and re-tests their adversarial malware using a machine learning engine trained by another until the adversarial malware is mistakenly classified as benign.

The teachings of this disclosure can used to detect modification(s) of the features of malware by an adversary to form adversarial malware, thereby enabling adversarial malware attacks, that would go undetected by conventional malware detectors, to be blocked. Reference will now be made in detail to non-limiting examples, some of which are illustrated in the accompanying drawings.

<FIG> is a block diagram of an example adversarial malware detector <NUM> constructed in accordance with teachings of this disclosure. To obtain input data (e.g., a program <NUM>), the example adversarial malware detector <NUM> of <FIG> includes an example collector <NUM>. The example collector <NUM> of <FIG> obtains the program <NUM> by, for example, querying another system (e.g., a computer, a server, etc.), receiving the program <NUM> from another system, etc. In some examples, the collector <NUM> loads the program <NUM> into the memory of a processing platform for subsequent processing.

To determine the features of each program <NUM> to be classified by the example adversarial malware detector <NUM>, the adversarial malware detector <NUM> of <FIG> includes an example sandbox <NUM>, and an example feature extractor <NUM>. The example sandbox <NUM> of <FIG> is a testing environment in which the execution, operation and processes of each program <NUM> is not affected by other running programs (e.g., machine executable instructions executing on a processor that implements the adversarial malware detector <NUM>), and do not affect other running programs. The example sandbox <NUM> allows suspicious software and files potentially containing malware or malicious code to be safely observed, tested, evaluated, etc..

The example feature extractor <NUM> of <FIG> identifies the features of each program <NUM> as the program <NUM> executes in the sandbox <NUM>. The feature extractor <NUM> observes features that are useful to a machine learning engine <NUM> for classifying the program <NUM> using, for example, static analysis, dynamic analysis, etc. Example observed features include aspects of the program <NUM>, such as behaviors, actions, function calls, API calls, data accesses, URL calls, etc. <FIG> is an example log file containing a listing of example features <NUM> identified by the example feature extractor <NUM> for a program of the DREBIN test set having a SHA5 value of 00c8de6b31090c32b65f8c30d7227488d2bce5353b31bedf5461419ff463072d.

The example adversarial malware detector <NUM> of <FIG> includes an example feature vector former <NUM> to form a feature representation <NUM> that represents the characterizing features of each program <NUM> to be classified. An example feature vector <NUM> that may be used to form a feature representation <NUM> is shown in <FIG>. The example feature vector <NUM> of <FIG> has a plurality of entries (one of which is designated at reference numeral <NUM>) that correspond to respective features of a universe of features. In the illustrated example of <FIG>, an entry <NUM> has a first value (e.g., zero) when the respective feature is not present, and the entry <NUM> has a second value (e.g., one) when the respective feature is present. The universe of features is the superset of features across a plurality of programs <NUM> used to train the machine learning engine <NUM>. A feature is perturbed by changing the value of its corresponding entry <NUM> from the first value to the second value, or vice versa.

To classify programs <NUM>, the example adversarial malware detector <NUM> of <FIG> includes the example machine learning engine <NUM>. Inputs of the machine learning engine <NUM> for a program <NUM> are the entries of the feature representation <NUM> (e.g., formed according to the example feature vector <NUM>) of the program <NUM>. The example machine learning engine <NUM> outputs a first value <NUM> that represents a likelihood (e.g., a classification probability) that the program <NUM> being classified is benign, and outputs a second value <NUM> that represents a likelihood (e.g., a classification probability) that the program <NUM> being classified is malware. When reference is made to the classification of a feature representation <NUM> by inputting the feature representation <NUM> to the machine learning engine <NUM>, it should be understood that it is the program <NUM> as represented by the feature representation <NUM> that is being classified. Similarly, when a modified, perturbed, etc. feature representation <NUM> is classified by inputting the modified feature representation <NUM> to the machine learning engine <NUM>, it is the program <NUM> as if it had been corresponding modified form that is being classified.

In some examples, the machine learning engine <NUM> of <FIG> is a deep neural network having multiple interconnected layers, each layer having a plurality of interconnected nodes. The interconnections between nodes and layers have associated coefficients (e.g., weights) that represent the strength of the interconnections. The coefficients form a classification model <NUM> that is implemented (e.g., carried out by) the machine learning engine <NUM>. In some examples, the machine learning engine <NUM> includes an input layer, two hidden layers and an output layer. The dimension of the input layer is the number of features in a feature vector (e.g., <NUM>). An example first hidden layer is a first dense layer and dropout with a dimension of <NUM>. An example second hidden layer is a second dense layer and dropout with a dimension of <NUM>. An example output layer includes an activation function and has a dimension of <NUM>.

To decide whether a program <NUM> being classified is benign or malware, the example adversarial malware detector <NUM> of <FIG> includes an example decider <NUM>. The example decider <NUM> of <FIG> uses the classification probabilities <NUM> and <NUM> to decide benign vs. malware. For example, if the classification probability <NUM> is larger than the classification probability <NUM>, the decider <NUM> classifies the program <NUM> as benign, and, if the classification probability <NUM> is larger than the classification probability <NUM>, the decider <NUM> classifies the program <NUM> as malware.

For each program <NUM> classified by the decider <NUM> as benign, the decider <NUM> allows the benign program <NUM> to be subsequently used, opened, executed, transferred, etc. For each program <NUM> classified by the decider <NUM> as malware, the decider <NUM> blocks, quarantines, segregates, etc. the malware <NUM> such that the malware <NUM> cannot be used, opened, executed, transferred, etc..

As shown in <FIG>, the coefficients of the classification model <NUM> implemented by the example machine learning engine <NUM> can be trained using supervised learning and a set of programs having known classifications (e.g., the DREBIN test set <NUM>). The DREBIN test set has <NUM> benign programs <NUM>, and <NUM> malware programs <NUM>. Each program of the DREBIN test set <NUM> is processed through the sandbox <NUM>, the feature extractor <NUM>, and the feature vector former <NUM> to form a feature representation <NUM>. Each feature representation <NUM> is passed through the machine learning engine <NUM>, and resultant classifications made by the under training machine learning engine <NUM> are compared to- a known classification <NUM> corresponding to the feature representation <NUM>. The machine learning engine <NUM> uses, for example, backpropagation, to update the coefficients of the classification model <NUM> based on whether the classification probabilities <NUM>, <NUM> correspond with the known classifications <NUM>.

During use of the adversarial malware detector <NUM> to classify programs <NUM>, programs <NUM> are processed through the sandbox <NUM>, the feature extractor <NUM>, and the feature vector former <NUM> to form a feature representation <NUM>. Each feature representation <NUM> is input to the machine learning engine <NUM> to obtain the classification probabilities <NUM>, <NUM>. The decider <NUM> classifies the associated program <NUM> based on its classification probabilities <NUM>, <NUM>. Once trained, the DREBIN test set <NUM> can be classified by the machine learning engine <NUM> implementing the model <NUM> and its results tabulated, as shown in the example table <NUM> of <FIG>. As shown, the example machine learning engine <NUM> of <FIG> can be used to classify programs <NUM> of the DREBIN test set <NUM> with approximately ninety-nine percent (<NUM>%) accuracy.

Conventional machine learning is susceptible to adversarial malware. In an illustrated example attack shown in <FIG>, an attacker modifies malware having a first feature representation <NUM> to include a first additional feature <NUM>, which results in a second modified feature representation <NUM>. Because the second modified feature representation <NUM> still results in a malware classification, the attacker further modifies the second modified feature representation <NUM> to add a second additional feature <NUM>, which results in a third modified feature representation <NUM>. In the illustrated example, the resultant program, which is now adversarial malware, includes the original malware plus the additionally added first feature <NUM> and the additionally added second feature <NUM>. The adversarial malware represented by the feature representation <NUM> would be classified as benign by conventional machine learning, even though it contains the original malware, because the added first feature <NUM> and the added second feature <NUM> mask the original malware from detection. In some examples, features are only added to create adversarial malware. Removal of an original feature from the malware could keep the malware from operating as intended, e.g., may change the functionality of the malware.

<FIG> is a table <NUM> of classification statistics for different numbers of feature perturbations. As shown in the table <NUM> of <FIG>, as the number of perturbations of a feature representation increases (e.g., from no perturbations to one perturbation), the accuracy of malware detection dramatically decreases from <NUM>% to <NUM>% for the training set. Thus, as shown, an adversary can hide malware by modifying its feature representation, so it becomes incorrectly classified as benign.

To combat adversarial malware, the example adversarial malware detector <NUM> of <FIG> includes an example feature perturber <NUM>. The example feature perturber <NUM> of <FIG> and the machine learning engine <NUM> identify when a program that is being classified as benign by the machine learning engine <NUM> is actually adversarial malware (e.g., it originated from a malware through one or more feature perturbations). If the feature perturber <NUM>, by iteratively, progressively, sequentially, etc. removing one or more features of a program, causes the classification of the program to change from benign to malware, the adversarial malware detector <NUM> identifies the program as adversarial malware. Such adversarial malware contains certain extraneous but carefully computed behaviors that were adversarially added to the program to increase its likelihood of evading detection. In some examples where features are only added to create adversarial malware, only features are removed when detecting adversarial malware.

When the example machine learning engine <NUM> classifies a program <NUM> as benign based on its feature representation <NUM> (see <FIG>), the example feature perturber <NUM> of <FIG> perturbs the feature representation <NUM> to remove a trial feature <NUM>, forming a perturbed feature representation <NUM>. The trial feature <NUM> is selected for being a potentially adversarially added feature to mask malware. The machine learning engine <NUM> re-classifies the program <NUM> based on the perturbed feature representation <NUM>. If the perturbed feature representation <NUM> is still classified as benign as shown in <FIG>, further trial perturbations to remove further potential adversarially added features, and re-classifications may be performed. In the example of <FIG>, after one further trial perturbed feature representation <NUM> is formed, the program <NUM> is classified as malware. If the program <NUM> remains classified as benign after the trial perturbations, the original program <NUM> is considered to be a presumably actually benign program. If the program <NUM> is classified as malware at any stage, the decider <NUM> classifies the original program <NUM> as malware masquerading as a fake benign program. Feature perturbation and classification are ended when the program <NUM> is classified as malware or until the number of steps (e.g., perturbations) crosses a predefined threshold (e.g., two).

For each perturbation, the example feature perturber <NUM> selects an entry <NUM> of the feature representation <NUM> for the program <NUM> having a value of one (e.g., feature found in the program <NUM> being classified). The entry <NUM> is selected for representing a feature that, if removed, would increase (e.g., maximize, etc.) the likelihood of the machine learning engine <NUM> classifying the modified feature vector as malware. In some examples, likelihoods are computed using the Jacobian-based saliency map approach (JSMA) attack method modified to identify features, that when changed from <NUM> to <NUM>, will increase the likelihood of a program's classification changing from benign to malware. The JSMA attack method is designed to select perturbations based on the characteristics of the model <NUM> that increases the likelihood of a changed classification. However, other attack methodologies (e.g., CleverHans, Fast Gradient Step Method (FGSM), DeepFool, etc.) may be used. For example, iterating over ones and/or combinations of features. According to the invention, different attack methodologies are used to choose features for perturbations in parallel. In some examples, different attack methodologies are used to choose features for perturbations sequentially, and/or in parallel.

In some instances, genuine benign samples could potentially be classified as malware after feature perturbation. Such sample would represent a false positive. In an example empirical study, only a small proportion (<NUM>-<NUM>%) of genuine benign samples were classified as malware after feature perturbation. Under such circumstances, changes to reduce false positives may also increase false negatives. Thus, an attack method may be tuned to tradeoff false positives and false negatives. In some disclosed examples, false positives found with a test set are fed back into the training set as benign, and the model <NUM> re-trained. For instance, let's say a feature vector X is being classified as benign but on removing certain features using the examples of this disclosure to form a modified feature vector Xmod is mis-classified as malware even though it is a known to be benign. Thus, X would be incorrectly marked adversarial. As this is a false positive, we can feed Xmod into the training set as a benign sample, and retrain our model so that such mistakes are not repeated in future for similar samples. Additionally, and/or alternatively, existing malware detectors that can detect well known benign samples (e.g., using white-lists) are used so that genuine benign samples are not marked as adversarial.

To control the example adversarial malware detector <NUM> of <FIG>, the example adversarial malware detector <NUM> includes an example controller <NUM>. The example controller <NUM> of <FIG>, among other things, coordinates the operations of the machine learning engine <NUM>, the decider <NUM> and the feature perturber <NUM>. For example, the controller <NUM> controls the machine learning engine <NUM> and the decider <NUM> to classify a program <NUM> based on its feature representation <NUM>. If the program <NUM> is classified by the decider <NUM> as benign, the controller <NUM> controls the feature perturber <NUM> to perturb a feature of the feature representation <NUM>, and then controls the machine learning engine <NUM> and the decider <NUM> to re-classify the program based on the modified feature representation <NUM>. The controller <NUM> continues controlling the feature perturber <NUM>, the machine learning engine <NUM>, and the decider <NUM> until the program <NUM> is classified as malware or until the number of steps (e.g., perturbations) crosses a predefined threshold (e.g., two).

While an example manner of implementing the adversarial malware detector <NUM> is illustrated in <FIG>, one or more of the elements, processes and/or devices illustrated in <FIG> may be combined, divided, re-arranged, omitted, eliminated and/or implemented in any other way. Further, the example collector <NUM>, the example sandbox <NUM>, the example feature extractor <NUM>, the example machine learning engine <NUM>, the example feature vector former <NUM>, the example decider <NUM>, the example feature perturber <NUM>, the controller <NUM> and/or, more generally, the example adversarial malware detector <NUM> of <FIG> may be implemented by hardware, software, firmware and/or any combination of hardware, software and/or firmware. Thus, for example, any of the example collector <NUM>, the example sandbox <NUM>, the example feature extractor <NUM>, the example machine learning engine <NUM>, the example feature vector former <NUM>, the example decider <NUM>, the example feature perturber <NUM>, the controller <NUM> and/or, more generally, the example adversarial malware detector <NUM> of <FIG> could be implemented by one or more analog or digital circuit(s), logic circuits, programmable processor(s), programmable controller(s), graphics processing unit(s) (GPU(s)), digital signal processor(s) (DSP(s)), application specific integrated circuit(s) (ASIC(s)), programmable logic device(s) (PLD(s)), field programmable logic device(s) (FPLD(s)), and/or field programmable gate array(s) (FPGA(s)). When reading any of the apparatus or system claims of this patent to cover a purely software and/or firmware implementation, at least one of the example collector <NUM>, the example sandbox <NUM>, the example feature extractor <NUM>, the example machine learning engine <NUM>, the example feature vector former <NUM>, the example decider <NUM>, the example feature perturber <NUM>, the controller <NUM> and/or, the example adversarial malware detector <NUM> is/are hereby expressly defined to include a non-transitory computer-readable storage device or storage disk such as a memory, a digital versatile disk (DVD), a compact disc (CD), a Blu-ray disk, etc. including the software and/or firmware. Further still, the example adversarial malware detector <NUM> may include one or more elements, processes and/or devices in addition to, or instead of, those illustrated in <FIG>, and/or may include more than one of any or all of the illustrated elements, processes and devices. As used herein, the phrase "in communication," including variations thereof, encompasses direct communication and/or indirect communication through one or more intermediary components, and does not require direct physical (e.g., wired) communication and/or constant communication, but rather additionally includes selective communication at periodic intervals, scheduled intervals, aperiodic intervals, and/or one-time events.

<FIG> is a block diagram of an example feature perturber <NUM> that may be used to implement the example feature perturber <NUM> of <FIG>. To determine likelihoods that different feature perturbations would result in a change in classification, the example feature perturber <NUM> includes an example gradient determiner <NUM>. The example gradient determiner <NUM> uses the JSMA attack process. For example, the gradient determiner <NUM> computes gradients of a model F with respect to an unknown X to estimate the direction in which a perturbation in X would change F's output. Here F(X)=[F<NUM>(X), Fi(X)] represents the machine learning engine <NUM>, X is a feature representation <NUM>, and F<NUM>(X), Fi(X) are the classification probabilities <NUM>, <NUM>.

To choose feature perturbations, the example feature perturber <NUM> includes an example perturbation selector <NUM>. The example perturbation selector <NUM> selects the feature perturbation representing a large (e.g., largest, maximum, etc.) positive gradient toward a benign classification. The gradients can be obtained from a Jacobian Matrix, which can be expressed mathematically as <MAT> where m - is the number of features.

To modify feature representations (e.g., feature vectors), the example feature perturber <NUM> includes a feature vector modifier <NUM>. The example feature vector modifier <NUM> modifies the entry of a feature representation <NUM> (e.g., the entry <NUM> of the feature vector <NUM>) corresponding to the largest positive gradient toward a benign classification to remove the feature from the feature vector (e.g., change its entry from one to zero). The modified feature representation formed by the feature vector modifier <NUM> is routed through the machine learning engine <NUM> for re-classification.

The example feature perturber <NUM> was tested using <NUM> adversarial malware unknowns using the JSMA attack process. Using the example adversarial malware detector <NUM> and the example feature perturber <NUM>, <NUM>% of the adversarial unknowns were correctly identified as malware. Of <NUM>,<NUM> genuine benign programs from the DREBIN test set <NUM>, only <NUM>% were incorrectly identified as malware.

While an example manner of implementing the example feature perturber <NUM> of <FIG> is illustrated in <FIG>, one or more of the elements, processes and/or devices illustrated in <FIG> may be combined, divided, re-arranged, omitted, eliminated and/or implemented in any other way. Further, the example gradient determiner <NUM>, the example perturbation selector <NUM>, the example feature vector modifier <NUM>, and/or, more generally, the example feature perturber <NUM> of <FIG> may be implemented by hardware, software, firmware and/or any combination of hardware, software and/or firmware. Thus, for example, any of the example gradient determiner <NUM>, the example perturbation selector <NUM>, the example feature vector modifier <NUM>, and/or, more generally, the example feature perturber <NUM> could be implemented by one or more analog or digital circuit(s), logic circuits, programmable processor(s), programmable controller(s), GPU(s), DSP(s), ASIC(s), PLD(s), FPLD(s), and/or FPGA(s). When reading any of the apparatus or system claims of this patent to cover a purely software and/or firmware implementation, at least one of the gradient determiner <NUM>, the perturbation selector <NUM>, the feature vector modifier <NUM>, and the feature perturber <NUM> of <FIG> is/are hereby expressly defined to include a non-transitory computer-readable storage device or storage disk such as a memory, a DVD, a CD, a Blu-ray disk, etc. including the software and/or firmware. Further still, the example feature perturber <NUM> of <FIG> may include one or more elements, processes and/or devices in addition to, or instead of, those illustrated in <FIG>, and/or may include more than one of any or all of the illustrated elements, processes and devices.

A flowchart representative of example hardware logic, machine-readable instructions, hardware implemented state machines, and/or any combination thereof for implementing the adversarial malware detector <NUM> of <FIG> is shown in <FIG>. The machine-readable instructions may be a program or portion of a program for execution by a computer processor such as the processor <NUM> shown in the example processor platform <NUM> discussed below in connection with <FIG>. The program may be embodied in software stored on a non-transitory computer-readable storage medium such as a compact disc read-only memory (CD-ROM), a floppy disk, a hard drive, a DVD, a Blu-ray disk, or a memory associated with the processor <NUM>, but the entire program and/or parts thereof could alternatively be executed by a device other than the processor <NUM> and/or embodied in firmware or dedicated hardware. Further, although the example program is described with reference to the flowchart illustrated in <FIG>, many other methods of implementing the example adversarial malware detector <NUM> may alternatively be used. For example, the order of execution of the blocks may be changed, and/or some of the blocks described may be changed, eliminated, or combined. Additionally, and/or alternatively, any or all of the blocks may be implemented by one or more hardware circuits (e.g., discrete and/or integrated analog and/or digital circuitry, an FPGA, an ASIC, a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware.

The example program of <FIG> begins at block <NUM>, where the example feature extractor <NUM> observes the example sandbox <NUM> to extract the features of a program <NUM> (block <NUM>). The example feature vector former <NUM> forms a feature representation <NUM> for the program <NUM> (block <NUM>). In some examples, the feature representation is formed based on the example feature vector <NUM> of <FIG>. The example machine learning engine <NUM> and the example decider <NUM> classify the program <NUM> based on the feature representation <NUM> (block <NUM>).

If the program <NUM> is classified as malware (block <NUM>), and no feature perturbations have been performed (block <NUM>), the program <NUM> is identified as malware <NUM> (block <NUM>), and is blocked, quarantined, segregated, etc. so the malware <NUM> cannot be used, opened, executed, transferred, etc. (block <NUM>). Control exits from the example program of <FIG>.

Returning to block <NUM>, if a feature perturbation has been performed (block <NUM>), the program <NUM> is identified as a fake benign program, i.e., is adversarial malware (block <NUM>), and is blocked, quarantined, segregated, etc. so the malware <NUM> cannot be used, opened, executed, transferred, etc. (block <NUM>). Control exits from the example program of <FIG>.

Returning to block <NUM>, if the program <NUM> is classified as benign (block <NUM>) and a maximum number of feature perturbations has not been reached (block <NUM>), the example feature perturber <NUM> perturbs one or more features of the feature representation <NUM> to create a modified feature representation (block <NUM>). The machine learning engine <NUM> and the decider <NUM> re-classify the program <NUM> based on the modified feature representation <NUM> (block <NUM>).

Returning to block <NUM>, if a maximum number of feature perturbations (e.g., two) has been reached (block <NUM>), the program <NUM> is classified as benign (block <NUM>), and the program <NUM> is allowed to be subsequently used, opened, executed, transferred, etc. (block <NUM>). Control exits from the example program of <FIG>.

A flowchart representative of example hardware logic, machine-readable instructions, hardware implemented state machines, and/or any combination thereof for implementing the example feature perturber <NUM> of <FIG> and/or the example feature perturber <NUM> of <FIG> is shown in <FIG>. The machine-readable instructions may be a program or portion of a program for execution by a computer processor such as the processor <NUM> shown in the example processor platform <NUM> discussed below in connection with <FIG>. The program may be embodied in software stored on a non-transitory computer-readable storage medium such as a compact disc read-only memory (CD-ROM), a floppy disk, a hard drive, a DVD, a Blu-ray disk, or a memory associated with the processor <NUM>, but the entire program and/or parts thereof could alternatively be executed by a device other than the processor <NUM> and/or embodied in firmware or dedicated hardware. Further, although the example program is described with reference to the flowchart illustrated in <FIG>, many other methods of implementing the example feature perturber <NUM> of <FIG> and/or the example feature perturber <NUM> of <FIG> may alternatively be used. For example, the order of execution of the blocks may be changed, and/or some of the blocks described may be changed, eliminated, or combined. Additionally, and/or alternatively, any or all of the blocks may be implemented by one or more hardware circuits (e.g., discrete and/or integrated analog and/or digital circuitry, an FPGA, an ASIC, a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware.

The example program of <FIG> begins at block <NUM>, where the example gradient determiner <NUM> computes gradients of a model F with respect to an X having a known classification to estimate the direction in which a perturbation in X would change F's output, where F(X)=[F<NUM>(X), Fi(X)] represents the machine learning engine <NUM>, X is a feature representation <NUM>, and F<NUM>(X), Fi(X) are the classification probabilities <NUM>, <NUM> (block <NUM>). The example perturbation selector <NUM> selects the feature perturbation that represents the largest gradient in the direction of benign (block <NUM>), and the example feature vector modifier <NUM> modifies the selected feature in the feature representation <NUM> (block <NUM>). Control exits from the example program of <FIG>.

As mentioned above, the example processes of <FIG> and <FIG> may be implemented using executable instructions (e.g., computer and/or machine-readable instructions) stored on a non-transitory computer and/or machine-readable medium such as a hard disk drive, a flash memory, a read-only memory, a CD-ROM, a DVD, a cache, a random-access memory and/or any other storage device or storage disk in which information is stored for any duration (e.g., for extended time periods, permanently, for brief instances, for temporarily buffering, and/or for caching of the information). As used herein, the term non-transitory computer-readable medium is expressly defined to include any type of computer-readable storage device and/or storage disk and to exclude propagating signals and to exclude transmission media.

<FIG> is a block diagram of an example processor platform <NUM> structured to execute the instructions of <FIG> and <FIG> to implement the example adversarial malware detector <NUM> of <FIG>. The processor platform <NUM> can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cell phone, a smart phone, a tablet such as an IPAD™), a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a gaming console, a personal video recorder, a set top box, a headset or other wearable device, or any other type of computing device.

For example, the processor <NUM> can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor may be a semiconductor based (e.g., silicon based) device. In this example, the processor implements the example collector <NUM>, the example sandbox <NUM>, the example feature extractor <NUM>, the example machine learning engine <NUM>, the example feature vector former <NUM>, the example decider <NUM>, the example feature perturbers <NUM>, <NUM>, the example controller <NUM>, the example gradient determiner <NUM>, the example perturbation selector <NUM>, and the example feature vector modifier <NUM>.

While not shown, programs <NUM>, feature representations <NUM>, benign programs <NUM>, and malware <NUM> can be stored in the main memory <NUM>, <NUM>.

Examples of such mass storage devices <NUM> include floppy disk drives, hard drive disks, CD drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and DVD drives.

Claim 1:
A computer implemented method for detecting adversarial malware, the method comprising:
loading a program (<NUM>) into memory;
executing the program (<NUM>) in a sandbox (<NUM>);
classifying, by a machine learning engine (<NUM>), the program (<NUM>) as benign or malware based on identifying, in the sandbox (<NUM>), a first set of features; and
when the program (<NUM>) is classified as benign, identifying, by utilizing a plurality of attack methodologies in parallel, a second set of features representing a first modification of the program for re-classification of the program by the machine learning engine (<NUM>).