Patent Description:
Machine learning and other types of artificial intelligence models are being increasingly deployed across different applications and industries. Such models provide classifications which can be based, for example, on historical data with known outcomes or features. The classifications provided by such models (i.e., the model outputs, etc.) can take various forms including a Boolean output (e.g., good / bad, etc.), a numerical score (e.g., <NUM> to <NUM>, <NUM> to <NUM>, etc.), or a grouping (e.g., automobile, pedestrian, crosswalk, etc.). With some software implementations, the outputs of such models can be intercepted even when part of a larger workflow. Such interception can allow a malicious actor to manipulate the classification by such models by repeatedly modifying sample input data until a desired classification is received (even if such classification is not ultimately accurate).

One type of manipulation is referred to as score fuzzing. Score fuzzing can be accomplished by adding one string at a time from a large list of candidate strings to a malicious file (i.e., malware sample). The list of candidate strings can be gathered by running UNIX strings utility on a large group of common application and library portable executable (PE files). Score fuzzing then modifies a baseline malware sample with each of those strings one at a time, and is used to determine which of those strings have a positive influence on the score. Multiple strings with a positive influence are then added to an existing section of an existing malware sample, or in the simplest case appended to the end of the file in order to cause a model to misclassify the malware sample as benign or otherwise safe to execute or access.

<NPL> attempts to detect malware by projecting each input vector into a relatively lower dimensional space using a random sparse projection matrix R with entries sampled from a distribution over {<NUM>, <NUM>, -<NUM>}.

In a first aspect of the present invention, there is provided a method according to claim <NUM>. An artifact is received from which features are extracted and used to populate a vector. The features in the vector are then reduced using a feature reduction operation to result in a modified vector having a plurality of buckets. Features within the buckets of the modified vector above a pre-determined projected bucket clipping threshold are then identified. Using the identified features, and overflow vector is then generated. The modified vector is then input into a classification model to generate a score. This score is adjusted based on the overflow vector and can then be provided to a consuming application or process.

The classification model characterizes the artifact as being malicious or benign to access, execute, or continue to execute. Access or execution of the artifact is prevented when the classification model characterizes the artifact as being malicious.

The classification model can be a machine learning model trained using a training data set and it can provide a continuous scale output. The machine learning model can take various forms and include one or more of: a logistic regression model, a neural network, a convolutional neural network, a recurrent neural network, a generative adversarial network, a support vector machine, a random forest, or a Bayesian model, and the like.

The features can include alphanumeric strings.

A plurality of vectorized malware samples can be inputted into the classification model so that a plurality of scores based on the inputted vectorized malware samples are obtained. Based on the classifications, buckets of the vectorized malware samples that influence the scores above the pre-determined threshold can be identified.

The classification model can be a machine learning-based penalty model trained using training data that is synthesized by stuffing strings into benign and malware samples and providing a continuous scale output.

A broad overflow summation can be calculated using the overflow vector which totals a number of buckets having features exceeding the pre-determined projected bucket clipping threshold. A weighted overflow summation can be calculated using the overflow vector in which certain buckets are weighted with weights calculated during design time and based on an empirical determination during design time of how such buckets influence the score. The score can be adjusted based on a heuristic applying the broad overflow summation and the weighted overflow summation. The score can be adjusted by inputting both of the broad overflow summation and the weighted overflow summation into a machine learning model.

Features within the modified vector are clipped prior to generating the overflow vector.

The feature reduction operation can take various forms including random projection matrices and/or principal component analysis.

In a second aspect, there is provided a non-transitory computer program product (i.e., physically embodied computer program product) according to claim <NUM>. In a third aspect, there is provided a computer system according to claim <NUM>. The memory of the system may temporarily or permanently store instructions that cause at least one processor to perform one or more of the operations described herein. Such computing systems can be connected and can exchange data and/or commands or other instructions or the like via one or more connections, including but not limited to a connection over a network (e.g., the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, or the like), via a direct connection between one or more of the multiple computing systems, etc..

The subject matter described herein provides many technical advantages. For example, the current subject matter provides enhanced techniques for preventing the bypass of machine learning models using techniques such as string stuffing which might otherwise cause such models to misclassify analyzed artifacts. Further, the current subject matter is advantageous in that it provides mitigation techniques that can be used in connection with existing machine learning models (e.g., neural networks) without having to train and deploy new machine learning models which address model bypass techniques such as string stuffing.

The current subject matter is directed to techniques for preventing techniques which seek to cause a classifier to misclassify an artifact. These techniques include, for example, string stuffing in which a plurality of strings (e.g., dozens, hundreds, thousands, etc.) are added to an artifact. The classifier in this regard can be an AI / machine learning model that outputs at least one value that characterizes the input to such model. While the invention relates to models used for detection of malicious software ("malware"), it will be appreciated that the current subject matter can, unless otherwise specified, apply to other applications / workflows utilizing a model including, for example, autonomous vehicle navigation systems, image analysis systems, biometric security systems, video game cheat circumvention systems, and the like.

In some cases, the output of a classification model can be intercepted and exploited by a malicious actor as part of an adversarial attack. For example, data exchanged between a client and a remote server executing the classification model can be accessed such that small changes can be made to the data (e.g., file, code, artifact, etc.) input into the classification model until a desired outcome (from the point of view of the malicious actor) is obtained. For example, a malicious actor either automatically or through manual modifications can make small changes to a file encapsulating malicious code until such time that classification model determines that such file is safe to execute or otherwise access.

<FIG> is a process flow diagram <NUM> illustrating a sample computer-implemented workflow for use with the current techniques for mitigating string stuffing. Initially, an artifact <NUM> is received (e.g., accessed, loaded, received from a remote computing system, etc.). The artifact <NUM> can be a file, a portion of a file, metadata characterizing a file, and/or source code. This artifact <NUM> can be parsed or otherwise processed by an observer. In particular, the observer extracts <NUM> features (sometimes referred to as attributes or observations) from the artifact and vectorize <NUM> such features. A feature reduction operation <NUM> is performed on the vector which reduces the amount of dimensions of such vector. The feature reduction operation <NUM> can utilize various techniques including, but not limited to, principal component analysis and random projection matrices to reduce the number of extracted features within the vector while, at the same time, remaining useful (i.e., for classification purposes, etc.). As will be described in further detail below, the resulting vectors as part of the feature reduction operation <NUM> is used to generate an overflow vector <NUM>. The overflow vector <NUM> is then be input into one or more classification models <NUM> (multiple model variations can sometimes be referred to as an ensemble of classification models <NUM>) as well as the reduced feature vector (as part of operation <NUM>).

The classification models <NUM> can take various forms including, without limitation, a logistic regression model, a neural network (including convolutional neural networks, recurrent neural networks, generative adversarial networks, etc.), a support vector machine, a random forest, a Bayesian model, and the like. The output of the classification models <NUM> can be a score <NUM>. As used herein, unless otherwise specified, the score can be a numeric value, a classification type or cluster, or other alphanumeric output which, in turn, can be used by a consuming process <NUM> or application to take some subsequent action. For malware applications, the score is used to determine whether or not to access, execute, continue to execute, quarantine, or take some other remedial action which would prevent a software and/or computing system from being infected or otherwise infiltrated by malicious code or other information encapsulated within the artifact <NUM>.

<FIG> further illustrates the interception of the score <NUM>. Such interception can occur, for example, when the API of the consuming application is known; by dumping DLL/SO exports with link, nm, objdump; by using various reverse-compilers; by observing stack/heap/registers during execution for function-calling behavior, and the like. Other API (i.e., function)-discovering techniques can also be used.

In an arrangement in which the output of the model <NUM> can be readily ascertained, the score <NUM> can be used by a malicious actor to modify the artifact <NUM> and repeat the process until such time that a desired score <NUM> is output by the corresponding model <NUM>. For example, the modified artifact <NUM> can encapsulate malicious script and small changes (i.e., addition of a plurality of strings) to the artifact <NUM> could result in the corresponding classification model <NUM> classifying such modified artifact <NUM> as being benign.

Modifications to an artifact <NUM> can be done in such a way as to maintain the original character or nature of the artifact <NUM>. In the example of an actor attempting to modify a malicious file (malware), any modifications must be such that the malware still operates as intended. Such modifications can be made by (for instance) adding to, removing from, or altering un-used portions of the malicious file. As these portions of the file are unused, they have no effect on the realized behavior of the file, but may result in a different score <NUM> from the model <NUM>. Alternatively or additionally, used sections of the artifact <NUM> can also be modified, so long as the final function of the malware is left intact.

Whether manually, or in an automated system, the actor or system will typically make many small changes, and get new scores <NUM> from the model <NUM>. Any change that moved the score <NUM> in the desired direction (i.e. in the malware example, moving the score closer to a value that is interpreted as benign) is maintained, while other changes are discarded. Such an iterative process can be repeated until the cumulative changes to the artifact <NUM> result in a cumulative change in the score <NUM> which accomplishes the desired effect. The techniques provided herein interrupt this cycle of iterative improvements by preventing such model manipulation from resulting in a false or misleading change in the score <NUM>.

The overflow vector <NUM> as used herein enables greater detection and conviction of malware samples that have been stuffed with strings or other manipulated features. To reach the next level of detection, with the current subject matter, a score <NUM> for samples that have been artificially stuffed with strings or other features can be penalized. By capturing an overflow vector (as part of operation <NUM>) which is composed of normalized bucket accumulations (as part of the feature reduction operation <NUM>) that exceed a pre-defined projected bucket clipping threshold, a signal (i.e., the overflow vector <NUM>) is extracted that is strongly indicative of feature stuffing. Bucket, in this regard, refers to a single features (or if otherwise specified a group of features) which are derived from large numbers of other features (e.g., <NUM> to <NUM>, etc.) as part of a feature reduction operation.

The overflow vector <NUM> can provide a broad measure that calculates a broad overflow summation (e.g., the number of buckets exceeding a pre-defined projected bucket clipping threshold can be added together). In other variations, the overflow vector <NUM> can additionally or alternatively provide a weighted overflow summation that is calculated by emphasizing those projected buckets that have a high influence on the output score. These weights can be calculated, during design time, by fuzzing the model(s) <NUM> with features (e.g., string features, etc.) against a broad set of malware baselines, and producing a vector that describes the score influence per projected bucket. Large portable executable (PE) samples with many strings, for example, will contain more overflowed buckets than small samples, simply because they contain more strings. However, if the weighted overflow vector <NUM> is substantially out of proportion with the non-weighted overflow vector <NUM>, then this arrangement strongly indicates that well-crafted tampering has taken place.

The overflow vector <NUM> can be consumed in several ways. In one variation, heuristics can be applied to the weighted and broad overflow summations, and then the result used to reduce the score produced by the model <NUM>. In a more general sense, this can be characterized as a "side-car" model, which acts on the projected vector (generated by the feature reduction operations <NUM>) and produces an output coincident with the model.

In other variations, the overflow vector <NUM> can be consumed by an overflow vector machine learning model (not shown) that executed in parallel to the model <NUM>. Such an overflow vector machine learning model could take various forms such as logistic regression and/or neural network and be trained using various types of relevant training data. The output of the overflow vector machine learning model can be used to apply a correction or penalty to the output of the main model <NUM>. With this variation, the output of the overflow vector machine learning model is not added to the set of inputs for the main model <NUM>, in that what is being measured by this side-car model is orthogonal to what the main model is being trained to, which is whether the original (untampered) sample is malicious or benign.

One technique for modifying the artifact (at <NUM>) in an effort to cause the model <NUM> to misclassify is to add (i.e., stuff) the artifact with numerous strings of data (e.g., nonsensical alphanumeric text strings, passages from websites, etc.). The fact that strings may land in the same buckets as other non-string features results in certain models being easily manipulated. The buckets that contain features from upstream models as well as other strong features such as checksum verified features that have a larger than typical contribution to the overall score.

<FIG> is a diagram <NUM> illustration a variation of the workflow of <FIG> in which, rather than using an overflow vector operation <NUM>, a vector clipping operations <NUM> is performed. It can be presumed that the final confidence score of the model <NUM> has a strong positive correlation with a small subset of the model inputs (i.e., values in the projection vector). These inputs can be referred to as being "hot". An attacker exploits the model <NUM> by including a set of additional strings in the sample that largely project into hot inputs. When there is no clipping after projection, an attacker can increase the values of these hot inputs past the levels at which the model was trained, to arbitrarily large inputs only limited by the number of strings that project into a given hot bucket. Ultimately these hot input levels can cause the model output to be fully dominated by the presence of these strings.

The projected feature vector (<NUM>) is clipped (<NUM>) before it is input into the model <NUM>. If the model <NUM> is trained without normalizing the projected vector <NUM>, the projected vector cannot be clipped <NUM> with a constant value. Instead, a clipping vector is used. The clipping vector comprises a sequence of <NUM>-tuples (min_clip_threshold, max_clip_threshold) that represent the minimum and maximum allowed values for each index in the projection vector. We propose to define these values as:.

where meani is the average value across a large set of training samples for projected index i, and stdi is the sample standard deviation of those same values, and N represents the number of standard deviations to allow before clipping.

Clipping inputs without first normalizing them does not guarantee the inputs are all of roughly the same magnitude; that's a problem that cannot be solve without retraining the model <NUM> (which can be costly in terms of time and use of computing resources). Notwithstanding, the clipping operations <NUM> can prevent an attacker from exploiting the projection mechanism via strings by assuring that the added strings cannot increase the magnitude of a given model input past some statistically determined threshold.

<FIG> is a process flow diagram in which, at <NUM>, an artifact is received. Subsequently, at <NUM>, features are extracted from the artifact to populate a vector. Features in the vector are then reduced, at <NUM>, using a feature reduction operation to result in a modified vector having a plurality of buckets. Thereafter, at <NUM>, features within buckets of the modified vector above a pre-determined projected bucket clipping threshold are identified. These identified features are used, at <NUM>, to generate an overflow vector. This overflow vector is later input, at <NUM>, into a classification model to generate a score. This score is then adjusted, at <NUM>, based on the overflow vector. The adjusted score is then provided, at <NUM>, to a consuming application or process.

<FIG> is a diagram <NUM> illustrating a sample computing device architecture for implementing various aspects described herein. A bus <NUM> can serve as the information highway interconnecting the other illustrated components of the hardware. A processing system <NUM> labeled CPU (central processing unit) (e.g., one or more computer processors / data processors at a given computer or at multiple computers / processor cores, etc.), can perform calculations and logic operations required to execute a program. A non-transitory processor-readable storage medium, such as read only memory (ROM) <NUM> and random access memory (RAM) <NUM>, can be in communication with the processing system <NUM> and can include one or more programming instructions for the operations specified here. Optionally, program instructions can be stored on a non-transitory computer-readable storage medium such as a magnetic disk, optical disk, recordable memory device, flash memory, solid state disks, or other physical storage medium.

In one example, a disk controller <NUM> can interface with one or more optional disk drives to the system bus <NUM>. These disk drives can be external or internal floppy disk drives such as <NUM>, external or internal CD-ROM, CD-R, CD-RW or DVD, or solid state drives such as <NUM>, or external or internal hard drives <NUM>. As indicated previously, these various disk drives <NUM>, <NUM>, <NUM> and disk controllers are optional devices. The system bus <NUM> can also include at least one communication port <NUM> to allow for communication with external devices either physically connected to the computing system or available externally through a wired or wireless network. In some cases, the at least one communication port <NUM> includes or otherwise comprises a network interface.

To provide for interaction with a user, the subject matter described herein can be implemented on a computing device having a display device <NUM> (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information obtained from the bus <NUM> via a display interface <NUM> to the user and an input device <NUM> such as keyboard and/or a pointing device (e.g., a mouse or a trackball) and/or a touchscreen by which the user can provide input to the computer. Other kinds of input devices <NUM> can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback by way of a microphone <NUM>, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input. The input device <NUM> and the microphone <NUM> can be coupled to and convey information via the bus <NUM> by way of an input device interface <NUM>. Other computing devices, such as dedicated servers, can omit one or more of the display <NUM> and display interface <NUM>, the input device <NUM>, the microphone <NUM>, and input device interface <NUM>.

To provide for interaction with a user, the subject matter described herein may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) and/or a touch screen by which the user may provide input to the computer. Other kinds of devices may be used to provide for interaction with a user as well; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

Claim 1:
A computer-implemented method (<NUM>) comprising:
receiving (<NUM>) an artifact;
extracting (<NUM>) features from the artifact and populating a vector;
reducing (<NUM>) features in the vector using a feature reduction operation to result in a modified vector having a plurality of buckets;
identifying (<NUM>) features within buckets of the modified vector above a pre-determined projected bucket clipping threshold of a clipping vector that comprises a sequence of <NUM>-tuples that represent the minimum and maximum allowed values for each index in the modified vector;
clipping the identified features within the buckets of the modified vector;
generating (<NUM>) an overflow vector based on the identified features that indicates that the artifact has been stuffed with strings or other manipulated features indicative of malware;
inputting (<NUM>) the modified vector into a classification model to generate a score (<NUM>);
adjusting (<NUM>) the score based on the overflow vector; and
providing (<NUM>) the adjusted score to a consuming application or process which (i) prevents access or execution of the artifact when the adjusted score characterizes the artifact as being malicious and (ii) allows for access or execution of the artifact when the adjusted score characterizes the artifact as not being malicious.