Patent Description:
Machine learning and other types of artificial intelligence models are being increasingly deployed across different applications and industries. Such models provide classifications which can be based, for example, on historical data with known outcomes or features. The classifications provided by such models (i.e., the model outputs, etc.) can take various forms including a Boolean output (e.g., good / bad, etc.), a numerical score (e.g., <NUM> to <NUM>, <NUM> to <NUM>, etc.), or a grouping (e.g., automobile, pedestrian, crosswalk, etc.). With some software implementations, the outputs of such models can be intercepted even when part of a larger workflow. Such interception can allow a malicious actor to manipulate the classification by such models by repeatedly providing sample input data until a desired classification is received (even if such classification is not ultimately accurate).

<CIT> discloses methods of automatic inline detection based on static data. A file being received by a recipient device is analyzed using an inline parser. The inline parser identifies sections of the file and feature vectors are created for the identified sections. The feature vectors are used to calculate a score corresponding to the malicious status of the file as the information is being analyzed. If a score is determined to exceed a predetermined threshold, the file download process is terminated. The received files, file fragments, feature vectors and/or additional data may be collected and analyzed to build a probabilistic model used to identify potentially malicious files.

TIANWEIZHANG ET AL: "Privacy-preserving Machine Learning through Data Obfuscation", arxiv. org, Cornell University Library, <NUM> Olin Library Cornell University Ithaca, NY <NUM> discloses privacy-preserving a classification model through data obfuscation by adding random noise to samples , or by adding random noise to extracted features of the samples, or by adding random noise to a training dataset of the classification model, or by adding random noise to a group of samples.

<CIT> discloses a speech recognition system where a node of a neural network is processed by determining a score for the node as a product of weights and inputs such that the weights are fixed point integer values, applying a correction to the score based a correction value associated with at least one of the weights, and generating an output from the node based on the corrected score.

According to the present invention, there is provided a method according to claim <NUM>. According to the method, an artefact is received. Features are extracted from this artefact which are, in turn, used to populate a vector. The vector is then input into a classification model to generate a score. The score is then modified using a step function to add noise to the score to obfuscate its actual value so that a classification of said modified score does not differ from a classification of said score generated by said classification model, wherein said classification is one of a numeric value, a classification type or cluster, or other alphanumeric output. Thereafter, the modified score can be provided to a consuming application or process.

In some variations, features in the vector can be reduced prior to it being input into the classification model. The features can be reduced, for example, using random projection matrices, principal component analysis, or other techniques.

The classification model can be a machine learning model trained using a training data set and providing a continuous scale output.

The classification model can characterize the artefact as being malicious or benign to access, execute, or continue to execute. If the artefact is deemed malicious by the classification model, access or execution of the artefact can be prevented.

The classification model can include one or more of: a logistic regression model, a neural network, a concurrent neural network, a recurrent neural network, a generative adversarial network, a support vector machine, a random forest, or a Bayesian model.

The step function can apply various types of noise to the score including, for example, position-dependent noise. Different types of step functions / algorithms incorporating step functions can be applied.

Non-transitory computer program products (i.e., physically embodied computer program products) are also described that store instructions, which when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform operations herein. Such computing systems can be connected and can exchange data and/or commands or other instructions or the like via one or more connections, including but not limited to a connection over a network (e.g., the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, or the like), via a direct connection between one or more of the multiple computing systems, etc..

The subject matter described herein provides many technical advantages. For example, the current subject matter provides enhanced techniques for obfuscating the output of an AI / machine learning model. Such obfuscation is particularly important for applications such as malware detection as it prevents a malicious actor from iteratively modifying a malicious file or code until such time that the model classifies such file or code as being safe to execute or otherwise access.

The current subject matter is directed to techniques for obfuscating an output of a software-based classifier. The classifier in this regard can be an AI / machine learning model that outputs at least one value that characterizes the input to such model. While the current subject matter provides examples relating to models used for detection of malicious software ("malware"), it will be appreciated that the current subject matter can, unless otherwise specified, apply to other applications / workflows utilizing a model including, for example, autonomous vehicle navigation systems, image analysis systems, biometric security systems, video game cheat circumvention systems, and the like.

In some cases, the output of a classification model can be intercepted and exploited by a malicious actor as part of an adversarial attack. For example, data exchanged between a client and a remote server executing the classification model can be accessed such that small changes can be made to the data (e.g., file, code, artifact, etc.) input into the classification model until a desired outcome (from the point of view of the malicious actor) is obtained. For example, a malicious actor either automatically or through manual modifications can make small changes to a file encapsulating malicious code until such time that classification model determines that such file is safe to execute or otherwise access.

<FIG> is a process flow diagram <NUM> illustrating a sample computer-implemented workflow for use with the current techniques for score obfuscation. Initially, an artefact <NUM> can be received (e.g., accessed, loaded, received from a remote computing system, etc.). The artefact <NUM> can be a file, a portion of a file, metadata characterizing a file, and/or source code. This artefact <NUM> can be parsed or otherwise processed by an observer. In particular, the observer can extract <NUM> features (sometimes referred to as attributes or observations) from the artefact and vectorize <NUM> such features. Further, depending on the complexity and/or quantity of features within a vector, a feature reduction operation <NUM> can be performed on the vector which reduces an amount of dimensions of such vector. The feature reduction operation <NUM> can utilize various techniques including, but not limited to, principal component analysis and random projection matrices to reduce the number of extracted features within the vector while, at the same time, remaining useful (i.e., for classification purposes, etc.) when input into the classification model <NUM>. The classification model <NUM> can take many forms including, without limitation, a logistic regression model, a neural network (including concurrent neural networks, recurrent neural networks, generative adversarial networks, etc.), a support vector machine, a random forest, a Bayesian mode, and the like. The output of the classification model <NUM> can be a score <NUM> which, as described in further detail below, can be obfuscated <NUM>. As used herein, unless otherwise specified, the score can be a numeric value, a classification type or cluster, or other alphanumeric output which, in turn, can be used by a consuming process <NUM> or application to take some subsequent action. In some variations, the entity consuming the score, at <NUM>, is provided with the utilized step function so that the underlying score can be determined (i.e., reverse engineered, etc.). In some variations, the obfuscated score can be consumed directly by the entity consuming the score at <NUM>. For malware applications, the score can be used to determine whether or not to access, execute, continue to execute, quarantine, or take some other remedial action which would prevent a software and/or computing system from being infected or otherwise infiltrated by malicious code or other information encapsulated within the artefact <NUM>.

<FIG> further illustrates the interception of the score <NUM>. Such interception can occur, for example, when the API of the consuming application is known; by dumping DLL/SO exports with link, nm, objdump; by using various reverse-compilers; by observing stack/heap/registers during execution for function-calling behavior, and the like. Other API (i.e., function)-discovering techniques can also be used.

In an arrangement in which the output of the model <NUM> can be readily ascertained (e.g., a single model <NUM> arrangement without score obfuscation), the score <NUM> can be used by a malicious actor to modify the artefact <NUM> and repeat the process until such time that a desired score <NUM> by the corresponding model <NUM>. For example, the modified artefact <NUM> can encapsulate malicious script and small changes to the artefact <NUM> could result in the corresponding classification model <NUM> classifying such modified artefact <NUM> as being benign.

Modifications to an artefact <NUM> can be done in such a way as to maintain the original character or nature of the artefact <NUM>. In the example of an actor attempting to modify a malicious file (malware), any modifications must be such that the malware still operates as intended. Such modifications can be made by (for instance) adding to, removing from, or altering un-used portions of the malicious file. As these portions of the file are unused, they have no effect on the realized behavior of the file, but may result in a different score <NUM> from the model <NUM>. Alternatively or additionally, used sections of the artefact <NUM> can also be modified, so long as the final function of the malware is left intact.

Whether manually, or in an automated system, the actor or system will typically make many small changes, and get new scores <NUM> from the model <NUM>. Any change that moved the score <NUM> in the desired direction (i.e. in the malware example, moving the score closer to a value that is interpreted as benign) is maintained, while other changes are discarded. Such an iterative process can be repeated until the cumulative changes to the artefact <NUM> result in a cumulative change in the score <NUM> which accomplishes the desired effect. The obfuscation techniques provided herein can interrupt this cycle of iterative improvements by masking the true effect of each change to an artefact <NUM> with a false or misleading change in the score <NUM> which is determined by the obfuscation techniques herein.

The score obfuscation, at <NUM>, causes the output score to be changed to a new value. For example, with reference to diagram <NUM> of <FIG>, there are three output scores <NUM>, <NUM>, <NUM>. For the first output score <NUM> (which corresponds to the obfuscated score), there can be multiple other scores <NUM>', <NUM>", etc. which, when obfuscated, all result in the same output score <NUM>. Similarly, there can be other scores <NUM>'+ <NUM>", <NUM>'+ <NUM>" which, when obfuscated, result in different output scores (respectively <NUM>, <NUM>). The score obfuscation operation <NUM> can use, as an example, a function to associate various scores <NUM> output by the model <NUM> with a particular output score (e.g., scores <NUM>, <NUM>, <NUM>). In some cases, the function can be a rounding function. In other cases, more complex functions can be utilized including, for example, a step function utilizing position-dependent noise. With the step function algorithm below, the "position" of the original score in the overall range of scores determines where the final score will end up. This change from original score to final score is the noise, and that noise is determined entirely by the position of the original score. A noise map can be used so that the input score is rounded, or binned, and then the rounded/binned value checked in the noise map, and the associated value is returned as the obfuscated score. In some variations, features of the input vector can be used as additional inputs to the mapping function, such that depending on the values both of the original score, and one or more features from the vector, different obfuscated scores would be the result.

The goal of the utilized step functions is that small changes to the input to the model, which would normally yield small score changes, by an adversary (i.e., malicious actor) result in no apparent score change.

Another such step function can use some aspect of the original score to determine the magnitude and direction of the noise to be added to that score. For instance, a trigonometric function, such as a simple sine or cosine function can be used. After the original score is calculated, this trigonometric function can be calculated on that score, and the result would be added to true score to produce the final, obfuscated score. The absolute size of the noise can be limited by a coefficient to ensure that the overall distribution of noises fell within a certain desirable range. Additionally, for scores near the boundary between one classification and another, care can be taken such that this noise would not cause the score to flip over the boundary. To avoid these kinds of flips, a noise attenuation function can be used such that when the true score approaches such a classification boundary, the scale of the noise added to the true score would be reduced such that it was always less than that which would cause a score flip.

Example trigonometric functions are provided below. It will be appreciated that other complex trigonometric functions and/or other types of cyclical functions can be utilized with the current subject matter.

Simple Trigonometric Noise: F(score) = score + (A * sin(B* score)); where A and B are parameters chosen by the implementer. A effects the magnitude of the added noise, and B effects how quickly the noise function changes between similar input scores.

Trigonometric Noise with Truncation: F(score, nearest_score_boundary) = score + min(A * sin(B*score), abs(score-nearest_score_boundary- C)); where A and B are as above, and nearest_score_boundary is the closest score to the input score that represents the boundary between two classifications. In the case were positive score values are interpreted as one class, and negative score values are interpreted as another, then the score boundary would be zero. C is a third parameter chosen by the implementer, which can further limit the score obfuscation from producing values very near a boundary. This function ensures that the obfuscate score never crosses a score boundary.

Trigonometric Noise with Attenuation: F(score) = score + logistic_fn(abs(score), theta) * A*sin(B*score)); where A and B are as above, logistic_fn is a parameterized logistic function, and theta are the parameters to the logistic function. The logistic function produces a value between <NUM> and <NUM>, such that when the input score is close to zero, the value of the logistic function is also approaches zero. The parameters, theta, can be chosen such that the logistic function only attenuates the noise within a certain range of zero, as desired by the implementer.

Utilizing complex stepping functions is advantageous in that it makes it more difficult to reverse engineer. <FIG> is a diagram <NUM> illustrating the output of a more complex function in which the scores <NUM> (before obfuscation) are altered (as shown on the obfuscated scores <NUM> line) in an apparently random manner. As an example of such a function, the range of scores which represent a particular classification (for example, all positive scores, or all negative scores, etc.) can be randomly cut into a large number of very small sections. Each section of the score range could then be mapped randomly to a different output in the same scoring range. When the score <NUM> is obfuscated <NUM>, this map would be used to convert the true score to the obfuscated score. This mapping could be retained for a period of time so that repeated queries to the model would yield the same obfuscated score. Another example of such a function might use information from the reduced feature vector <NUM> in the mapping from the true scores to the obfuscated scores.

With the current arrangement, the ultimate classification of the obfuscated scores <NUM> are maintained. Stated differently, a positive score (indicating that the model output is good) is maintained after the obfuscation and similarly, a negative score is maintained after the obfuscation.

Another sample obfuscation equation is as provided: <MAT> where y is the obfuscated score <NUM> and x is the original score <NUM>. This obfuscation equation can result in scores as illustrated in diagram <NUM> of <FIG> in which line <NUM> represents the original scores <NUM> and the other values <NUM> correspond to the obfuscated scores <NUM>.

<FIG> is a process flow diagram in which, at <NUM>, an artefact is received. Thereafter, at <NUM>, features are extracted from the artefact so that a vector can be populated with such features. Next, at <NUM>, the vector is input into a classification model to generate a score. This score is modified, at <NUM>, using a step function to obfuscate its actual value. Therefore, the modified score is provided, at <NUM>, to a consuming application or process. For example, the consuming application or process can use such score to make a determination of whether or not to access, execute, or continue to execute the artefact (i.e., it can be used to prevent malware from infiltrating a computing and/or software system, etc.).

<FIG> is a diagram <NUM> illustrating a sample computing device architecture for implementing various aspects described herein. A bus <NUM> can serve as the information highway interconnecting the other illustrated components of the hardware. A processing system <NUM> labeled CPU (central processing unit) (e.g., one or more computer processors / data processors at a given computer or at multiple computers / processor cores, etc.), can perform calculations and logic operations required to execute a program. A non-transitory processor-readable storage medium, such as read only memory (ROM) <NUM> and random access memory (RAM) <NUM>, can be in communication with the processing system <NUM> and can include one or more programming instructions for the operations specified here. Optionally, program instructions can be stored on a non-transitory computer-readable storage medium such as a magnetic disk, optical disk, recordable memory device, flash memory, or other physical storage medium.

In one example, a disk controller <NUM> can interface with one or more optional disk drives to the system bus <NUM>. These disk drives can be external or internal floppy disk drives such as <NUM>, external or internal CD-ROM, CD-R, CD-RW or DVD, or solid state drives such as <NUM>, or external or internal hard drives <NUM>. As indicated previously, these various disk drives <NUM>, <NUM>, <NUM> and disk controllers are optional devices. The system bus <NUM> can also include at least one communication port <NUM> to allow for communication with external devices either physically connected to the computing system or available externally through a wired or wireless network. In some cases, the at least one communication port <NUM> includes or otherwise comprises a network interface.

To provide for interaction with a user, the subject matter described herein can be implemented on a computing device having a display device <NUM> (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information obtained from the bus <NUM> via a display interface <NUM> to the user and an input device <NUM> such as keyboard and/or a pointing device (e.g., a mouse or a trackball) and/or a touchscreen by which the user can provide input to the computer. Other kinds of input devices <NUM> can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback by way of a microphone <NUM>, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input. The input device <NUM> and the microphone <NUM> can be coupled to and convey information via the bus <NUM> by way of an input device interface <NUM>. Other computing devices, such as dedicated servers, can omit one or more of the display <NUM> and display interface <NUM>, the input device <NUM>, the microphone <NUM>, and input device interface <NUM>.

To provide for interaction with a user, the subject matter described herein may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) and/or a touch screen by which the user may provide input to the computer. Other kinds of devices may be used to provide for interaction with a user as well; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

Claim 1:
A computer-implemented method (<NUM>) comprising:
receiving (<NUM>) an artefact;
extracting (<NUM>) features from the artefact and populating a vector;
inputting (<NUM>) the vector into a classification model to generate a score (<NUM>);
modifying (<NUM>) the score using a step function to add noise to the score to obfuscate its actual value so that a classification of said modified score (<NUM>) does not differ from a classification of said score (<NUM>) generated by said classification model, wherein said classification is one of a numeric value, a classification type or cluster, or other alphanumeric output; and
providing (<NUM>) the modified score (<NUM>) to a consuming application or process.