Patent Description:
Now days, most of the data processing and decision making systems are implemented using artificial intelligence modules. The artificial intelligence modules use different techniques like machine learning, neural networks, deep learning etc..

Most of the AI based systems, receive large amount of data, process the data to train AI models. Train AI models generate output based on the use cases requested by the user. Typically the AI systems are used in the fields of computer vision, speech recognition, natural language processing, audio recognition, healthcare, autonomous driving, manufacturing, robotics etc. where they process data to generate required output based on certain rules/intelligence acquired through training.

To process the inputs, the AI systems use various models which are trained using the training data. Once the AI system is trained using the training data, the AI systems use the models to analyze the real time data and generate appropriate result. The models may be fine-tuned in real-time based on the results.

The models in the AI systems form the core of the system. Lot of effort, resources (tangible and intangible), and knowledge goes into developing these models.

It is possible that some adversary may try to capture/copy/extract the model from AI systems. The adversary may use different techniques to capture the model from the AI systems. One of the simple techniques used by the adversaries is where the adversary sends different queries to the AI system iteratively, using his own test data. The test data may be designed in a way to extract internal information about the working of the models in the AI system. The adversary uses the generated results to train his own models. By doing these steps iteratively, it is possible to capture the internals of the model and a parallel model can be built using the same logic. This will cause hardships to the original developer of the AI systems. The hardships may be in the form of business disadvantages, loss of confidential information, loss of lead time spent in development, loss of intellectual properties, loss of future revenues etc..

There are methods known in the prior arts to identify such attacks by the adversaries and to protect the models used in the AI system. The prior art <CIT> discloses one such method.

The method disclosed in above prior art receives the inputs, the input data is processed by applying a trained model to the input data to generate an output vector having values for each of the plurality of pre-defined classes. A query engine modifies the output vector by inserting a query in a function associated with generating the output vector, to thereby generate a modified output vector. The modified output vector is then output. The query engine modifies one or more values to disguise the trained configuration of the trained model logic while maintaining accuracy of classification of the input data.

Non-patent Literature "<NPL>, provides a model extraction monitor that quantifies the extraction status of models by continually observing the API query and response streams of users.

Different modes of the invention are disclosed in detail in the description and illustrated in the accompanying drawing:.

The present invention is set out by the appended independent claims. Preferred embodiments are set out by the dependent claims.

Shown in <FIG> is a block diagram of an AI based system <NUM> capable of preventing stealing of a model <NUM> implemented in the AI system <NUM>. The AI based system <NUM> is also referred just as System or as a data processing system in this document. The term stealing is also referred as capturing in this document.

Only the important components of the system <NUM> are disclosed in this document as all other components are commonly known. The system <NUM> has an input interface <NUM> to receive inputs, an AI module <NUM> to process the inputs, an output interface <NUM> to provide the outputs. The AI module <NUM> contains a model which processes the inputs using AI techniques and generates required output. The model is also referred as AI model. The model may be implemented as a set of software instructions, combination of software and hardware or any combination of the same. The input interface102 may be a keyboard, a touch screen, images, videos, suitable stimulus etc. It is also possible that the inputs may come over a bus or wirelessly or through any other communication. The output interface <NUM> may comprise a display or a bus. The output may be displayed on a display or sent through the bus which can be read by other devices. The model may be implemented using a neural network. Neural network is only an example here as there are other techniques available for implementing AI modules.

Neural networks are a set of algorithms, modeled after the human brain and cognition theory that are designed to recognize patterns.

Neural networks help us cluster and classify the data. They help to group unlabeled data according to similarities among the training (example) inputs, and they classify data when they have a labeled dataset to train on. Neural networks can also extract features that are fed to other algorithms for clustering and classification.

As shown in <FIG>, the AI model comprises a neural network. It is also possible that the AI model may be implemented using other techniques. The neural network shown in <FIG>, typically has input layer <NUM>, hidden layers <NUM>, <NUM>, <NUM> etc. and an output layer <NUM>. In deep neural networks there may be multiple hidden layers. The data is processed in hidden layers. The output of one hidden layer is passed as input to next hidden layer for further processing. There may be different weightages assigned to the inputs.

The layers are made of nodes represented as circles. A node is just a place where computation happens, loosely patterned on a neuron in the human brain, which fires when it encounters sufficient stimuli. A node combines input from the data with a set of coefficients, or weights that either amplify or dampen that input, thereby assigning significance to inputs with regard to the task the algorithm is trying to learn; e.g. which input is most helpful in classifying data without error? These input-weight products are summed and then the sum is passed through a node's so-called activation function, to determine whether and to what extent that signal should progress further through the network to affect the ultimate outcome, say, an act of classification. If the signals passes through, the neuron has been "activated.

Shown in <FIG> are samples of inputs shown as input <NUM>. The system contains a set of classes represented as part of output <NUM>.

Deep learning neural network maps inputs to outputs. It finds correlations/patterns between the inputs <NUM> and outputs <NUM>. The neural networks can learn to approximate an unknown function f(x) = y between any sets of input x and any sets of output y, assuming they are related. In the process of learning, a neural network finds the right f, or the correct manner of transforming and mapping x into y.

And determining the function which correlates and/or finds the pattern to map the input to the output, forms the AI model which may be stolen by an adversary.

Some of the typical tasks performed by AI systems are classification, clustering, regression etc..

All classification tasks depend upon labeled datasets; that is, the data sets are labelled manually in order for a neural network to learn the correlation between labels and data. This is known as supervised learning. Some of the typical applications of classifications are: face recognition, object identification, gesture recognition, voice recognition etc..

Clustering or grouping is the detection of similarities in the inputs. Deep learning does not require labels to detect similarities. Learning without labels is called unsupervised learning. Unlabeled data is the majority of data in the world. One law of machine learning is: the more data an algorithm can train on, the more accurate it will be. Therefore, unsupervised learning has the potential to produce highly accurate models.

As the model forms the core of the AI system, the model needs to be protected against stealing by adversaries. The invention proposes a method to prevent any such attempts to steal the model.

A model stealing attack is a kind of attack vector that can make a digital twin/replica/copy of a pre-trained machine learning model. This attack was demonstrated in different research papers, where the model was captured/copied/extracted to build a substitute model with similar performance.

The attacker typically generates random queries of the size and shape of the input specifications and starts querying the model with these arbitrary queries. This querying produces input-output pairs for random queries and generates a secondary dataset that was inferred from the pre-trained model. The attacker then take this I/O pairs and trains the new model from scratch using this secondary dataset. This is blackbox model attack vector where no prior knowledge of original model is required. As the prior information level regarding model is available, attacker moves towards more intelligent attacks. The attacker chooses relevant dataset at his disposal to extract model more efficiently. This is domain intelligence model backed attack vector.

With these approaches, it is possible to demonstrate model stealing attack across different models and datasets. With regard to this attack, the invention discloses a method to prevent such attacks for stealing the models.

If Model is trained for N classes, then invention proposes to introduce a new class, therefore, making it an N+<NUM> Classes. The new class is labeled as "Don't Know Class (DKC) <NUM>". Thus the system generates output classes <NUM>, <NUM>. The DKC class is also referred as pre-defined class.

From the attack vectors, optimal attack vector and other sub-optimal attack vectors are identified. These attack vectors are random queries, as discussed in the previous part. All these attack vectors are labelled as DKC. During training, the DKC class is randomly sampled to ensure that DKC class is sufficiently represented to avoid class imbalance problem.

Now, the model is trained with N+<NUM> classes where N classes <NUM> are pre-labeled, and new class DKC <NUM> is generated synthetically. The trained model now treats DKC class as default class. If the trained model is not able to associate any given input accurately to any of the classes to <NUM>, then it puts the entry into DKC class <NUM> and returns the DKC class.

Thus the invention proposes that, in run time, if any user tries to run the attack vector, the system will refuse to classify this kind of data and therefore will provide protection, i.e. if the attacker tries to capture the model by iteratively sending inputs which cannot be classified, then the model returns it as a DKC <NUM>. The system calculates the information gain achieved using DKC events and based on gain adaptive counter increment value a threshold is derived. One of the method is using simple thresholding and incrementing. A threshold may be set to store the number of times an input is returned as DKC <NUM>. Every time DKC is detected, a counter is incremented. The incremental value of the counter depends upon the amount of information gained by the attacker through the DKC. The incremental value of the counter may change dynamically. The incremental value may be a simple increment of one step at a time, or the system <NUM> determines threshold for said counter based on at least one of the function usage, time and price. Here the system usage is referred to the amount of time used by the user. Price refers to the amount of money charged to the user. Function usage refers to the functions used by the user. Once the threshold is reached for the counter, the system <NUM> locks the user from accessing the AI system further. The system may be unlocked only after an unlocking criteria is met. The unlocking criteria may be a certain event, for example, a fixed duration of time, a fixed number of right inputs, a manual override etc. Manual override is where unlocking is done by providing manual inputs.

In the absence of a valid input-output pair, the attacker cannot exploit model stealing attack.

In step S1, the user input is received. In step S2 the input is classified against the classes in the data set for a match. If there is a match, in step S3, matching class is returned. If there is no match, a predefined class is returned in step S4. Based on DKC class detection, in step S4a, the system <NUM> calculates counter increment value based on predefined functions. In step S5, a counter is incremented based on values in S4a. In step S6, it is checked whether counter reached threshold. If the counter reaches the threshold, the system is locked in step S7. If the counter does not reach threshold, the control goes to step <NUM>. In the event of system lock, the system <NUM> evaluates the unlocking criteria in step S8. The unlocking criteria is provided in step S9. If unlocking criteria is not met, system remains in lock down state in step S7. If unlocking criteria is met then system control goes to step <NUM>.

The invention proposes a simple system which has capability to prevent any attack from an adversary to steal a module which is based on AI. The system comprises: an input interface <NUM>; an output interface <NUM>; a model <NUM> to process received inputs, said model <NUM> using artificial intelligence techniques; a set of data <NUM> where different classes are stored; said data processing system adapted to receive an input <NUM> from an user (S1); check whether said input <NUM> corresponds to at least one of classes <NUM> in a data set (S2); return a class <NUM> which corresponds to said input <NUM>, from said data set, if said input <NUM> corresponds to at least one of said classes <NUM> in a data set (S3); return a pre-defined class <NUM> (Don't Know Class - DKC) from said data set if said input does not correspond to any of said classes in said data set (S4).

The invention proposes a method to prevent attack from an adversary to steal a model. The method comprises steps of: receiving an input <NUM> from an user; checking whether said input <NUM> corresponds to at least one of classes <NUM> in a data set; returning a class <NUM> which corresponds to said input <NUM>, from said data set, if said input <NUM> corresponds to at least one of said classes <NUM> in a data set; returning a pre-defined class <NUM> (Don't Know Class - DNC) from said data set if said input does not correspond to any of said classes in said data set.

Claim 1:
A method to prevent capturing of models in an Artificial Intelligence based system, AI system, (<NUM>), wherein the AI based system (<NUM>) has an AI module (<NUM>) containing a model implemented as a set of software instructions, said method comprising the steps:
- receiving an input (<NUM>) from an user by means of an input interface of the AI system;
- checking whether said input (<NUM>) corresponds to at least one of classes (<NUM>) in a data set by means of the AI module (<NUM>);
- returning a class (<NUM>) which corresponds to said input (<NUM>), from said data set, if said input (<NUM>) corresponds to at least one of said classes (<NUM>) in said data set;
- returning a pre-defined synthetically generated class (<NUM>) Don't Know Class, DKC, which the model is trained to associate to random input queries, if said input (<NUM>) does not correspond to any of said classes in said data set;
- when said Don't Know Class, DKC, is returned:
o calculating information gain achieved using DKC events ;
o incrementing a gain adaptive counter to derive a threshold;
- locking the AI system (<NUM>) from an access of the user once said gain adaptive counter reaches said threshold to prevent capture of model in the AI system (<NUM>).