Patent Description:
Machine Learning (ML) is an approach that allows a programmer to implement a program by finding patterns in data samples. A program that is obtained through Machine Learning is called a Machine Learning model. The model can be used in various analysis tasks, for example classification or regression. A dataset of samples used to build the model is also known as a training set. The model parameters and data samples are mostly confidential while source code for training and execution of Machine Learning models is increasingly becoming available as open source. As such, high quality data samples are often the most valuable commodity in ML. Software that has a Machine Learning model as one of its core modules is called an ML-assisted service. The Machine Learning model that is deployed to a ML-assisted service is called the deployed model.

A model extraction adversary is an adversary (or "attacker") aiming to replicate the functionality of a deployed model inside an ML-assisted service. This type of attack, also known as a model extraction attack, does not aim to exploit software vulnerabilities. Rather, adversaries query the model over an exposed API in the same way as authorized users, the difference being that the adversary makes a great many queries in order to build up a dataset of query-response pairs with which to train a new model. In this way, an attacker can effectively steal the underlying dataset used to train the model and generate a training dataset for free.

Motivations of an attacker can be to replicate the model, so as to be able to use it for free, sell it on, or for further improvement for other purposes. The goal of an adversary may go beyond model replication. The replicated model can also be used as a stepping stone for data reconstruction, also known as a model inversion attack, or in a testbed for evasion attacks looking for vulnerabilities in the functioning of the deployed model in an ML-assisted service.

There are various known methods to address different aspects of extraction attacks. For example, an attacker can be distinguished from an ordinary (genuine) user of an API through abnormal query detection. This method analyzes successive queries from users and consequently identifies suspicious queries. See the paper by<NPL>.

Extraction progress, e.g. the proportion of the feature space that has been made available to the user through query-response pairs, can also be estimated. The goal of this monitoring method is to detect an adversary by estimating extraction progress. This is described in the paper by <NPL>.

Another technique is dimensionality reduction. The purpose of this preventive method is to limit the degrees of freedom available to an adversary by reducing the number of input features to the model. This is described in the paper by <NPL>.

Another mitigation technique is perturbation-based defense whereby outputs of a model are offset or perturbed before being sent to the user. This approach aims to preserve accuracy while truncating the output information. See paper by <NPL>).

Another technique is to watermark the output. This method aims to prove ownership of a model by detecting watermarks on suspicious models. See paper by <NPL>).

Another technique is describe in the paper by <NPL>) wherein the defense involves flagging "suspicious" queries and adaptively servicing these queries with incorrect predictions from an auxiliary "misinformation model" which produces uncorrelated predictions.

Extraction attacks aim to extract information from an ML model through selective querying in order to build up a database of query-response pairs. There are various problems associated with the known techniques described in the background for managing extraction attacks, for example, none provide a solution in a potentially harmful situation, for example where an extraction attack is suspected but not yet confirmed.

The preventive, monitoring, mitigation or detection approaches described in the background section are rather designed for use in either on-going monitoring of a normal situation or a failed situation where a user is confirmed as being an adversary. However, taking an appropriate reactive action in a potentially harmful (but unconfirmed) situation is not a trivial problem. The service cannot just be stopped in this situation since a detection technique may eventually (erroneously) flag a non-malicious user as malicious over time simply because enough information to replicate the model could have been collected by the user if they use the service for long enough. This would be a false positive alert and if it is used to block access to the Machine Learning model, it would lead to negative consequences such as service disruption.

Furthermore, "normal query" patterns can be difficult to detect because they tend to vary between customers. Moreover, abnormal query detection methods need to take the context into account to determine if queries detected by the method are sent from attackers or not, since the root cause for those queries might be actual software defects or hardware malfunctions on a network node and not model extraction attacks. Therefore, flagging attackers based on query detection patterns alone is not always helpful in a potentially harmful situation.

Estimations of extraction progress are designed for use in a normal situation but do not address the issue of how to react to a suspected model extraction adversary.

Dimensionality Reduction cannot claim to mitigate, but rather only marginally delay, an attack by increasing the number of queries that would be required to replicate the model. It also sacrifices accuracy if the method is applied only for model extraction defense, not for data refinement.

Perturbation-based defenses are still vulnerable to attacks, despite being capable of delaying an attack. Moreover, this mitigation approach can be detected by adversaries. Users of an ML-based service may even request that this approach is avoided, with valid reasons.

Lastly, watermarking only works after an attack has successfully extracted the model. It is also based on the strong assumption that the owner can detect infringement and that the attackers will open the replicated models to be audited. A watermark can be hard to find and may not be replicated in a model trained using data obtained through an extraction attack. In other words, only a true copy ensures that a watermark is transferred to a stolen model. Moreover, adversaries who anticipate watermarking defenses can take steps to prevent detection, such as double extraction and fine-tuning.

It is an object of the invention herein to improve the security of ML models exposed over APIs such as those in ML-assisted services.

Thus, according to a first aspect herein there is a method for use in protecting a first machine learning model that is queryable over an application programming interface, API, against an adversary querying the first machine learning model through the API in order to build up a database of query-response pairs. The method comprises identifying a user of the API as a potential adversary; and in response to a query from the potential adversary, through the API, providing a response from a second machine learning model instead of the first machine learning model, wherein the first machine learning model has been trained on a first dataset and wherein the second machine learning model has been trained on a second dataset that is different to the first dataset.

According to a second aspect there is an apparatus for use in protecting a first machine learning model that is queryable over an application programming interface, API, against an adversary querying the first machine learning model through the API in order to build up a database of query-response pairs. The apparatus comprises: a memory comprising instruction data representing a set of instructions; and a processor configured to communicate with the memory and to execute the set of instructions. The set of instructions, when executed by the processor, cause the apparatus to: identify a user of the API as a potential adversary; and in response to a query from the potential adversary, through the API, provide a response from a second machine learning model instead of the first machine learning model, wherein the first machine learning model has been trained on a first dataset and wherein the second machine learning model has been trained on a second dataset that is different to the first dataset.

According to a third aspect there is an apparatus for use in protecting a first machine learning model that is queryable over an application programming interface, API, against an adversary querying the first machine learning model through the API in order to build up a database of query-response pairs. The apparatus is configured to: identify a user of the API as a potential adversary; and in response to a query from the potential adversary, through the API, provide a response from a second machine learning model instead of the first machine learning model, wherein the first machine learning model has been trained on a first dataset and wherein the second machine learning model has been trained on a second dataset that is different to the first dataset.

According to a fourth aspect there is a computer program comprising instructions which, when executed on at least one processor, cause the at least one processor to carry out a method according to the first aspect.

According to a fifth aspect there is a carrier containing a computer program according to the fourth aspect, wherein the carrier comprises one of an electronic signal, optical signal, radio signal or computer readable storage medium.

A computer program product comprising non transitory computer readable media having stored thereon a computer program according to the fourth aspect.

Thus, described herein are apparatus and methods that can be used to protect a first machine learning model that is exposed over an API, against a suspected extraction attack, whilst still providing a service to a user. When a user is identified as a suspected adversary, instead of continuing to service requests from the user, in the normal way, a second, different, model is used instead. The second model is trained on different training data. This way, the first model is protected as it is no longer made available to the user, however service is still provided to the user which prevents service interruption.

As will be described in more detail below, the second model can take many forms, but in some embodiments, can be trained using previous query-response pairs requested by the user. In this example, the user can still obtain accurate responses for queries that are similar to their previous queries, but can be prevented from extracting further information from the first model. Thus, service is maintained to the user whilst protecting the first model.

For a better understanding and to show more clearly how embodiments herein may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings, in which:.

The disclosure herein relates to methods and apparatus for protecting (e.g. restricting access to or use of) a first machine learning model (e.g. a first model trained using a machine learning process) that is deployed and made available over an API, for example, as part of an ML-assisted service.

<FIG> shows an example apparatus <NUM> that may be used to protect a first machine learning model according to some embodiments herein. The apparatus <NUM> is configured (e.g. adapted, operative, or programmed) to perform any of the embodiments of the method <NUM> as described below.

The apparatus <NUM> herein comprises a processor (e.g. processing circuitry or logic) <NUM>. The processor <NUM> may control the operation of the apparatus <NUM> in the manner described herein. The processor <NUM> can comprise one or more processors, processing units, multi-core processors or modules that are configured or programmed to control the apparatus <NUM> in the manner described herein. In particular implementations, the processor <NUM> can comprise a plurality of computer programs and/or hardware modules that are each configured to perform, or are for performing, individual or multiple steps of the functionality of the apparatus <NUM> as described herein.

The apparatus <NUM> comprises a memory <NUM>. In some embodiments, the memory <NUM> of the apparatus <NUM> can be configured to store a computer program <NUM> with program code or instructions that can be executed by the processor <NUM> of the apparatus <NUM> to perform the functionality described herein. Alternatively, or in addition, the memory <NUM> can be configured to store any requests, resources, information, data, signals, or similar that are described herein. The processor <NUM> may be configured to control the memory <NUM> to store any requests, resources, information, data, signals, or similar that are described herein.

It will be appreciated that the apparatus <NUM> may comprise one or more virtual machines running different software and/or processes. The apparatus <NUM> may therefore comprise one or more servers, switches and/or storage devices and/or may comprise cloud computing infrastructure or infrastructure configured to perform in a distributed manner, that runs the software and/or processes.

It will be appreciated that the apparatus <NUM> may comprise other components in addition or alternatively to those indicated in <FIG>. For example, in some embodiments, the apparatus <NUM> may comprise a communications interface. A communications interface may be for use in communicating with other apparatuses e.g. via a communications network, (e.g. such as other physical or virtual computing nodes). For example, the communications interface may be configured to transmit to and/or receive from nodes or network functions requests, resources, information, data, signals, or similar. The processor <NUM> may be configured to control such a communications interface to make/receive such transmissions.

The apparatus <NUM> may be implemented in (e.g. form part of) a communications network. In some embodiments herein, the apparatus <NUM>, may be implemented in a management layer of a communications network.

More generally, the apparatus <NUM> may be implemented in any node/network device of a communications network. For example, the apparatus <NUM> may comprise any component or network function (e.g. any hardware or software) in a communications network suitable for performing the functions described herein. Examples of nodes include but are not limited to core network functions such as, for example, core network functions in a Fifth Generation Core network (5GC). It is realized that the apparatus <NUM> may be included as a node/device in any future network, such as a future 3GPP (<NUM>rd Generation Partnership Project) sixth generation communication network, irrespective of whether the apparatus <NUM> would there be placed in a core network or outside of the core network.

A communications network or telecommunications network may comprise any one, or any combination of: a wired link (e.g. ASDL) or a wireless link such as Global System for Mobile Communications (GSM), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), New Radio (NR), WiFi, Bluetooth or future wireless technologies. The skilled person will appreciate that these are merely examples and that a communications network may comprise other types of links. A wireless network may be configured to operate according to specific standards or other types of predefined rules or procedures. Thus, a wireless network may implement communication standards, such as Global System for Mobile Communications (GSM), Universal Mobile Telecommunications System (UMTS), Long Term Evolution (LTE), and/or other suitable <NUM>, <NUM>, <NUM>, or <NUM> standards; wireless local area network (WLAN) standards, such as the IEEE <NUM> standards; and/or any other appropriate wireless communication standard, such as the Worldwide Interoperability for Microwave Access (WiMax), Bluetooth, Z-Wave and/or ZigBee standards.

Generally (as will be described in more detail below), the apparatus <NUM> is for use in protecting (or managing) a first machine learning model that is queryable over an application programming interface, API, against an adversary querying the first machine learning model through the API in order to build up a database of query-response pairs. Briefly the apparatus <NUM> is configured to identify a user of the API as a potential adversary and, in response to a query from the potential adversary, through the API, provide a response from a second machine learning model instead of the first machine learning model. The first machine learning model has been trained on a first dataset and the second machine learning model has been trained on a second dataset that is different to the first dataset.

The skilled person will be familiar with machine learning and models that can be trained using machine learning processes, but briefly, a machine learning process may comprise a procedure that is run on data to create a machine learning model. The machine learning process comprises procedures and/or instructions through which training data, may be processed or used in a training process to generate a machine learning model. The machine learning process learns from the training data. For example, the process may be used to determine how one set of parameters in the training data (input parameters of the model) are correlated with another set of parameters in the training data (output parameters of the model). The machine learning process may be used to fit the model to the training data. Machine learning processes can be described using math, such as linear algebra, and/or pseudocode, and the efficiency of a machine learning process can be analyzed and quantized. There are many machine learning processes, such as e.g. algorithms for classification, such as k-nearest neighbors, algorithms for regression, such as linear regression or logistic regression, and algorithms for clustering, such as k-means. Further examples of machine learning models are Decision Tree models and Artificial Neural Network models. Machine learning algorithms can be implemented with any one of a range of programming languages.

The model, or machine learning model, may comprise both data and procedures for how to use the data to e.g. make the predictions described herein. The model is what is output from the machine learning (e.g. training) process, e.g. a collection of rules or data processing steps that can be performed on the input data in order to produce the output. As such, the model may comprise e.g. rules, numbers, and any other algorithm-specific data structures or architecture required to e.g. make predictions.

Different types of models take different forms. Some examples of machine learning processes and models that may be used herein include, but are not limited to: linear regression processes that produce models comprising a vector of coefficients (data) the values of which are learnt through training; decision tree processes that produce models comprising trees of if/then statements (e.g. rules) comprising learnt values; and neural network models comprising a graph structure with vectors or matrices of weights and biases with specific values, the values of which are learnt using machine learning processes such as backpropagation and gradient descent.

In some embodiments, the first machine learning model is a decision tree or a random forest-based classifier, see papers: <NPL>); and <NPL>. In other embodiments, the first machine learning model is a deep neural network, see paper by <NPL>.

More generally, the first machine learning model (otherwise referred to herein as the first model) can be any type of classification or regression model, trained using a supervised or semi-supervised learning process. The first model has been trained on (e.g. using) a first dataset. The first dataset or first training dataset comprises training data examples, each training data example comprises an example set of input parameters and a ground truth value or label for said training data example.

The first dataset may comprise confidential data, e.g. such as confidential user information. The first dataset may comprise valuable data, for example, data that is difficult or expensive to acquire. The first dataset may be very large, e.g. containing hundreds or thousands of training examples. In other words, the first dataset may be valuable and thus a target for an adversary.

The first machine learning model is queryable over an application programming interface (API). For example, the first machine learning model is deployed, for example, as part of an ML-assisted service through which a user can query the model. "Queryable" in this sense means capable of being queried, e.g. that the user can send inputs to and receive outputs from the first machine learning model. For example, the user sends data that can be provided to the first machine learning model as input parameters over the API. The first machine learning model produces an output and this is sent to the user as the response from the API.

In this sense, a user is any third party who queries the first machine model. The user may be a person. Alternatively, the user may be a computer program or computer hardware that is configured to send one or more queries to the first machine learning model. A user may be a subscriber of an ML-assisted service, who is able to query the first machine learning model, via an API. A user may be a genuine or non-malicious user. A user may be a malicious user, who is performing an extraction attack. In embodiments where the user is a computer program or computer hardware, the computer program or computer hardware may thus be a malicious computer program or computer hardware that is configured e.g. to send a large number of queries in a coordinated or automated way in the form of an extraction attack. It is an object of the disclosure herein to deal with users who are suspected to be malicious and may be performing an extraction attack.

The apparatus <NUM> is for managing a suspected adversary. As described in the background section herein, an adversary is a type of user or "attacker" who queries the first machine learning model, in the normal way over the API, in order build up a database of query-response pairs. If enough query-response pairs are obtained or extracted from the model, then they can be used either directly to deduce model parameters or as a training set to train a new model. This is known as an extraction attack.

Turning now to <FIG>, there is a computer implemented method <NUM> for use in protecting a first machine learning model that is queryable over an application programming interface, API, against an adversary querying the first machine learning model through the API in order to build up a database of query-response pairs. Briefly, in a first step the method comprises: identifying <NUM> a user of the API as a potential adversary. In a second step the method then comprises: in response to a query from the potential adversary, through the API, providing <NUM> a response from a second machine learning model instead of the first machine learning model, wherein the first machine learning model has been trained on a first dataset and wherein the second machine learning model has been trained on a second dataset that is different to the first dataset.

The method <NUM> is computer implemented. For example, the method <NUM> may be performed by an apparatus such as the apparatus <NUM> illustrated in <FIG>. The method <NUM> is generally for use in managing a suspected extraction attack by a potential adversary, as described above.

In more detail, in step <NUM> a user of the API is identified as a potential adversary. As used herein, an adversary is a user who queries the first machine learning model, via the API, for the purposes of extracting data from the model, e.g. for use in replicating the model, or stealing the underlying data. It is often difficult to determine whether a user is a genuine user or an adversary, particularly in the early stages of an extraction attack. Thus, in embodiments herein, in step <NUM>, users may be initially flagged as potential/possible adversaries.

Users may be identified (e.g. flagged) as possible adversaries in various different ways. For example, by comparing a query pattern of the user to query patterns of other users to determine whether the user is performing an abnormal query pattern compared to the other users. The other users may be other genuine users of the API or other services. In this way, suspicious behaviour may be identified by comparing the user to others.

As another example, step <NUM> may comprise comparing a query pattern of the user to previous query patterns of the user (e.g. historical query patters of the same user) to determine whether the user is currently performing an abnormal query pattern compared to their previous query patterns. For example, an otherwise legitimate user account may have been hacked by an adversary. Thus, historical or "normal" user patterns may be used as a reference against which to detect changes in user behaviour that may be indicative of an adversary.

As another example, step <NUM> may comprise comparing a query pattern of the user to query patterns associated with extraction attacks to determine whether the user is performing a query pattern consistent with an extraction attack. Generally, an adversary will want to sample the feature space so as to locate the decision boundaries of the model or to directly find the parameters of the ML model. Thus, examples of query patterns consistent with an extraction attack include, but are not limited to: a sequence of queries sampling the full input parameter space at (regular) intervals; a random sequence of queries sampling the full input feature space; or a query pattern from an adaptive method searching for decision boundaries or parameter values. In embodiments where the first machine learning model is a decision tree, in an adaptive method, an adversary may perform a query pattern designed to hone-in on a decision boundary, for example by determining first input parameters that produce a "yes" result, and second input parameters that produce a "no" result, and interpolating between the first input parameters and the second parameters in order to determine the input parameter values corresponding to the boundary.

One example abnormal query detection method is further described in the paper by Juuti, Mika, et al. (<NUM>), referenced above. In their method, they compute the distance between input query points. They assume that query points will be approximately uniformly distributed, leading to a roughly Gaussian distance distribution, and detect deviations resulting from active probing for decision boundaries (similar to the interpolation previously mentioned). For comparing query patterns against other users or historical data, one may look at the distribution of query points in feature space and use known statistical methods, like hypothesis tests or metrics like KL-divergence.

In some embodiments, another machine learning model may be trained and used to predict whether a query pattern is malicious (e.g. consistent with an extraction attack). This is described in the paper entitled "<NPL>). In this paper a separate ML model is trained to determine if the incoming queries probe out-of-distribution or not (yes/no classification).

In another embodiment, step <NUM> can be based on data extraction level. As used herein, the data extraction level is an estimate of the proportion of the input feature space coverage of (previous) queries submitted by the user to the API. In other words, the "extraction progress". Thus, data extraction level can be a measure of feature space coverage of previous queries submitted by the user to the API. The data extraction level can be estimated using the techniques described in the paper by Kesarwani, Manish, et al. (<NUM>), referenced in the background section above. The second method described in their paper uses feature space coverage to estimate model extraction progress by calculating, for each predicted class, the outer boundary of query points and comparing the resulting area with corresponding area covered by the training data points. This method, based on feature, space coverage can be advantageous in terms of saving computation. On the other hand, it assumes decision boundaries similar to decision tree-based models, and will be an approximation for other types of models. Another option for estimating extraction is to simply evaluate the achieved accuracy of the proxy model, as described in the paper, on a test data set. This requires training/retraining the proxy model each time an estimate is computed and new queries have been made.

In some examples, if a user has made queries that cover more than a predetermined data extraction level, then this may trigger step <NUM>. For example, step <NUM> may comprise identifying or flagging the user of the API as a potential adversary if an estimation of a third data extraction level Is above a third threshold data extraction level. step <NUM> may comprise determining whether an estimation of a third data extraction level is above a third threshold data extraction level. In this way, users may be prevented from extracting further data from the first machine learning model if they extract data above the third data extraction level.

The skilled person will appreciate that the embodiments above are merely examples, and that a user may be identified as a potential adversary in other ways to those described herein. For example, through other user profiling methods (e.g. behavior of the user on other APIs, user credit history, missed subscription payments of the user etc).

Once a user is identified as a potential adversary, then the aim of embodiments herein is to service subsequent queries from the user in as accurate a manner as possible without revealing further information from the model. In this way, the user will continue to receive uninterrupted, high quality service (which is important in the event that the user is a genuine user) whilst preventing further information from being extracted from the model (which is important if the user is actually an adversary).

To this end, in response to a query from the potential adversary, through the API, the method <NUM> comprises providing <NUM> a response from a second machine learning model instead of the first machine learning model, wherein the first machine learning model has been trained on a first dataset and wherein the second machine learning model has been trained on a second dataset that is different to the first dataset.

Thus, the second machine learning model is substituted for the first machine learning model and acts as a surrogate for the first machine learning model. The first machine learning model is thus made inaccessible to the user (potential adversary), protecting the first machine learning model from further data extraction. At the same time, the second model allows the user to continue to receive uninterrupted service.

The second machine learning model may be any machine learning model that takes the same, or at least a subset of the same input parameters as the first machine learning model and provides as output the same, or at least a subset of the same outputs as the first machine learning model. In this way, the outputs of the second machine learning model can be, on the face of it, indistinguishable from the outputs of the first machine learning model.

The second machine learning model may be the same type of machine learning model as the first machine learning model (e.g. the first and second machine learning models may both be neural networks, decision trees, or any other type of supervised machine learning model as described above). The second machine learning model may have the same or similar architecture, for example the same or similar hyper-parameters.

Alternatively, the first and second machine learning models may be different types of models. the first machine learning model may be a neural network and the second machine learning model may be a decision tree, or any other combination of the types of supervised machine learning models described above.

The second machine learning model is trained on a second dataset that is different to the first dataset used to train the first model. Generally, the second ML model is similar to the deployed first ML model, however, it has lower performance in terms of accuracy. As described below, in some embodiments, the second dataset is selected so that the second ML model has lower performance in terms of accuracy for inputs dissimilar to previous queries, whilst retaining high accuracy for inputs that are similar to previous queries (from the suspected adversary). For example, the second ML model could be trained based on query response pairs that have already been sent to adversary and/or based on a smaller number of data samples.

Some example second models are as follows:.

In some embodiments, the second dataset comprises cached query-response pairs from previous queries requested by the potential adversary through the API, as stored in a cache. For example, the second model may be trained on previous (e.g. historical) queries made by the user, for which a response through the API has already been made. In this way, any new query that is in the same region of feature space as a previous query made by the user will receive a high quality response, but queries in different regions of the feature space that have not previously been queried by the user will receive a lower quality response. This prevents the user from extracting further data from the first model, whilst ensuring continuity (so that previously-made queries receive the same response). Thus, in the context of an ML-assisted service, this enables the service to provide a model that is as good as possible without leaking more information from the first machine learning model.

Examples where the second model is trained using cached query-response pairs may be described herein as "Incomplete Models" as the second data set used to train the second machine learning model covers only part of the feature space of the first dataset. An incomplete model is built from query response pairs that have been already sent to a potential model extraction adversary. There can be different levels of incompleteness defined by an incompleteness ratio. This measures the share of the second dataset that is from the query response pairs. For example, an incomplete model with <NUM>% incompleteness ratio means that <NUM>% of the training set used to train the incomplete model consists of query response pairs while the rest of the training set is selected from other sources (some or all of which may be from the first dataset and thus be confidential). This approach aims to ensure that the deployed model will not leak any additional information and reacts proportionally to different level of extraction.

Query-response pairs may be stored in a per-user cache or database. The cache is a database that stores a collection of query-response pairs between the ML-assisted service and users. For example, every query made by each user of the API (or ML-assisted service) may be stored in the cache following the query. The main motivation for using a cache is to deliver the same responses for the same incoming queries, thus providing the appearance to the potential adversary that the same model is being queried following the switch from the first ML model to the second ML model.

In examples where the second dataset comprises cached query-response pairs, the second dataset may be complemented with further data in order to build up sufficient training examples with which to train the second model. For example, the method <NUM> may further comprise using a data augmentation process to generate synthetic training data from the cached query-response pairs, and supplementing the second dataset with the synthetic training data. The skilled person will be familiar with data augmentation processes, which may be used to generate further training examples from existing training data examples (e.g. the cached query-response pairs) that can be added to the second dataset. As an example, a data augmentation process may be used to perturb input parameter values, without affecting the output label. As an example, where the training data comprises images that are labelled according to content, the images may be blurred, rotated, mirrored, cropped, tilted, or their contents moved around, or otherwise transformed, without affecting the labelling (e.g. an image of a blurred cat is still labelled as a cat). Papers describing augmentation techniques are: "<NPL>) and "<NPL>.

In other examples, the second dataset may be supplemented with synthetic training data, e.g. generated using an emulator.

More generally, in examples where the second dataset comprises cached query-response pairs, the second dataset may be complemented with any other training examples for which inclusion in the second dataset would not lead to leakage of valuable data. As an example, the cached query-response pairs might be supplemented with further data from the first data set that is low quality or that is not confidential. For example, the second dataset may be complemented with training data from the first dataset that is on average of lower quality compared to the average quality of the first dataset as a whole; or that has a quality rating below a first quality rating threshold. In this way, high quality data may be prevented from further leakage and the second machine learning model may be trained to provide high quality outputs in regions of the feature space that have previously been queried, and lower quality, but still useful, outputs in other regions of the feature space.

As another example, the second dataset may be complemented with training data from the first dataset that is not confidential, on average less confidential compared to the average confidentiality level of the first dataset as a whole. Put another way still, the first dataset may comprise confidential training data that is not comprised in (e.g. removed from) the second dataset.

In another example, the second dataset may be complemented with data that is publicly available or unrestricted, e.g. from open-source databases. In this way, leakage of further high quality, confidential data can be prevented, whilst still generally providing high quality service to the user/suspected adversary.

Incomplete models may be generated/trained when the ML assisted service is up and running, e.g. real-time training on the cached query-response pairs. Incomplete models are user-specific (as they are trained on the cached query-response pairs saved for the particular user). The skilled person will be familiar with real-time training methods that can be used, for example, standard ML model libraries can be used such as the sci-kit-learn library which is described in the paper "<NPL>.

In other embodiments, instead of providing a response from a second model based on cached query-response pairs for the user, responses may be provided from a "default" or back-up model that is used as the second model. A default model may be a ready-to-use second machine learning model. One motivation of the use of a default model is to operate the software without latency, because training a new incomplete model from cached query-response pairs in real-time can take time to prepare. A default model on the other hand can be deployed to the ML-assisted service together with a deployed model.

Examples of the second ML model that may be considered "default models" in this manner will now be described. In some embodiments, the second ML model may be trained on a subset of the first dataset used to train the first ML model. The subset of the first dataset used in the second dataset may be training data from the first dataset that is on average of lower quality compared to the average quality of the first dataset as a whole. In another example, a second machine learning model may be provided that has just been trained on non-confidential data. In other words, the second machine learning model may be trained on training examples from the first dataset that are on average less confidential compared to the average confidentiality level of the first dataset as a whole. Put another way still, the first dataset may comprise confidential training data that is not comprised in the second dataset. In this way, a default model of lower quality and/or lower confidentiality may be substituted in for the first machine learning model if an extraction attack is suspected.

In other examples, a default model may be trained using training data from the first dataset, the values of which have been offset with random offset values. In this way a default model may be provided that is of lower quality to the first model, but that is still capable of providing responses across the full input feature space, whilst still mitigating or delaying an extraction attack. In other examples, a default model may be trained on synthetic data, or data that is publicly available. In this way, service is maintained without leaking any sensitive or otherwise valuable data.

As noted above, an advantage of using a default model as the second model is that, as it does not require any training or cached query-response pairs, it can be trained and deployed at the same time as the first model and this reduces latency, e.g. the "default" second model can be deployed instantaneously, without any delay, in the event that the user is identified as a potential adversary.

It is noted that step <NUM> may comprise selecting the second model from a plurality of models. For example, there may be one or more default models and/or one or more incomplete models prepared as described above, and in step <NUM> one of the plurality of models may be selected as the second model. For example, as described below, initially a default model may be selected whilst an incomplete model is trained. Once trained, the incomplete model may then be used as the second model. In another example, the second model may be retrained/replaced over time, as an extraction attack progresses.

Turning now to other embodiments, in some embodiments, the training of the second model may be triggered responsive to the step of identifying <NUM> the user as a potential adversary. For example, the method <NUM> may comprise, responsive to identifying the user of the API as a potential adversary, training the second machine learning model on the second dataset. In examples where the second dataset comprises cached-query response pairs, this has the advantage of triggering training only when the user is identified as a potential adversary. This ensures the second model is trained on an up-to-date list (or the most up to date list possible) of the cached queries that the user has made and also ensures that models are not trained unnecessarily, e.g. for non-adversaries.

In embodiments where the second dataset comprises query-response pairs from previous queries requested by the potential adversary through the API, as stored in a cache, the second machine learning model may be trained in an incremental manner on the previous queries, as they are cached. The training may take place in a pre-emptive manner. For example, the second machine learning model may be trained responsive to an estimation of a first data extraction level being above a first threshold data extraction level and/or responsive to a first estimation of likelihood that the user of the API is actually an adversary being above a first likelihood threshold. The first data extraction level and first likelihood threshold may thus correspond to the "preparation" thresholds at which a second model is prepared for use, e.g. trained, in the event that the user is flagged as a potential adversary. The first data extraction level and first likelihood threshold can be set by a human engineer and specified in a defence policy rule.

In some embodiments, a combination of the default and incomplete models above may be used to respond to queries of an identified potential adversary in step <NUM>. For example, a default model may initially be used whilst an incomplete model is trained for the user on the user's previous query-response pairs. As such, in embodiments where the second machine learning model is trained on cached query-response pairs, in response to a query from the potential adversary through the API, the method <NUM> may comprise providing a response from a third machine learning model instead of the first machine learning model, whilst the second machine learning model is being trained. There may be a delay between identifying the user as a potential adversary and training the second machine learning model. Thus, in order to prevent further information leakage, the third model may be deployed whilst the second model is trained. The third model may be a "default" model, as described above with respect to embodiments of the second model. The third model may be any type of supervised machine learning model, as described above with respect to the first and second machine learning models. For example, the third model may have been trained on low confidential data from the first dataset; low quality data from the third dataset; synthetic data; data from the first dataset offset with random offsets; data that is publicly available, or any other data that is less valuable, as described above.

As noted above, the training of the second machine learning model may be performed in a pre-emptive manner, before the user is identified, or flagged as a potential adversary. As noted above, the training may be performed responsive to an estimation of a first data extraction level being above a first threshold data extraction level and/or a first estimation of likelihood that the user of the API is actually an adversary being above a first likelihood threshold.

In some embodiments, thresholds are also defined at which the second machine learning model should be deployed. For example, the step of providing a response from a second machine learning model instead of the first machine learning model may be performed responsive to an estimation of a second data extraction level being above a second threshold data extraction level, and/or a second estimation of likelihood that the user of the API is actually an adversary being above a second likelihood threshold. The second threshold data extraction level and the second likelihood threshold may be considered "deployment" thresholds for the second model, e.g. the thresholds at which the second ML model is deployed in place of the first ML model. The use of two thresholds in this manner allows a second model to be trained in preparation for deployment in an appropriate time-frame. The second threshold data extraction level and the second likelihood threshold may be set by a human engineer and defined in a defence policy rule.

Thus, there is described herein, reactive methods and apparatuses for protecting a first ML model from a suspected extraction attack whilst maintaining service to a user.

Turning now to <FIG>, which illustrates an ML assisted service <NUM> according to some embodiments herein. The ML assisted service <NUM> hosts a first machine learning model <NUM> that is queryable over an API managed by a query handler <NUM>. The ML assisted service <NUM> receives queries from a plurality of genuine users <NUM>, as well as a potential adversary <NUM>. An ML Protector function <NUM> is used to manage the response of the ML assisted service <NUM> towards the potential adversary <NUM> in order to protect the first machine learning model against the potential adversary <NUM> querying the first machine learning model through the API in order to build up a database of query-response pairs.

In this example, the ML-assisted service <NUM> also comprises a model extraction detector function <NUM> and a per-user cache <NUM>, e.g. a database of query-responses.

The ML-assisted service <NUM> may be any service containing a first ML model <NUM>. Upon initialization of the ML-assisted service <NUM>, the first ML model is deployed model as a part of the ML-assisted service. For example, the ML-assisted service may include an analysis engine for measuring an impact of LTE eNodeB upgrade, which uses a Machine Learning model to provide an intervention analysis.

The ML-assisted service <NUM> provides an exposed API for its users to access its functionality. One example of the software is the Machine Learning as a Service (MLaaS) platforms operated by cloud vendors providing recommendations via Representational State Transfer (RESTful) API or a small software function providing predictions via an API. In more detail, in this embodiment, the ML assisted service <NUM> comprises the following functions.

The first ML model <NUM>: A Machine Learning model that is deployed to the ML assisted service. The first ML model is trained based on confidential data samples. The first ML model may also be referred to as the original model, confidential model or protected model.

Second ML model <NUM>: A surrogate Machine Learning model that is designed to protect against an adversary extracting more information from the deployed model, as described in detail above.

User <NUM>: An authorized user of the ML-assisted service. A user may pay per usage of the ML-assisted service or for a periodic subscription to the service.

Potential Adversary <NUM>: A user that might be performing a model extraction attack, e.g. a potential adversary.

ML-protector <NUM>: This function intercepts incoming ML inference queries and enforces defense policy rules in order to protect the first ML model <NUM> from model extraction attacks. The ML-protector <NUM> comprises:.

Security operation Center <NUM>: A centralized unit that manages security issues by analyzing attacks and deciding the appropriate actions to take.

The ML-assisted service <NUM> may form part of an apparatus such as the apparatus <NUM> described above. For example, the first ML model, the second ML model and/or any other data described with respect to the ML-assisted service <NUM> may be stored in the memory <NUM> of the apparatus <NUM>. The processor <NUM> of the apparatus <NUM> may be configured to perform any of the operations of the functions described herein with respect to the ML-assisted service <NUM>.

The functions above operate in the following manner. The detector function <NUM> performs step <NUM> of the method <NUM> described above and identifies a user as a potential adversary <NUM>. In the embodiment of <FIG>, the detector <NUM> performs step <NUM> based on techniques such as estimated data extraction levels and/or abnormal query pattern detection as described above. Whenever there are incoming queries from users <NUM>, the detector <NUM> updates the estimated data extraction levels of each monitored user. The detector <NUM> determines how to respond to each incoming query based on the determined data extraction levels, in order to react proportionally to the potential threat. Rules for how to respond, e.g. defence policy rules, are provided to the ML protector by a security operation centre <NUM> and may be set e.g. by a human-engineer. Based on the defense policy rules, the ML assisted service <NUM> will provide a response either from the first ML model <NUM>, or from a second ML model <NUM>. In other words, if the estimated data extraction level of a user <NUM> becomes too high then the user will be identified <NUM> as an adversary <NUM> and the model handler <NUM> will perform step <NUM> of the method <NUM> and replace the first ML model with the second ML model, in order to protect the first ML model against further data extraction by the identified potential adversary <NUM>.

In this embodiment, a second ML model is used that is trained using cached query-response pairs stored in the per-user cache <NUM> (e.g. an incomplete model as described above). As noted above, the ML-assisted service <NUM> uses a set of defense policy rules (obtained from security operation center <NUM>) for determining the level of severity of a model extraction attack and for defining how to react. A defense policy rule may include a component or rule for preparation and training of the second ML model and a component for execution of the second ML model. Each component may contain its own threshold, for example, a first threshold data extraction level defining a threshold at which to prepare e.g. begin to train the second ML model on the cached query-response pairs and a second threshold data extraction level defining a threshold at which to being to use the second ML model when replying to user-queries in place of the first ML model. The main motivation of having the first threshold data extraction level is to prepare the second ML model in time to operate the ML-assisted service <NUM> without latency.

Recall that one objective of embodiments herein is continuous service to the API user to minimize risk of negative impact from false positive extraction alerts. Latency to deploy a second ML model leads to the following dilemma: if the service waits until the second ML model is trained in order to serve requests, the delay to the user can negatively affect the system; on the other hand, if the service continues to serve requests using the first ML model whilst the second ML model is being trained, then if the potential adversary <NUM> is a real adversary, there is the risk that they will be able to complete the extraction attack before the second ML model is in place. Consequently, the defense policy rules may comprise options to start preparing and training the second ML model at a lower data extraction level (the first threshold data extraction level) than the threshold for deployment of the second ML model is reached (second threshold data extraction level).

If the first ML model is of a type that supports incremental training (aka. online training), then at the first threshold data extraction level (preparation threshold) a second ML model of the same type as the first ML model may trained on all data points in the cache up to that point, and from then on it can be trained, incrementally, on further query points until the second threshold data extraction level is reached and the switch is made. Thus, latency will be minimal as it only requires incremental training using the last data point to make the switch.

If the first ML model (the model being protected) does not support incremental/online training, as is the case, e.g., for traditional ML models like Random Forest, then there are three options options: i) use a model of a different type than the original first ML model, as the second ML model, ii) use the same type of model as the first ML model as the second ML model, on data points cached up to a point preceding the switch over time (e.g., missing out on the most recent queries) or iii) use the same type of model as the first ML model as the second ML model but substitute a third ML model for the first ML model (e.g. in the form of a default model) whilst the second ML model is being trained.

For option (i) there is a potential discrepancy between the first ML model and the second ML model due to the differences arising from using a different model type, while (ii) has a discrepancy as it does not include all of the cached query-response data points for the user. A similar trade off applies to the use of a default models in option iii), which if used lack information about which queries have been performed previously, and may thus exhibit discrepancies to what the user/adversary has seen previously. In any event the advantages of all three options are that the user is prevented from extracting further information from the model, with minimal service interruption to the user.

Turning now to <FIG> which illustrates of a set of example defense policy rules <NUM>, <NUM>, <NUM>. A rule may comprise components to related to when to prepare and execute the second ML model. In this example, the thresholds are described with respect to data extraction level (as described above) however it will be appreciated that this is merely an example and that preparation and execution thresholds could equally be set based on other extraction progress estimations or likelihood measures relating to attack progression /likelihood of the user being an adversary.

As an example, a defense policy rule may comprise the following fields: Name: A label for the defense policy rule. One example is to use severity levels, which could be categorized as: warning, severe and critical. For example, the warning level may refer to a situation that is concerning, the severe level may refer to a situation where the first ML model may be under attack, and the critical level refers to a situation where the functionality of the first ML model is nearly (fully) extracted by a model extraction adversary.

An example rule may be:
"Name: Warning. Threshold to prepare: <NUM>%. Threshold to execute: <NUM>%. Incompleteness ratio: <NUM>%". This rule can be interpreted as: "In a warning situation, prepare an incomplete model (for use as the second ML model described above) with <NUM>% incompleteness ratio (e.g. <NUM>% of the training dataset is comprised of previously submitted query response pairs and where the rest of the data is selected from a pool of data examples). The incomplete model should be prepared if the monitored user's extraction progress has reached the "Threshold to prepare", which in this example is set at <NUM>% extraction. The example rule then specifies that a response should be provided from the incomplete model (e.g. the second model should be substituted for the first model) when the monitored user's extraction progress reaches <NUM>% extraction.

To further illustrate this example rule, if it was detected that a potential adversary has reached <NUM>% attack progress by sending <NUM> query-response pairs, the service will start to prepare a new incomplete model. The new incomplete model will be built using <NUM> data samples, which consists of <NUM> query-response pairs (<NUM>%) and <NUM> confidential data samples (<NUM>%). The new incomplete model will serve the potential adversary's query requests when the adversary reaches <NUM>% attack progress.

As illustrated in <FIG>, the severity levels for defense policy rule names may be categorized as: warning, severe and critical; where warning level refers to a situation that is concerning, severe level refers to a situation that Machine Learning functionality may be under attack, and critical level refers to a situation that Machine Learning functionality is nearly extracted by model extraction adversaries. These are merely examples however, and the skilled person will appreciate that different rules with different names and/or thresholds may be implemented to those in <FIG>, dependent on the particular circumstances and system requirements.

Turning now to <FIG>, which illustrates an example method that may be performed by the ML-assisted service <NUM> described above.

At <NUM>, the ML-assisted service receives a query with feature vector vq from a user <NUM> or group of users, U, to the first ML model. For example, a user <NUM> sends a query via the network API of the first ML model or to an offline software that contains the first ML model.

At <NUM>, the ML-assisted service determines whether the incoming query is already cached for user(s) U.

At <NUM>, if the query is cached, the ML-assisted service provides a cached response to the incoming query.

At <NUM>, the data extraction level (e.g. extraction progress, P) is estimated e.g. using a technique such as that described in Kesarwani, Manish, et al. (<NUM>) referenced in the background section, to determine how much (P) of the model has been extracted and if user(s) U is attempting model extraction.

At <NUM>, the system determines whether, in case this is an attack, the data extraction level has reached a first threshold data extraction level (preparation threshold, threshold T<NUM>) at which a level, a second ML model in the form of an incomplete MI model needs to be prepared.

At <NUM>, if a second ML model needs to be prepared, then a separate asynchronous thread is spawned to train and deploy an incomplete model. These steps are shown as <NUM>-<NUM>. As described earlier, a set of policies may define these thresholds and some conditions for how the incomplete models should be prepared (incompleteness ratio). A third ML model such as a default model described above may be deployed whilst the second ML model (incomplete model) is being trained. In such examples, the incomplete model trained in <NUM> and <NUM> will replace the default model after it is trained. This can be an iterative process, with steps <NUM> and <NUM> being repeated later, and the incomplete model replaced again.

At <NUM>, the ML-assisted service performs step <NUM> of the method <NUM> described above and identifies whether the user who sent the incoming query is a potential adversary. This step uses at least one model extraction detection technique in order to estimate the potential adversary's attack progress. As an example, this step may use the techniques described in Kesarwani, Manish, et al. (<NUM>) referenced in the background section to estimate the data extraction level and may additionally weigh in results from techniques such as Out-of-Distribution detection or query pattern anomaly detection described above.

If the combined evidence indicates the user is a potential adversary (e.g. indicates a potential attack), then the method continues at <NUM> to choose appropriate action. The system may also send a warning to the Security operation Center, for further analysis and actions.

If the user is not identified as a potential adversary in step <NUM>, then a normal response is returned in <NUM> from the first ML model to the incoming query.

At <NUM>, if the user is identified as a potential adversary, but the extraction progress as not yet reached a second threshold data extraction level (action threshold), then the ML-assisted service may still provide a normal response <NUM> (This is a conservative choice, with lowest risk of degrading the service. ) However, if the extraction progress has reached the second threshold data extraction level (action threshold), then the method proceeds to step <NUM>.

At <NUM>, the ML-assisted service provides a response from a second ML model instead of the first ML model (e.g. step <NUM> of the method <NUM> described above is performed).

Turning now to <FIG>, which shows a flowchart illustrating a method of providing a response from a second ML model according to some embodiments herein. The method may be performed as part of step <NUM> of <FIG> in order to determine which type of model to substitute for the first ML model in step <NUM> of the method <NUM> above.

In this embodiment, the second ML model is an incomplete model, trained on cached query-response pairs as described above. In other words, an incomplete model that is expected to be execute according to the defense policy rule. This model is built on query-response pairs between the ML-assisted service and a potential adversary; it will be trained while the service is up and running. Incomplete models were described in detail above with respect to <FIG> and the detail therein will be understood to apply equally to incomplete models described with respect to <FIG>.

Whilst the second (incomplete) ML model is being trained, responses to queries from an identified potential adversary may be responded to using outputs of a third ML model of one of the following types:.

In the scenario that the second ML model (an incomplete model) is not yet available, e.g. it is in the process of being trained, the ML-assisted service uses a fallback incomplete model or a default incomplete model as follows:.

This process may be repeated for each subsequent query received from the potential adversary.

Turning now to other embodiments, there is also provided a computer program product comprising a computer readable medium, the computer readable medium having computer readable code embodied therein, the computer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform the method or methods described herein.

Thus, it will be appreciated that the disclosure also applies to computer programs, particularly computer programs on or in a carrier, adapted to put embodiments into practice. The program may be in the form of a source code, an object code, a code intermediate source and an object code such as in a partially compiled form, or in any other form suitable for use in the implementation of the method according to the embodiments described herein.

<FIG> shows a carrier <NUM> containing a computer program <NUM>. A carrier may be an electronic signal, optical signal, radio signal or computer readable storage medium. For example, the carrier may be or include a computer readable storage medium, such as a ROM, for example, a CD ROM or a semiconductor ROM, or a magnetic recording medium, for example, a hard disk.

<FIG> shows a computer program product <NUM> comprising non transitory computer readable media <NUM> having stored thereon a computer program <NUM> as described above.

Claim 1:
A computer-implemented method for use in protecting a first machine learning model that is queryable over an application programming interface, API, against an adversary querying the first machine learning model through the API in order to build up a database of query-response pairs, the method comprising:
identifying (<NUM>) a user of the API as a potential adversary; and
in response to a query from the potential adversary, through the API, providing (<NUM>) a response from a second machine learning model instead of the first machine learning model, wherein the first machine learning model has been trained on a first dataset and wherein the second machine learning model has been trained on a second dataset that is different to the first dataset,
wherein the second dataset comprises cached query-response pairs from previous queries requested by the potential adversary through the API, as stored in a cache.