Protecting artificial intelligence IPs: a survey of watermarking and fingerprinting for machine learning

CAAI Transactions on Intelligence TechnologyVolume 6, Issue 2 p. 180-191 REVIEWOpen Access Protecting artificial intelligence IPs: a survey of watermarking and fingerprinting for machine learning Francesco Regazzoni, Corresponding Author f.regazzoni@uva.nl regazzoni@alari.ch orcid.org/0000-0001-6385-0780 University of Amsterdam, Amsterdam, The Netherlands ALaRI – USI, Lugano, Switzerland Correspondence Francesco Regazzoni, University of Amsterdam, Amsterdam, The Netherlands and ALaRI – USI, Lugano, Switzerland. Email: f.regazzoni@uva.nl and regazzoni@alari.chSearch for more papers by this authorPaolo Palmieri, orcid.org/0000-0002-9819-4880 University College Cork, Cork, IrelandSearch for more papers by this authorFethulah Smailbegovic, Delft University of Technology, The NetherlandsSearch for more papers by this authorRosario Cammarota, Intel Labs, San Diego, USASearch for more papers by this authorIlia Polian, University of Stuttgart, Stuttgart, GermanySearch for more papers by this author Francesco Regazzoni, Corresponding Author f.regazzoni@uva.nl regazzoni@alari.ch orcid.org/0000-0001-6385-0780 University of Amsterdam, Amsterdam, The Netherlands ALaRI – USI, Lugano, Switzerland Correspondence Francesco Regazzoni, University of Amsterdam, Amsterdam, The Netherlands and ALaRI – USI, Lugano, Switzerland. Email: f.regazzoni@uva.nl and regazzoni@alari.chSearch for more papers by this authorPaolo Palmieri, orcid.org/0000-0002-9819-4880 University College Cork, Cork, IrelandSearch for more papers by this authorFethulah Smailbegovic, Delft University of Technology, The NetherlandsSearch for more papers by this authorRosario Cammarota, Intel Labs, San Diego, USASearch for more papers by this authorIlia Polian, University of Stuttgart, Stuttgart, GermanySearch for more papers by this author First published: 04 April 2021 https://doi.org/10.1049/cit2.12029AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract Artificial intelligence (AI) algorithms achieve outstanding results in many application domains such as computer vision and natural language processing. The performance of AI models is the outcome of complex and costly model architecture design and training processes. Hence, it is paramount for model owners to protect their AI models from piracy – model cloning, illegitimate distribution and use. IP protection mechanisms have been applied to AI models, and in particular to deep neural networks, to verify the model ownership. State-of-the-art AI model ownership protection techniques have been surveyed. The pros and cons of AI model ownership protection have been reported. The majority of previous works are focused on watermarking, while more advanced methods such fingerprinting and attestation are promising but not yet explored in depth. This study has been concluded by discussing possible research directions in the area. 1 INTRODUCTION The amount of data collected from all kinds of personal devices reaches staggering levels. The data collection process includes distributed and heterogeneous computing devices densely interconnected. The expectation is in the capacity of artificial intelligence (AI) algorithms to leverage the large amount of data and learn to perform tasks commonly associated to intelligent beings, reliably and automatically [1]. We see AI today used in different application domains ranging from robotics in manufacturing, speech processing in retail, knowledge reasoning in healthcare or control in autonomous driving. The development of AI and its applications is supported by the proliferation of both software development frameworks and dedicated hardware accelerators. From the cybersecurity standpoint, AI systems are susceptible to cyberthreats, including piracy, in addition to AI algorithm specific attacks. The rate of cyberattacks increases year after year, with the introduction of more efficient and targeted attacks. Moving forward, it is predicted that damages caused by cyberattacks will rise to six trillions USD by the end of 2021 [2]. Cyberattacks do not target only classical ICT technology. Attacks specifically conceived against AI systems have been proven to be effective and are becoming a real threat, as indicated by some examples recently reported in scientific studies. Goodfellow et al. [3] presented an attack that showed the simplicity of fooling image recognition algorithms based on AI. Originally, the picture of a panda has been recognized with approximately 57% confidence. But, if a carefully constructed noise is added to the picture, the system will recognize a gibbon instead, with a level of confidence up to 99%. Similarly, Sharif et al. [4] demonstrate how to fool facial recognition by constructing adversarial glasses. Image recognition is a fundamental instrument in some applications such as autonomous driving. In this context, it was demonstrated [5, 6] that simple graffiti sprayed on traffic sign or even a single patch placed on specific position on the traffic sign will easily lead to wrong recognition. Functionality is not the only target of cyberattacks in the context of AI. AI system consists of a network having some topology which is trained with the specific data. The exact definition of the utilized network topology, the number of layers, and the specific parameter used to design it are direct consequences of a long and costly process of design space exploration. Similarly, the weight corresponding to the trained network is also the outcome of the training process. This can be a long and resource consuming process, and it certainly requires data that may not be easily available (e.g. it has to be properly and legitimately collected, may be formatted and finally used for training the networks). As a result, a trained AI algorithm is a very valuable intellectual property (IP) for a company realizing it, and as such it becomes a relevant target for adversary. IP piracy, over-production and illegitimate use of a device or software are problems that have plagued the cyberworld for some time. However, in the context of AI, the problem is very recent and is not yet addressed in depth. Even though the problem is relatively new, a number of approaches have been proposed to deal with the protection of AI IP and model ownership. For example, Juuti et al. propose a generic detection mechanism for model extraction attacks through querying [7]. Chakraborty et al. [8] rely on secure hardware to grant services only to authorized end users. The rationale behind the assumption of availability of such trusted hardware devices is as follows: propose a framework in which the deep neural network is firstly trained as a function of a secret key, and then hosted on a public platform. Only devices embedding the secret key will be able to run the trained networks using the published model. However, the large majority of protection mechanisms in the study focus on watermarking and fingerprinting. We review the state-of-the-art of the watermarking and fingerprinting schemes designed for protecting AI algorithms, and we discuss the problems that are still open and the possible future research directions. Due to the rapidly evolving nature of this domain, we also include in this survey study that, to this date, have not been formally published in peer-reviewed venues, but have appeared on preprint archives, such as the computing research repository or the arXiv open-access preprint repository managed by Cornell University. The rest of the study is organized as follows. Section 2 summarizes the main approaches used in AI and the architectures implementing it. Section 3 introduces the threat model and the background on watermarking and fingerprinting. Section 4 discusses the protection techniques proposed so far to mitigate IP violations in AI algorithms. Section 5 reports existing attacks applied to IP protection techniques in the context of AI. 2 BACKGROUND ON ARTIFICIAL INTELLIGENCE Research in AI is often described as the study of intelligent agents [9] and, according to Stephanie Dick's chronology, started in the 1950s at Dartmouth college, USA [10]. AI has been widely applied in a variety of settings, from competing against humans in strategic games (most notably Chess [11] and, more recently, Go [12]), to the interpretation of human speech [13], artificial vision [14], autonomous vehicles [15] and countless more. In the following we briefly analyze some of the most widely used AI techniques, including machine learning (ML), deep learning, swarm intelligence, expert systems and intelligent planning, schedule and optimization techniques. 2.1 Machine learning ML is arguably the most researched and studied AI technique. ML techniques can be further subdivided into three paradigms: supervised learning, unsupervised learning and reinforcement learning [16]. While other categories of ML schemes have been proposed, such as transfer learning and online learning, such techniques can be often traced back to a special case of one of the basic three schemes [17]. Fundamentally, ML comprises two main stages: training and testing [18]. In the former stage, a model is trained based on an initial set of known data. Subsequently, the trained model is used to make predictions in the latter stage. In supervised learning, the training data set is composed of labelled data. Supervised learning can be divided into two categories, based on the characteristics of training data [17]. If the training data only comprise discrete values, then the problem is a classification, and the output of the trained model will be a classification which is also discrete. Conversely, if the training data also comprise continuous values then the problem is a regression, and the output of the model is a prediction. In unsupervised ML, training is instead based on unlabelled data. The goal is to find an efficient representation of the unlabelled data, by analyzing its features. For instance, Bayesian techniques can be used to capture hidden variables that underlie such features. Clustering is a form of unsupervised learning aimed at grouping together samples with similar features. Examples of clustering algorithms are k-means, spectrum clustering and hierarchical clustering [19]. Reinforcement learning uses rewards to direct the learning process, where the agent is incentivized to take actions that would maximize the notion of cumulative reward. In reinforcement learning the focus is on finding a balance between exploration (of uncharted territory) and exploitation (of current knowledge) [20]. Markov decision process is an often utilized model for reinforcement learning [20]. 2.2 Deep learning Deep learning is closely related to ML based on artificial neural networks, but uses deeper networks of neurons in multiple layers [21]. Deep learning is aimed at extracting knowledge from data representations that can be generated using ML. The artificial neural network is a system inspired by biological systems, where each node in the network (an artificial neuron) can transmit a signal to other nodes. A deep learning neural network consists of an input layer, a number of hidden layers and an output layer: the word 'deep' refers to the number of layers through which the data is transformed. Each of these layers learns to transform its input data (which is the output of the preceding layer) into a slightly more abstract and composite representation, which is in turn fed as input to the subsequent layer [22]. 2.3 Swarm intelligence Swarm intelligence (SI) can be described as a collective behaviour of self-organized and decentralized systems [23]. The behaviour of ant colonies, flocks of birds, schools of fish, as well as collective microbial intelligence such as bacterial growth all are real-world examples of SI. In an experiment performed by Deneubourg in 1990 [24], a group of ants was given two paths (short/long) that connect their nest to the food location. It was discovered from their results that ant colonies had a high probability to collectively select the shortest path. In computing, the term Swarm Intelligence (SI) was first introduced by Beni and Wang in the context of the cellular robotic systems [25]. In computing, SI is shown by a population of agents that interact locally with one another and their environment. This could be the case, for instance, of autonomous vehicles or peers in a network, which operate without any centralized control system. SI can be further categorized according to the collective strategy used and the specific setting. Most notably: particle swarm optimization, a global optimization algorithm used to solve a problem whose solution can be represented as a point or a surface in an n-dimensional space [26]; ant colony optimization (ACO), which aims to find near-optimal solutions to problems that can be represented as graph optimization problems [27] and P2P SI, sometimes referred to as swarm casting, the application of SI to peer-to-peer P2P file sharing systems [28]. 2.4 Expert systems Expert systems (ES) aim to emulate the human ability to make decisions in specific contexts. The goal of ES is therefore to solve complex problems by following a line of reasoning that is derived from human knowledge. This reasoning is normally represented by if–then–else statements, instead of conventional procedural code [29]. Fundamentally, a knowledge-based system is based on two main components [30]: a knowledge base and an inference engine. The knowledge base is the set of rules that have been extracted from human knowledge on the specific setting. The inference engine then applies the rules from the knowledge base to known facts, in order to deduce new facts. Inference can follow a forward strategy, starting with the available data and using inference rules to extract more data (known as forward chaining), or proceed inversely (backward chaining) [31]. More advanced systems also include explanation capabilities, which can motivate particular decisions made by the system. 2.5 Planning, scheduling and optimization The AI research community has also produced extensive results in the area of intelligent planning and scheduling, starting from the 1960s. Most of the published work focuses on the development of algorithms (known as planners) that implement planning models. Although different definitions have been used for planning, Spyropoulos defines planning as 'the process of putting in a sequence or partial order a set of activities/actions to fulfil temporal and resource constraints required to achieve a certain goal' [32]. On the other hand, scheduling is 'the allocation of activities/actions over time to resources according to certain performance criteria' [32]. However, there are many researchers who consider scheduling as a special case of planning [33]. Recently, AI planning researchers have developed algorithms able to manage distributed planning activity, where plan generation can happen concurrently with plan execution. These often require distributed systems of cooperating agents for continuous planning and decision support [34]. Current areas of research include multiagent planning architectures, distributed planning, mixed-initiative planning, distributed scheduling and work-flow management methods. 3 BACKGROUND AND THREAT MODEL The threat model that watermarking and fingerprinting want to address is illegitimate use of IP. In our case, the IP is the trained AI algorithm that is available to the adversary. This can happen, for instance, in the following two scenarios. In the first, the device or the software that implements the AI algorithm and the AI algorithms (and the related model) already trained are available to the attacker, but the exact model and the trained parameters of the model are not known to the attacker. The adversary can be the legitimate user of the device or the software, for instance, after having purchased it. But it is also possible that the adversary has gained access illegitimately, for example, by manipulating an edge or IoT device with embedded AI that is deployed in a hostile environment. In the second scenario, the adversary uses an open source AI framework (such as the PyTorch [35] or Caffe [36] deep learning frameworks), so she/he has full control to the implementation, but uses previously trained models purchased as an IP. Both scenarios are going to be very relevant in the near future, where a significant portion of the AI will be carried out on edge or IoT devices rather than in the cloud. Unlike cloud implementation, edge and IoT devices are more open to piracy and other forms of IP right violations. In all the cases, the goal of the adversary is to gain information about the topology and the training parameters of the algorithms and use them illegitimately, for instance, to illegally copy the trained neural network and redistribute it, to use it in more devices than the purchased ones, or to do any other type of abuse against the intellectual properties right of the legitimate owners. This attack is quite common and can be easily carried out when the AI algorithm does not include any sort of protection. Illegitimate use of AI algorithms can potentially cause a huge economical lost for the owner of the trained AI algorithms, and thus must be avoided. At the high level, the problem is similar to other scenarios where protection of IP is needed, and for which several solutions have been proposed in the past. The main goals of mitigation techniques against IP right violations are (1) try to avoid the misuse of the IP and (2) try to guarantee the identification of parts illegitimately copied. To achieve these goals three main solutions have been proposed in the past: watermarking and fingerprinting. In the reminder of the section we will discuss the high-level properties of each of them. The two main techniques used for IP protections are watermarking and fingerprinting, and they are depicted in Figure 1. FIGURE 1Open in figure viewerPowerPoint Techniques for protecting IPs: watermarking (a) and fingerprinting (b) Watermarking is a technique that aims at proving the authorship. A watermark, as defined by Kahng et al. [37] is a mark that has three main properties: (1) it is embedded in the intellectual properties that have to protect, (2) it is specifically designed to identify the source of the IP and (3) it has to be difficult to be removed. As it can be seen, watermarking does not avoid the illegitimate use of the IP it intends to protect, but it aims at discouraging that behaviour by making a violation easily and certainly identifiable (thus, for instance, suitable to be used in a court to prove ownership). Depending on the type of IP that has to be protected, requirements on watermarking can be more strict. For instance, a watermark for use in an integrated circuit must ensure that the circuit continues to operate correctly after the watermark's insertion. This requirement reduces the design space available to the designer of the watermark, in contrast to watermarks embedded, for instance, into an image. In this case, the watermark can alter the original image up to the point that the introduced alteration is not captured by the user, thus practically transparent. Fingerprinting is a technique that allows to track each individual IP. A fingerprint should guarantee the following three properties [38]: (1) the correct functionality of the IP that has to be protected needs to be preserved, (2) each fingerprint should be unique and associated to only one IP to track the source of the IP right violation and (3) the fingerprint should be maintained in all the illegitimate copies of IP to keep trace of the IP right violation. Watermarking and fingerprinting are often used together. In this case, when the author of an IP suspects an episode of IP thief, he or she uses watermarking to verify if the episode really happened and fingerprinting to track and uniquely identify the source of the IP thief. Previous study often divides IP protection techniques in two categories, according with the type of information that needs to be observable for the verifier. In the so-called 'white-box' setting, the verifier of the watermark has access also to internal information and parameters of the protected IP. In the 'black-box' setting instead, the only information available to carry out the verification is the output of the protected IP. Embedding antipiracy support in the IP is often not sufficient to discourage advanced attackers. The adversary can try to remove them, alter their insertion or to make them somehow ineffective. The strength of the attack depends on the exact capability that the adversary has (for instance, if he/she has access to the module injecting the watermark or the fingerprint, or if he/she just has access to the inputs to the IP). In last part of the study we will revise attacks directed towards IP protection mechanism. 4 PROTECTIONS Distinction between white-box and black-box settings is used also when IP protection techniques are applied to AI. Black-box setting has exactly the same meaning, namely that the verifier can access only the output of the IP. In the white box setting, the verifier has access to the internals of the AI algorithm, and in particular to the exact model used and to the weights. Another common way to classify IP protection techniques applied to AI is by the way in which they are activated [39]. Following this, we can distinguish between feature-based and trigger-based IP protection techniques. In the first approach, the protection technique is embedded into the protected AI model by imposing additional regularization parameters to the weights of the model. In the second approach, the embedding technique is similar to the one used for adversarial training and consists of training the AI algorithms with samples that will be classified using specific labels. Watermarking and fingerprinting techniques can also be classified based on the moment in which they are embedded into the AI algorithm. The embedding can be carried out in three main moments [40]: (1) during the training phase, when the AI algorithm is trained and the labels of the model are available and can be modified by the IP protection algorithm; (2) during the fine-tuning phase, when a pretrained network is used and the labels of the model are already initialized, but the fine-tuning of the AI algorithm has to be carried out to tailor the model to the specific task; (3) after the training phase is completed and when the labels of the model are not available using the 'distilling' [41] approach to train a network without label and (iv) after the model and its weight has been finalized. We will use these categories to classify the IP protection techniques that we review in this section. In the rest of this section we will revise the main watermarking and fingerprinting techniques that have been presented in the context of AI. The surveyed techniques are also summarized and compared in Table 1. TABLE 1. The schemes surveyed, categorized according to the AI technique they focus on (e.g. deep learning or ML), the access required to the verifier (white or black box) and the moment of application (training, fine-tuning, distilling or after the model has been finalized) Scheme AI Access Stages of application White box Black box Training Fine-tuning Distilling After Watermarking Uchida et al. [40] Deep neural networks ✓ ✓ ✓ ✓ Adi et al. [42] Deep neural networks ✓ ✓ Zhang et al. [43] Deep neural networks ✓ ✓ Guo et al. [44] Deep neural networks ✓ ✓ Fan et al. [39] Deep neural networks ✓ ✓ Namba et al. [45] Deep neural networks ✓ ✓ DeepSigns [46] Deep neural networks ✓ ✓ ✓ Sakazawa et al. [47] Deep neural networks ✓ ✓ Li et al. [48] Deep neural networks ✓ ✓ Le Merrer et al. [49] Deep neural networks ✓ ✓ BlackMarks [50] Deep neural networks ✓ ✓ Fingerprinting DeepMarks [51] Machine learning ✓ ✓ IPGuard [52] Deep neural networks ✓ ✓ 4.1 Watermarking Watermarking applied to AI shares most of the requirements of watermarking applied to other domains, with mostly two notable exceptions [40]. In all cases, watermarking has to be secure, be capable of storing all required information and has to be included and extracted in an efficient way. Specifically to the AI domain, the watermark is allowed to the specific parameters of the model. However, the performance of the watermarked model has to be comparable with the original one. Also, the watermark should be robust against modification of the models and should propagate into the novel models. Watermarking applied to AI appeared for the first time when Uchida et al. [40] proposed to use digital watermarking technology to protect the trained models of a deep neural network from IP theft and to detect infringement of their use. The authors propose to embed the watermark during the training phase of the network. The choice is done in order to avoid that a direct modification of the weights within the network would negatively affect the performance of the model. The watermark consists of a binary string that is embedded into the weights of selected layers of the AI algorithm implementation. Practically, this is achieved using a specific parameter, called regularizer that is added to the cost function of the model to regularize the mean of the weights. Because of the presence of the regularizer, the weights of the model are biased, and so are their distributions. The watermark is extracted by projecting the means of the weight in specific layers of the model using the embedding regularizer. The watermark can be embedded during each of these three phases: training, fine-tuning and distilling. Access to internal layers of the model is required to complete the verification step, thus the approach belongs to the 'white-box' watermarking. The authors experimentally measured that the performance of a deep neural network was not negatively affected by the inclusion of this type of watermark and that the watermark was not removed after fine tuning or compression of the model (reported results show that the watermark is still complete when up to 65% or the parameters are removed). Adi et al. propose to utilize the over-parameterization of a model as a watermarking technique [42]. Normally over-parameterization is considered a weakness (from a security perspective) as it allows backdooring [53], that is, the training of a model with the purpose of deliberately outputting specific incorrect labels for a particular set of inputs. Adi et al. argue that this technique can also be used to hide watermarking that is resistant to removal. As such, the watermark is embedded into the model during training, and the authors demonstrated that the watermark is not affected by fine-tuning, including adversarial fine-tuning aimed at watermark removal. The proposed verification step includes the presence of a trusted third party, which is involved in all the interactions needed to complete it. The participation of such a third party guarantees the security of the verification process, but it can be quite costly. A benefit of this model is that it follows the black-box paradigm, and therefore allows public verifiability without the need to access the network weights. Zhang et al. [43] also explored watermarks implementing the black-box paradigm in deep neural networks. The basic idea behind the approach is to exploit the intrinsic capabilities of the deep neural network to automatically learn specifically designed patterns—which serve as watermarks—and to generate specific predictions for them. To exploit this capability, the embedding of the watermark is carried out during the training phase. The authors explore three different ways for embedding watermarks: include during the training meaningful data samples to which unique features are superimposed and use them as watermark; include data samples not related to the current task during the training phase and use them as watermark; include noise during the training phase and use it as watermark. The key for the watermark is composed by the pairs patterns, predictions that are known by the legitimate owner of the deep neural network. The verification of the watermark is carried out by performing normal requests to the deep neural network using the watermarking patterns as inputs and matching the results with the expected predictions. The proposed approach is evaluated using benchmark data sets taken from image processing, and the watermarking technique is verified to be robust against classical watermark removal attacks including fine-tuning, parameter pruning and model inversion attacks. A similar approach to that of Zhang et al. is proposed by Li et al. [48]. The proposed framework uses an encoder to embed a watermark in the dataset. The encoder combines the regular input dataset with a 'logo', and is targeted at making the final watermarked images barely distinguishable from the original images. The discriminator is trained to distinguish between training samples and 'trigger' samples embedded with the watermark. The model is trained so that a specific label is assigned to watermarked inputs – and this backdoor provides the basis for model ownership claims. The work by Sakazawa et al. [47] introduces a novel idea for