# Multipurpose Watermarking Approach for Copyright and Integrity of Steganographic Autoencoder Models

Authors: Wei Gu; Ching-Chun Chang; Yu Bai; Yunyuan Fan; Liang Tao; Li Li
Journal: Security and Communication Networks
 (2021)

---

## Abstract

With the great achievements of deep learning technology, neural network models have emerged as a new type of intellectual property. Neural network models’ design and training require considerable computational resources and time. Watermarking is a potential solution for achieving copyright protection and integrity of neural network models without excessively compromising the models’ accuracy and stability. In this work, we develop a multipurpose watermarking method for securing the copyright and integrity of a steganographic autoencoder referred to as “HiDDen.” This autoencoder model is used to hide different kinds of watermark messages in digital images. Copyright information is embedded with imperceptibly modified model parameters, and integrity is verified by embedding the Hash value generated from the model parameters. Experimental results show that the proposed multipurpose watermarking method can reliably identify copyright ownership and localize tampered parts of the model parameters. Furthermore, the accuracy and robustness of the autoencoder model are perfectly preserved.


---

## Body

## 1. Introduction

The latest achievements in deep learning (DL) have gained remarkable success in a number of fields [1], such as speech recognition [2, 3], visual computing [4, 5], and natural language processing [6, 7]. DL methods have been reported to outperform traditional methods substantially [6–10].The production of a deep neural network model is remarkably costly, requiring a great quantity of training data and consuming massive amounts of computing resources and time. If the deep neural network model is maliciously copied, transmitted, or stolen, then the owner will suffer a terrible loss. Therefore, it is crucial to prevent the copyright and integrity of such intellectual property (IP) from being violated. The recent development of various watermarking methods has triggered research attention in addressing the IP issues over DL models [11–13].The following real-world application scenario is considered. For example, an organization has developed a product based on DL technology and put it into the market to achieve profitability. This action of the organization indicates that the purchaser of the product has the right to use the service within the scope allowed by law. However, if the customer uses this product for commercial purposes or provides it to other organizations, such use will be considered a serious violation. So, protecting the IP of the product is a difficult problem that must be solved in this scenario.Some previous works [14–18] applied DL in many watermarking systems for images, videos, and audios to achieve better experimental results. However, rather than the used DL model, these works aim to protect multimedia copyright information. This condition motivated the current investigation regarding the IP protection of DL models.First, Uchida et al. [14] and Nagai et al. [19] proposed a generic watermark embedding framework based on deep neural networks (DNNs) using a parametric regularizer; thus they could embed watermarks in the training phase of the model. Wang et al. [20] extended the work of Uchida et al. by adding a separate neural network to form a relationship mapping between the network weights and the watermark information. However, such an improvement cannot withstand the ambiguity attacks. To solve this problem, Rouhani et al. [21] proposed an end-to-end IP protection framework: DeepSigns that allows developers to insert watermarking information systems into DL models before distributing models. Fan et al. [22] applied their proposed DNN copyright verification algorithm for antiforgery authentication about passports. This technique remains robust after the network is modified, especially for DNN ambiguity attacks. These articles mainly discussed the issue of IP certification through watermarking DNNs in the extensively used white-box scenario. The accuracy of the watermark model remains unaffected. However, it is necessary to know all the DNN parameters to extract the watermark information during ownership verification of the DL model. The white-box technique restricts its universal use in any scenario.IP protection in black-box scenarios is proposed in [15, 16, 23–27]. Compared with the white-box technique, the black-box watermarking methods are suitable for DNN model protection. The DNN model should be able to provide API services during its ownership verification; this model can also withstand statistical attacks [15, 16].Adi et al. [4] selected hundreds of abstract images and attached labels as a trigger set and simultaneously utilized it with other training sets to train the classification neural network. Zhang et al. [25] proposed that watermarks embedding can be achieved in conjunction with a remote verification mechanism. Next, they designed an algorithm that can identify the ownership of DL models, which in turn can be trained while learning user-exclusive watermarks. Finally, they executed prespecified predictions when observing watermark modes at inference. Zhao et al. [26] proposed a watermarking framework for GNNs, in which an indeterminate figure related to features and labels is initialized as the trigger input. By training the main GNN model with the trigger figure, the watermark can be distinguished from its result during certification. Wu et al. [28] introduced a novel digital watermarking framework suitable for deep neural networks that output images as a result. All the output of the images from a watermarking DNN in this framework will contain an exclusive watermark. The basic idea of these methods is to introduce backdoor or Trojan horse watermarking [17, 29, 30] to certify the ownership of DL models, and only legitimate users can extract the full watermark.In recent years, information hiding about DNN has become a popular research issue [18, 27, 31–37]. Kandi et al. [6] proposed an innovative learning-based autoencoder convolutional neural network (CNN) for nonblind watermarking, which adds an additional dimension to the use of CNNs for secrecy and outperforms methods using traditional transformations in terms of both agnosticism and robustness. Hayes and Danezis [37] used adversarial training techniques to learn a steganographic algorithm for the discrimination task. However, the DNN model of information hiding is radically different from other models in that if the DNN model is tampered, it means that the model parameters are also modified, reducing the accuracy of the image watermark detected by the model.The abovementioned methods focused on protecting the model copyright. Meanwhile, the current study considers not only model copyright but also model integrity. Thus, in this paper, we propose a novel multipurpose watermarking method for protecting the copyright and integrity of a steganographic autoencoder network.The main contributions of this work are summarized as follows:(I)
A method to protect DNN models by using multiple watermark association mechanisms is proposed. This method verifies not only the copyright information of the DNN model but also its integrity and can locate model tampering parts.(II)
The proposed work can ensure the accuracy of the image watermark extracted by the model according to the correlation between the model and image watermarks.(III)
The information hiding model adopts the average pooling method. Therefore, the designed symmetrical modification mechanism can ensure that the parameter mean value of the modified layers in the model remains relatively stable, so it has minimal impact on the average pooling results and ensures the stability of the model output.The rest of this paper is structured as follows. First, we briefly describe HiDDen model and embedding strategy in Section2, and then we detail the proposed method in Section 3 and demonstrate extensive experiments and analysis in Section 4. Finally, we conclude this paper in Section 5.


## 2. Related Works

### 2.1. HiDDen Model

A robust DNN model for data hiding was designed [10]. This approach generates visually indistinguishable watermarked images using an encoder given the input information and cover image. A decoder is also used to recover the input information from the encoded image. This model is robust against dropout, crop-out, cropping, Gaussian noise, and other image attacks, as shown in Figure 1.Figure 1
HiDDen flowchart.The HiDDen model comprises the following four main components: an encoderEθ, a decoder Dϕ, a parameter-less noise layer N, and an adversarial discriminator Aγ. First, the watermark information W1in and the cover image Ico (size C×H×W1) are fed into the encoder Eθ. The encoder Eθ then applies convolutions to the cover image to form a few intermediate representations and embeds the watermark information of length L in the encoder. After multiple convolutional layers process, the encoded image Ien is produced. Afterward, the noise layer N adds noise to the encoded image Ien to produce a noisy encoded image Ien′. Next, the noise-laden encoded image Ien′ is fed to the decoder Dϕ. This decoder Dϕ then applies some convolutional layers to generate L feature channels in these intermediate representations. Global spatial average pooling and a fully connected layer are, respectively, applied to initialize a message vector M of the same size and then activated with a fully connected layer to decode the watermark W1out. The adversary Aγ is analogous to a decoder that serves to discriminate whether an image is an encoded image or a cover image and outputs a binary classification. The total loss function LTotal comprises LW1, LG, and LI and the associated loss function is defined below.The lossLI between the original image Ico and the encoded image Ien (image distortion loss) is defined by (1)LIIco,Ien=Ico−Ien22CHW1.The loss of the watermark informationW1in and the decoded information W1out (watermark distortion loss) LW1 is defined as(2)LW1W1in,W1out=W1in−W1out22L.TheLG (adversarial loss) for the adversarial discriminator Aγ to detect whether an image is a watermarked image is defined as(3)LGIen=log1−AIen,where AI∈0,1 is the probability of the watermarked image.The classification lossLA of the adversarial discriminator Aγ is defined as(4)LAIco,Ien=log1−AIco+logAIen.In the original paper stochastic gradient descent onθ and ϕ is performed such that the total loss function LTotal is optimal in the following cases:(5)EIco,W1LW1W1in,W1out+λILIIco,Ien+λGLGIen,where λI and λG are regulators. Moreover, EIco,W1LAIco,Ien is minimized by training Aγ. At this point, the final decoded image is the watermarked image Iem.


### 2.2. Embedding Strategy for Model Watermarks with Modified Parameters

HiDDen trains robust coders and decoders using DNNs, but DNNs also require copyright protection. Thus, embedding watermark into a DNN is an excellent approach to prove its copyright ownership. The most typical method of watermark embedding is the parameter regularizer method adopted by Uchida et al. [14], by which a novel term is added into the initial cost function for the initial assignment. The cost function Eτ with a regularizer is defined as(6)Eτ=EOτ+λERτ,where EOτ is the original loss function, and ERτ is the regularization term that imposes certain restrictions on parameter τ, and λ is an adjustable parameter.Compared with the standard regularizer, the forced parameterw of this regularizer has a certain statistical deviation, which is used as the embedded watermark. This regularizer is called the embedded regularizer. Given a (mean) parameter vector τ∈Rm and an embedding key X∈RT×M，the watermark can be extracted only by using τ and X, and the threshold is set to 0. Specifically, the extraction of the j–th bit watermark is(7)wj=sΣiXjiτi,where sx is a step function:(8)sx=1,ifx≥0,0,else.The flow of the algorithm is a binary classification problem with a single-layer perceptron. This means that it is straightforward to set up the loss functionERτ for the embedding regularizer by using (binary) cross-entropy as a direct approach:(9)ERτ=−∑j=1Twjlog2yj+1−wjlog21−yj,where yj=σΣiXjiτi and σ. is the sigmoid function:(10)σΣiXjiτi=11+exp−ΣiXjiτi.The loss function is applied to updateτ instead of X. τ is the embedded target, and X=xji is the embedding key, xji∼N0,1. xji is embedded into each element about the parameter τ with random embedding weights.


## 2.1. HiDDen Model

A robust DNN model for data hiding was designed [10]. This approach generates visually indistinguishable watermarked images using an encoder given the input information and cover image. A decoder is also used to recover the input information from the encoded image. This model is robust against dropout, crop-out, cropping, Gaussian noise, and other image attacks, as shown in Figure 1.Figure 1
HiDDen flowchart.The HiDDen model comprises the following four main components: an encoderEθ, a decoder Dϕ, a parameter-less noise layer N, and an adversarial discriminator Aγ. First, the watermark information W1in and the cover image Ico (size C×H×W1) are fed into the encoder Eθ. The encoder Eθ then applies convolutions to the cover image to form a few intermediate representations and embeds the watermark information of length L in the encoder. After multiple convolutional layers process, the encoded image Ien is produced. Afterward, the noise layer N adds noise to the encoded image Ien to produce a noisy encoded image Ien′. Next, the noise-laden encoded image Ien′ is fed to the decoder Dϕ. This decoder Dϕ then applies some convolutional layers to generate L feature channels in these intermediate representations. Global spatial average pooling and a fully connected layer are, respectively, applied to initialize a message vector M of the same size and then activated with a fully connected layer to decode the watermark W1out. The adversary Aγ is analogous to a decoder that serves to discriminate whether an image is an encoded image or a cover image and outputs a binary classification. The total loss function LTotal comprises LW1, LG, and LI and the associated loss function is defined below.The lossLI between the original image Ico and the encoded image Ien (image distortion loss) is defined by (1)LIIco,Ien=Ico−Ien22CHW1.The loss of the watermark informationW1in and the decoded information W1out (watermark distortion loss) LW1 is defined as(2)LW1W1in,W1out=W1in−W1out22L.TheLG (adversarial loss) for the adversarial discriminator Aγ to detect whether an image is a watermarked image is defined as(3)LGIen=log1−AIen,where AI∈0,1 is the probability of the watermarked image.The classification lossLA of the adversarial discriminator Aγ is defined as(4)LAIco,Ien=log1−AIco+logAIen.In the original paper stochastic gradient descent onθ and ϕ is performed such that the total loss function LTotal is optimal in the following cases:(5)EIco,W1LW1W1in,W1out+λILIIco,Ien+λGLGIen,where λI and λG are regulators. Moreover, EIco,W1LAIco,Ien is minimized by training Aγ. At this point, the final decoded image is the watermarked image Iem.


## 2.2. Embedding Strategy for Model Watermarks with Modified Parameters

HiDDen trains robust coders and decoders using DNNs, but DNNs also require copyright protection. Thus, embedding watermark into a DNN is an excellent approach to prove its copyright ownership. The most typical method of watermark embedding is the parameter regularizer method adopted by Uchida et al. [14], by which a novel term is added into the initial cost function for the initial assignment. The cost function Eτ with a regularizer is defined as(6)Eτ=EOτ+λERτ,where EOτ is the original loss function, and ERτ is the regularization term that imposes certain restrictions on parameter τ, and λ is an adjustable parameter.Compared with the standard regularizer, the forced parameterw of this regularizer has a certain statistical deviation, which is used as the embedded watermark. This regularizer is called the embedded regularizer. Given a (mean) parameter vector τ∈Rm and an embedding key X∈RT×M，the watermark can be extracted only by using τ and X, and the threshold is set to 0. Specifically, the extraction of the j–th bit watermark is(7)wj=sΣiXjiτi,where sx is a step function:(8)sx=1,ifx≥0,0,else.The flow of the algorithm is a binary classification problem with a single-layer perceptron. This means that it is straightforward to set up the loss functionERτ for the embedding regularizer by using (binary) cross-entropy as a direct approach:(9)ERτ=−∑j=1Twjlog2yj+1−wjlog21−yj,where yj=σΣiXjiτi and σ. is the sigmoid function:(10)σΣiXjiτi=11+exp−ΣiXjiτi.The loss function is applied to updateτ instead of X. τ is the embedded target, and X=xji is the embedding key, xji∼N0,1. xji is embedded into each element about the parameter τ with random embedding weights.


## 3. Proposed Method

The flow diagram of the proposed algorithm is shown in Figure2, and the details of three watermark embedding and extraction methods are described in this section. The HiDDen model introduced in [10] is selected as a carrier for the model watermarks W2 and W3 to authenticate the integrity of the DNN used for information hiding. The input for the HiDDen network is watermark W1 and cover image Ico, and the output is the watermarked image. It includes three modules: an encoder Eθ, a decoder Dϕ, and an adversarial discriminator Aγ, which can be trained jointly to be able to perform information hiding. Figure 1 shows the process of embedding the image watermark W1 into the watermarked image Iem during the DNN training phase. In order to achieve multiple verifications of model integrity, in this work, we have modified the decoder module for the model by embedding additional model watermarks W2 and W3 to achieve multiple verifications of model integrity. This modification includes not only the model copyright information but also the image watermark information W1 and the Hash values of the model parameters. In the model training phase, the original image is fed to this DNN model for training, and the final output is the watermarked image Iem. Blind detection of watermark information can be achieved in the watermark detection phase by extracting the output image watermark W1 and the model watermark W2. In addition, this work makes it possible to extract the model watermark W3 and identify the tampering location of the DNN model when necessary. Model watermarks W2 and W3 will be embedded in the decoder of this model to protect its copyright.Figure 2
The proposed method framework. (a) Watermark embedding process and (b) watermark extraction process.
(a)(b)The image watermarkW1 includes image copyright, comparison, and redundancy information. A certain region is divided into the other convolutional layers while selecting the fully connected layer in the DNN model to embed the model watermark W2 to calculate the Hash value, which can initialize the model watermark W3. The model watermark W3 will be embedded in the redundancy parameters of the fully connected layer, which corresponds to the redundancy information of the image watermark W1. The model watermark W3 is extracted first to prove the integrity of the DNN model and locate the tampering location. The image watermark W1 and the comparative model watermark W2 can then be extracted and compared to determine the accuracy of the image watermark information. Thus, the copyright information of the image and model can be obtained.


### 3.1. Image WatermarkW1 for the Host Network

The HiDDen model proposed by Zhu et al. [10] is chosen in this work as a carrier. Compared with other models, the HiDDen model has the advantage of robustness to various attacks. The watermark embedded in the input image is referred to as the image watermark W1 in this work. The image watermark W1 comprises the following: image copyright information w11 and validation information w12, W1=w11,w12.


### 3.2. Model WatermarksW2 and W3 in the Network

The ownership of the HiDDen model is further protected from copyright threats to enable cross-validation of watermarked information and identify the tampered location in the model. Suitable parameters in the convolutional and fully connected layers of HiDDen can be used in this work as carriers for model watermarks, thus achieving a small influence on the performance of the HiDDen model and an accurate location of the tampered parameter coefficients of the HiDDen model. To this end, model watermarksW2 and W3 are embedded in the DNN model in this work.


#### 3.2.1. Model WatermarksW2 and W3 Generation

The structures ofW2 and W3 are shown in Figure 3. Their specific compositions are as follows.Figure 3
Composition of model watermarks (W)2 and (W)3.Composition of the model watermarkW2: model copyright information w21 and validation information w12, W2=w21,w12.Composition of the model watermarkW3: the chunked Hash values of all convolutional and fully connected layers constitute model watermark W3=h1,h2,…,h42.


#### 3.2.2. Model WatermarkW2 Embedding Position

The proposed method generally embeds the model watermark on some layers of the network. For example, Uchida et al. [14] chose to embed the watermark on one of the intermediate layers of the network, while Feng et al. [36] embedded the watermark in multiple intermediate layers. Considering the suitable location for embedding the watermark, experiments revealed that embedding the watermark information in the middle layer closest to the output layer has the least impact on the model. Therefore, watermark information is embedded into the fully connected layer of the self-coding network.The HiDDen model has the model parameters of the fully connected layer with sizeL1×L1. Thus, the model watermark W2 with maximum capacity L1×L1 is denoted as W2=W21,W22,…,W2NW2 (maxNW2=L1×L1, NW2≤L1×L1). The length of the model watermark W2 was controlled to L2≪L1×L1 considering the accuracy of the HiDDen model training (avoid excessive increase in watermark capacity). The effect of watermark capacity on the training accuracy of the HiDDen model is shown in Figure 4.Figure 4
The effect of model watermark (W)2 capacity on the training accuracy of the HiDDen model.Each parameter in the HiDDen network is a 32-bit floating-point number, and the watermark is embedded ink decimal places. The imperceptibility of the algorithm increases with k but it is susceptible to truncation errors, weakening the robustness of the watermark extraction. Conversely, the robustness of the algorithm improves as k decreases. However, the accuracy of the HiDDen model is again affected, resulting in a decrease in model performance. Experimental verifications revealed that the performance of the HiDDen model is ideally balanced with the robustness of the watermarking algorithm with k=4. The accuracy of the model for different k values is shown in Figure 5.Figure 5
Model accuracy of different modification bits.


#### 3.2.3. Model WatermarkW2 Embedding Strategy

Given a HiDDen model network with trained parameters, the mission of watermark embedding is defined as the embedding of the model watermarkW2 to value Dk of the k–th decimal place of the fully connected layer model parameters.Maintaining the model accuracy of decoders trained by neural networks is crucial when embedding watermarks. The HiDDen model performs average pooling on all convolutional layers, which reduces the impact on model accuracy if the mean value of the model parameters after watermark embedding is the same as that before embedding. Therefore, in this paper, we propose a symmetric watermark embedding strategy. The mean value is 4.5 assuming that numbers 0 to 9 fit the mean distribution at thek–position. The two states of the watermark are taken as (2, 7), which is a state pair as shown in Figure 6, to ensure that the mean value remains constant and the distance between the numbers is kept at a maximum. The specific embedding method is shown in(11)Dk=2,ifW2i=0,7,ifW2i=1.Figure 6
Model accuracy of different embedding state pairs: (a) (2, 7) state pair, (b) (0, 9) state pair, (c) (3, 9) state pair, and (d) (7, 8) state pair.
(a)(b)(c)(d)The value ofk in this paper is chosen within a median range. Therefore, the presences of the watermark neither affect the accuracy of the model nor are disturbed by quantization errors. At this point, the mean of the k–th bit is 4.5, which is equal to the mean of this bit of the model itself. The experimental data show no effects on the model accuracy when modified to lie in the fourth and subsequent decimal places. The watermark is embedded in all layers with minimal effect due to the slightly low bit count and for the convenience of extracting the model watermark W2, which is embedded in the final fully connected layer in this work.


#### 3.2.4. Model WatermarkW3 Embedding Position and Strategy

This work chunks the convolutional and fully connected layers of the HiDDen model to enable tampering localization. The small size of the block results in the large capacity of the model watermarkW3 and the high accuracy of the HiDDen model integrity certification. In practice, the different parameters can be freely chosen in accordance with the application needs, such as the capacity of the model watermark W2.Step 1.
Calculate the Hash value of each block using the Hash function. These Hash values are known as the model watermarkW3, which is W3=h1,h2,…,h42,Step 2.
Write the Hash value of each chunk to the redundant bits of the fully connected layer. Therefore, the extracted Hash values of each block during HiDDen integrity verification can be compared with the model watermarkW3 for data integrity authentication.
Table1 shows the corresponding experiments for different chunks and lists the effect of different numbers of chunks on model accuracy (the magnitude of change is 0.01).Table 1
Model accuracy with different chunks.
Number of chunks per levelModification rate of blocks (%)Model accuracy (%)2492.47891.141290.201688.4541092.222090.893089.124087.5551092.462091.163090.294088.4781093.772092.013091.354090.84


### 3.3. Watermark Extraction

The three watermarks are extracted in reverse order of embedding, model watermarkW3, model watermark W2, and then image watermark W1.


#### 3.3.1. Model WatermarkW3 Extracting

The model watermarkW3 is extracted according to the embedding rules; the coefficients corresponding to the eight convolutional layers and one fully connected layer in the decoder are found for chunking. The Hash value h1,h2,…,h42 of each block is then calculated and compared to the model watermark W3 stored in the redundant bits of the fully connected layer. If they are equal, then no tampering will occur. Otherwise, the model block corresponding to hi has been tampered.


#### 3.3.2. Model WatermarkW2 Extraction

The watermark is embedded in the fully connected layer in the decoder. The watermark length of the model watermarkW2 is selected as the first L2 model parameter in the fully connected layer in the decoder. The model watermark W2 is then extracted in accordance with (12)W2i=−1,if Dk∈04,1,if Dk∈59.Model watermarkW2 and image watermark W1 have the same validation information w12. Thus, multiple validations of image watermark information and extraction of model copyright information can be achieved by comparing the detected model watermark W2 and image watermark W1.


#### 3.3.3. Image WatermarkW1 Extraction

The watermarked imageIem is decoded into the HiDDen model, which first generates L feature channels using eight convolutional layers. A global spatially averaged pooling is then used to generate watermark vectors of the same size. The performance of the watermark decoder has been continuously improved after uninterrupted iterations of the coefficients in the fully connected layer [38]. Finally, the output image watermark W1 is obtained through the fully connected layer.


## 3.1. Image WatermarkW1 for the Host Network

The HiDDen model proposed by Zhu et al. [10] is chosen in this work as a carrier. Compared with other models, the HiDDen model has the advantage of robustness to various attacks. The watermark embedded in the input image is referred to as the image watermark W1 in this work. The image watermark W1 comprises the following: image copyright information w11 and validation information w12, W1=w11,w12.


## 3.2. Model WatermarksW2 and W3 in the Network

The ownership of the HiDDen model is further protected from copyright threats to enable cross-validation of watermarked information and identify the tampered location in the model. Suitable parameters in the convolutional and fully connected layers of HiDDen can be used in this work as carriers for model watermarks, thus achieving a small influence on the performance of the HiDDen model and an accurate location of the tampered parameter coefficients of the HiDDen model. To this end, model watermarksW2 and W3 are embedded in the DNN model in this work.


### 3.2.1. Model WatermarksW2 and W3 Generation

The structures ofW2 and W3 are shown in Figure 3. Their specific compositions are as follows.Figure 3
Composition of model watermarks (W)2 and (W)3.Composition of the model watermarkW2: model copyright information w21 and validation information w12, W2=w21,w12.Composition of the model watermarkW3: the chunked Hash values of all convolutional and fully connected layers constitute model watermark W3=h1,h2,…,h42.


### 3.2.2. Model WatermarkW2 Embedding Position

The proposed method generally embeds the model watermark on some layers of the network. For example, Uchida et al. [14] chose to embed the watermark on one of the intermediate layers of the network, while Feng et al. [36] embedded the watermark in multiple intermediate layers. Considering the suitable location for embedding the watermark, experiments revealed that embedding the watermark information in the middle layer closest to the output layer has the least impact on the model. Therefore, watermark information is embedded into the fully connected layer of the self-coding network.The HiDDen model has the model parameters of the fully connected layer with sizeL1×L1. Thus, the model watermark W2 with maximum capacity L1×L1 is denoted as W2=W21,W22,…,W2NW2 (maxNW2=L1×L1, NW2≤L1×L1). The length of the model watermark W2 was controlled to L2≪L1×L1 considering the accuracy of the HiDDen model training (avoid excessive increase in watermark capacity). The effect of watermark capacity on the training accuracy of the HiDDen model is shown in Figure 4.Figure 4
The effect of model watermark (W)2 capacity on the training accuracy of the HiDDen model.Each parameter in the HiDDen network is a 32-bit floating-point number, and the watermark is embedded ink decimal places. The imperceptibility of the algorithm increases with k but it is susceptible to truncation errors, weakening the robustness of the watermark extraction. Conversely, the robustness of the algorithm improves as k decreases. However, the accuracy of the HiDDen model is again affected, resulting in a decrease in model performance. Experimental verifications revealed that the performance of the HiDDen model is ideally balanced with the robustness of the watermarking algorithm with k=4. The accuracy of the model for different k values is shown in Figure 5.Figure 5
Model accuracy of different modification bits.


### 3.2.3. Model WatermarkW2 Embedding Strategy

Given a HiDDen model network with trained parameters, the mission of watermark embedding is defined as the embedding of the model watermarkW2 to value Dk of the k–th decimal place of the fully connected layer model parameters.Maintaining the model accuracy of decoders trained by neural networks is crucial when embedding watermarks. The HiDDen model performs average pooling on all convolutional layers, which reduces the impact on model accuracy if the mean value of the model parameters after watermark embedding is the same as that before embedding. Therefore, in this paper, we propose a symmetric watermark embedding strategy. The mean value is 4.5 assuming that numbers 0 to 9 fit the mean distribution at thek–position. The two states of the watermark are taken as (2, 7), which is a state pair as shown in Figure 6, to ensure that the mean value remains constant and the distance between the numbers is kept at a maximum. The specific embedding method is shown in(11)Dk=2,ifW2i=0,7,ifW2i=1.Figure 6
Model accuracy of different embedding state pairs: (a) (2, 7) state pair, (b) (0, 9) state pair, (c) (3, 9) state pair, and (d) (7, 8) state pair.
(a)(b)(c)(d)The value ofk in this paper is chosen within a median range. Therefore, the presences of the watermark neither affect the accuracy of the model nor are disturbed by quantization errors. At this point, the mean of the k–th bit is 4.5, which is equal to the mean of this bit of the model itself. The experimental data show no effects on the model accuracy when modified to lie in the fourth and subsequent decimal places. The watermark is embedded in all layers with minimal effect due to the slightly low bit count and for the convenience of extracting the model watermark W2, which is embedded in the final fully connected layer in this work.


### 3.2.4. Model WatermarkW3 Embedding Position and Strategy

This work chunks the convolutional and fully connected layers of the HiDDen model to enable tampering localization. The small size of the block results in the large capacity of the model watermarkW3 and the high accuracy of the HiDDen model integrity certification. In practice, the different parameters can be freely chosen in accordance with the application needs, such as the capacity of the model watermark W2.Step 1.
Calculate the Hash value of each block using the Hash function. These Hash values are known as the model watermarkW3, which is W3=h1,h2,…,h42,Step 2.
Write the Hash value of each chunk to the redundant bits of the fully connected layer. Therefore, the extracted Hash values of each block during HiDDen integrity verification can be compared with the model watermarkW3 for data integrity authentication.
Table1 shows the corresponding experiments for different chunks and lists the effect of different numbers of chunks on model accuracy (the magnitude of change is 0.01).Table 1
Model accuracy with different chunks.
Number of chunks per levelModification rate of blocks (%)Model accuracy (%)2492.47891.141290.201688.4541092.222090.893089.124087.5551092.462091.163090.294088.4781093.772092.013091.354090.84


## 3.2.1. Model WatermarksW2 and W3 Generation

The structures ofW2 and W3 are shown in Figure 3. Their specific compositions are as follows.Figure 3
Composition of model watermarks (W)2 and (W)3.Composition of the model watermarkW2: model copyright information w21 and validation information w12, W2=w21,w12.Composition of the model watermarkW3: the chunked Hash values of all convolutional and fully connected layers constitute model watermark W3=h1,h2,…,h42.


## 3.2.2. Model WatermarkW2 Embedding Position

The proposed method generally embeds the model watermark on some layers of the network. For example, Uchida et al. [14] chose to embed the watermark on one of the intermediate layers of the network, while Feng et al. [36] embedded the watermark in multiple intermediate layers. Considering the suitable location for embedding the watermark, experiments revealed that embedding the watermark information in the middle layer closest to the output layer has the least impact on the model. Therefore, watermark information is embedded into the fully connected layer of the self-coding network.The HiDDen model has the model parameters of the fully connected layer with sizeL1×L1. Thus, the model watermark W2 with maximum capacity L1×L1 is denoted as W2=W21,W22,…,W2NW2 (maxNW2=L1×L1, NW2≤L1×L1). The length of the model watermark W2 was controlled to L2≪L1×L1 considering the accuracy of the HiDDen model training (avoid excessive increase in watermark capacity). The effect of watermark capacity on the training accuracy of the HiDDen model is shown in Figure 4.Figure 4
The effect of model watermark (W)2 capacity on the training accuracy of the HiDDen model.Each parameter in the HiDDen network is a 32-bit floating-point number, and the watermark is embedded ink decimal places. The imperceptibility of the algorithm increases with k but it is susceptible to truncation errors, weakening the robustness of the watermark extraction. Conversely, the robustness of the algorithm improves as k decreases. However, the accuracy of the HiDDen model is again affected, resulting in a decrease in model performance. Experimental verifications revealed that the performance of the HiDDen model is ideally balanced with the robustness of the watermarking algorithm with k=4. The accuracy of the model for different k values is shown in Figure 5.Figure 5
Model accuracy of different modification bits.


## 3.2.3. Model WatermarkW2 Embedding Strategy

Given a HiDDen model network with trained parameters, the mission of watermark embedding is defined as the embedding of the model watermarkW2 to value Dk of the k–th decimal place of the fully connected layer model parameters.Maintaining the model accuracy of decoders trained by neural networks is crucial when embedding watermarks. The HiDDen model performs average pooling on all convolutional layers, which reduces the impact on model accuracy if the mean value of the model parameters after watermark embedding is the same as that before embedding. Therefore, in this paper, we propose a symmetric watermark embedding strategy. The mean value is 4.5 assuming that numbers 0 to 9 fit the mean distribution at thek–position. The two states of the watermark are taken as (2, 7), which is a state pair as shown in Figure 6, to ensure that the mean value remains constant and the distance between the numbers is kept at a maximum. The specific embedding method is shown in(11)Dk=2,ifW2i=0,7,ifW2i=1.Figure 6
Model accuracy of different embedding state pairs: (a) (2, 7) state pair, (b) (0, 9) state pair, (c) (3, 9) state pair, and (d) (7, 8) state pair.
(a)(b)(c)(d)The value ofk in this paper is chosen within a median range. Therefore, the presences of the watermark neither affect the accuracy of the model nor are disturbed by quantization errors. At this point, the mean of the k–th bit is 4.5, which is equal to the mean of this bit of the model itself. The experimental data show no effects on the model accuracy when modified to lie in the fourth and subsequent decimal places. The watermark is embedded in all layers with minimal effect due to the slightly low bit count and for the convenience of extracting the model watermark W2, which is embedded in the final fully connected layer in this work.


## 3.2.4. Model WatermarkW3 Embedding Position and Strategy

This work chunks the convolutional and fully connected layers of the HiDDen model to enable tampering localization. The small size of the block results in the large capacity of the model watermarkW3 and the high accuracy of the HiDDen model integrity certification. In practice, the different parameters can be freely chosen in accordance with the application needs, such as the capacity of the model watermark W2.Step 1.
Calculate the Hash value of each block using the Hash function. These Hash values are known as the model watermarkW3, which is W3=h1,h2,…,h42,Step 2.
Write the Hash value of each chunk to the redundant bits of the fully connected layer. Therefore, the extracted Hash values of each block during HiDDen integrity verification can be compared with the model watermarkW3 for data integrity authentication.
Table1 shows the corresponding experiments for different chunks and lists the effect of different numbers of chunks on model accuracy (the magnitude of change is 0.01).Table 1
Model accuracy with different chunks.
Number of chunks per levelModification rate of blocks (%)Model accuracy (%)2492.47891.141290.201688.4541092.222090.893089.124087.5551092.462091.163090.294088.4781093.772092.013091.354090.84


## 3.3. Watermark Extraction

The three watermarks are extracted in reverse order of embedding, model watermarkW3, model watermark W2, and then image watermark W1.


### 3.3.1. Model WatermarkW3 Extracting

The model watermarkW3 is extracted according to the embedding rules; the coefficients corresponding to the eight convolutional layers and one fully connected layer in the decoder are found for chunking. The Hash value h1,h2,…,h42 of each block is then calculated and compared to the model watermark W3 stored in the redundant bits of the fully connected layer. If they are equal, then no tampering will occur. Otherwise, the model block corresponding to hi has been tampered.


### 3.3.2. Model WatermarkW2 Extraction

The watermark is embedded in the fully connected layer in the decoder. The watermark length of the model watermarkW2 is selected as the first L2 model parameter in the fully connected layer in the decoder. The model watermark W2 is then extracted in accordance with (12)W2i=−1,if Dk∈04,1,if Dk∈59.Model watermarkW2 and image watermark W1 have the same validation information w12. Thus, multiple validations of image watermark information and extraction of model copyright information can be achieved by comparing the detected model watermark W2 and image watermark W1.


### 3.3.3. Image WatermarkW1 Extraction

The watermarked imageIem is decoded into the HiDDen model, which first generates L feature channels using eight convolutional layers. A global spatially averaged pooling is then used to generate watermark vectors of the same size. The performance of the watermark decoder has been continuously improved after uninterrupted iterations of the coefficients in the fully connected layer [38]. Finally, the output image watermark W1 is obtained through the fully connected layer.


## 3.3.1. Model WatermarkW3 Extracting

The model watermarkW3 is extracted according to the embedding rules; the coefficients corresponding to the eight convolutional layers and one fully connected layer in the decoder are found for chunking. The Hash value h1,h2,…,h42 of each block is then calculated and compared to the model watermark W3 stored in the redundant bits of the fully connected layer. If they are equal, then no tampering will occur. Otherwise, the model block corresponding to hi has been tampered.


## 3.3.2. Model WatermarkW2 Extraction

The watermark is embedded in the fully connected layer in the decoder. The watermark length of the model watermarkW2 is selected as the first L2 model parameter in the fully connected layer in the decoder. The model watermark W2 is then extracted in accordance with (12)W2i=−1,if Dk∈04,1,if Dk∈59.Model watermarkW2 and image watermark W1 have the same validation information w12. Thus, multiple validations of image watermark information and extraction of model copyright information can be achieved by comparing the detected model watermark W2 and image watermark W1.


## 3.3.3. Image WatermarkW1 Extraction

The watermarked imageIem is decoded into the HiDDen model, which first generates L feature channels using eight convolutional layers. A global spatially averaged pooling is then used to generate watermark vectors of the same size. The performance of the watermark decoder has been continuously improved after uninterrupted iterations of the coefficients in the fully connected layer [38]. Finally, the output image watermark W1 is obtained through the fully connected layer.


## 4. Experiments

Experimental Evaluation: the hardware used for the experiments was a graphics card of NVIDIA GeForce RTX 3090/PCIe/SSE2, Intel® Core™ i9–10900X CPU @ 3.70 GHz × 20, and 62.5 GB memory. The standard Structure-Datasets applied for the experiments include coco-2014, coco-2017, and Boss.


### 4.1. Fidelity Assessment

In the proposed scheme, the coefficients of the embedded watermark are substantially smaller than the entire coefficients of the model. The watermark embedding takes an LSB-like approach, which has little impact on the model and hardly affects the model output accuracy. Taking standard Structure-Dataset coco-2014, coco-2017, and Boss as examples, the middle layer parameters of the model are approximately 223,812. The experiments show that the accuracy of the model does not diminish despite modification of 15% (33571) of parameters for watermark embedding as shown in Table2. Figure 7 reveals the accuracy of different change rates. We refer to the HiDDen as the baseline accuracy and the accuracy of the watermarking model as the watermarking accuracy, and also separately for different kinds of images. The results indicate that the accuracy of the watermarked model is close to the baseline.Table 2
Fidelity evaluation of the three sets of our watermarking model.
Structure–DatasetClassBaseline accuracy (%)Rate of watermarked weights (%)Watermarked accuracy (%)Coco-2014All96.021595.51Cat96.041595.54Car96.001595.51Banana95.941595.51Person95.881595.43Coco-2017All96.101595.58Cat96.161595.52Car96.001595.47Banana96.071595.56Person96.141595.57BossAll96.051595.52Figure 7
The comparison of watermarked accuracy and baseline accuracy.


### 4.2. Image Quality

Only some layers of the decoder model in the HiDDen network are modified, and model watermarksW2 and W3 are embedded in the decoding layer of the self-coding network model. Thus, the quality of the output image is maintained despite the addition of the image watermark, as shown in Figure 8. Both the image watermarked and the final watermarked images of our proposed method have excellent visual quality compared with the original images.Figure 8
Comparison of image quality in three cases: (a) original images, (b) image watermarked images, and (c) final watermarked images.
(a)(b)(c)


### 4.3. Model Integrity Certification

The model watermarkW3 is extracted in accordance with the embedding rules of the watermark, and the Hash value of each block in each layer is also calculated and compared with the model watermark W3. The corresponding blocks of the convolutional and fully connected layers corresponding to the Hash value hi have been tampered with when the comparison of the Hash value hi differs from that in the model watermark W3. A digit after the decimal point is selected in the experiment to modify and embed the watermark (details are presented in subsection 3.2.4). Such a selection saves time and cost compared with that of Uchida et al. [14] and has advantages in watermark extraction accuracy. The test accuracy of the proposed watermarked model with different watermark capacities (in bits) is shown in Table 3.Table 3
Test accuracy of watermarked model with different watermark capacities.
Embedded bitsOur watermarked model (%)Uchida et al. (2017) (%)25695.559795.454251295.558895.4563102495.554395.4563204895.554595.4588


### 4.4. Image Watermark Authentication

The model watermarkW2 and the image watermark W1 have some mutual information between them. Thus, verification of the image watermark information and extraction of the model copyright information can be achieved by comparing the detected model watermark W2 and the image watermark W1.


## 4.1. Fidelity Assessment

In the proposed scheme, the coefficients of the embedded watermark are substantially smaller than the entire coefficients of the model. The watermark embedding takes an LSB-like approach, which has little impact on the model and hardly affects the model output accuracy. Taking standard Structure-Dataset coco-2014, coco-2017, and Boss as examples, the middle layer parameters of the model are approximately 223,812. The experiments show that the accuracy of the model does not diminish despite modification of 15% (33571) of parameters for watermark embedding as shown in Table2. Figure 7 reveals the accuracy of different change rates. We refer to the HiDDen as the baseline accuracy and the accuracy of the watermarking model as the watermarking accuracy, and also separately for different kinds of images. The results indicate that the accuracy of the watermarked model is close to the baseline.Table 2
Fidelity evaluation of the three sets of our watermarking model.
Structure–DatasetClassBaseline accuracy (%)Rate of watermarked weights (%)Watermarked accuracy (%)Coco-2014All96.021595.51Cat96.041595.54Car96.001595.51Banana95.941595.51Person95.881595.43Coco-2017All96.101595.58Cat96.161595.52Car96.001595.47Banana96.071595.56Person96.141595.57BossAll96.051595.52Figure 7
The comparison of watermarked accuracy and baseline accuracy.


## 4.2. Image Quality

Only some layers of the decoder model in the HiDDen network are modified, and model watermarksW2 and W3 are embedded in the decoding layer of the self-coding network model. Thus, the quality of the output image is maintained despite the addition of the image watermark, as shown in Figure 8. Both the image watermarked and the final watermarked images of our proposed method have excellent visual quality compared with the original images.Figure 8
Comparison of image quality in three cases: (a) original images, (b) image watermarked images, and (c) final watermarked images.
(a)(b)(c)


## 4.3. Model Integrity Certification

The model watermarkW3 is extracted in accordance with the embedding rules of the watermark, and the Hash value of each block in each layer is also calculated and compared with the model watermark W3. The corresponding blocks of the convolutional and fully connected layers corresponding to the Hash value hi have been tampered with when the comparison of the Hash value hi differs from that in the model watermark W3. A digit after the decimal point is selected in the experiment to modify and embed the watermark (details are presented in subsection 3.2.4). Such a selection saves time and cost compared with that of Uchida et al. [14] and has advantages in watermark extraction accuracy. The test accuracy of the proposed watermarked model with different watermark capacities (in bits) is shown in Table 3.Table 3
Test accuracy of watermarked model with different watermark capacities.
Embedded bitsOur watermarked model (%)Uchida et al. (2017) (%)25695.559795.454251295.558895.4563102495.554395.4563204895.554595.4588


## 4.4. Image Watermark Authentication

The model watermarkW2 and the image watermark W1 have some mutual information between them. Thus, verification of the image watermark information and extraction of the model copyright information can be achieved by comparing the detected model watermark W2 and the image watermark W1.


## 5. Conclusion

In this paper, we propose an integrity authentication algorithm embedding multiple watermarks in the HiDDen model. These multiple watermarks include one image watermark and two model watermarks. The three watermarks are applied to protect the copyright information of the model and can pinpoint the exact location of model tampering. The fourth decimal place of the model parameters is modified to ensure the robustness and imperceptibility of the watermarking algorithm. The Hash values of all convolutional layers and fully connected layer are also used as one of the model watermarks for tampering location. Compared with previous algorithms, the proposed method achieves remarkable performance in various experiments considering fidelity, imperceptibility, model integrity authentication, and watermark authentication, rather than its practical value.



---

*Source: 9936661-2021-12-30.xml*