# Deep Image Watermarking to JPEG Compression Based on Mixed-Frequency Channel Attention

Authors: Jun Tan; Yinan Hu; Ziming Shi; Bin Wang
Journal: Computational and Mathematical Methods in Medicine
 (2022)

---

## Abstract

Deep blind watermarking algorithms based on an end-to-end encoder-decoder architecture have recently been extensively studied as an important technology for protecting copyright. However, none of the existing algorithms can fully utilize the channel features of the image to improve the robustness against JPEG compression while obtaining high visual quality. Therefore, we propose firstly a mixed-frequency channel attention method in the encoder, which utilizes different frequency components of the 2D-DCT domain as weight coefficients during channel squeezing and excitation. Its essence is to suppress the useless feature maps and enhance the feature maps suitable for watermarking embedding by introducing frequency analysis in the channel dimension. The experimental results indicate that the PSNR of our method reaches over 38 and the BER is less than 0.01% under the JPEG compression with quality factorQ=50. Besides, the proposed framework also obtains excellent robustness for a variety of common distortions, including Gaussian filter, crop, crop out, and drop out.


---

## Body

## 1. Introduction

As the mobile Internet industry develops rapidly, people gain access to large amounts of multimedia information. However, the deluge of multimedia information has resulted in a series of issues, including copyright conflicts and malicious tampering. Image encryption [1, 2], steganography [3, 4], digital watermarking [5], and other technologies came into being to solve the problem caused by information leakage. Digital watermarking, an effective technology for protecting copyright, has been used in image, audio, video, and other fields [6–11]. Digital image watermarking is one of the most important research directions for digital watermarking. The principle of digital image watermarking is to embed secret messages into the cover image in a way that is imperceptible to the human visual system, and the secret messages can still be recovered even if the encoded image is modified.Traditional digital image watermarking algorithms are mainly divided into spatial watermarking and frequency watermarking. The spatial watermarking algorithms embed the watermark directly by modifying the image pixel, but this method is easily detected by a statistical method [12]. Therefore, researchers began to pay attention to the frequency domain, and they found that watermark embedding in DCT [13], DWT [14], and other frequency domains has better robustness and image visual quality. However, these traditional methods rely heavily on artificial shallow feature extraction, and they cannot make full use of the cover image, which greatly limits the robustness of the algorithm.In recent years, with the success of deep neural networks in information hiding [15, 16] and other fields [17–21], some digital watermarking algorithms based on the deep neural network (DNN) have emerged [22, 23]. Kandi et al. [24] firstly applied a Convolutional Neural Network (CNN) to watermarking, which offers superior invisibility and robustness over traditional methods. However, the method is nonblind watermarking, which only applies in a narrow area. Ahmadi et al. [25] proposed a blind watermarking based on CNN, in which the circular convolution blocks are used to expand secret messages into the whole cover image to withstand geometric distortions. Zhu et al. [26] proposed an end-to-end DNN-based model for watermarking and a method called JPEG-Mask, which simulates the nondifferential JPEG compression. However, the simulated JPEG compression added as a noise layer to the training cannot achieve the effect that real JPEG compression plays. Therefore, a two-stage separable deep watermarking framework [27] was proposed. In stage I, only the encoder and decoder were initially trained to perform powerfully in encoding and decoding, and the decoder is individually fine-tuned by nondifferential distortions in stage II. The two-stage method may find the locally optimal results but cannot find the globally optimal results. Jia et al. [28] proposed a Mini-Batch of Real and Simulated JPEG compression (MBRS) method. For each minibatch image, one of the simulated JPEG, real JPEG compression, and a noise-free layer (identity) is selected randomly as the noise layer, and the gradient direction is updated in real time to find the globally optimal result. However, the above-mentioned methods ignore the frequency analysis, which can be combined with channel feature selection to improve the visual quality and robustness.In order to address the aforementioned problems, based on the previous work [29, 30] about frequency analysis being introduced into DNN, we proposed a new attention method in this paper, which consists of two branches. One branch utilizes several squeeze-and-excitation (SE) [31] blocks to extract the lowest-frequency components of the DCT domain [32] from the channel feature maps to obtain the basic information of the cover image. The other branch utilizes frequency channel attention (FCA) [29] blocks to extract the low-frequency components of channel feature maps to reserve some details. Intuitively, we think that multifrequency components can capture more details to improve visual quality and the combined components of channels can withstand JPEG compression. Besides, we add a diffusion block that is a fully connected layer used in [28] into the message processor to diffuse the secret message into the whole image. In our architecture, we use the strength factor to adjust the trade-off between robustness and imperceptibility. The results indicate that under JPEG compression, our method can achieve higher image quality and the decoding bit error rate (BER) is close to almost 0%. Moreover, we can train a model with a combined noise layer, making it robust for many common distortions.In summary, the contributions of this paper are as follows:(i)
To our knowledge, we are the first to introduce the frequency channel attention into digital watermarking, and we propose a mixed-frequency channel attention method for robust and blind image watermarking(ii)
We choose 16 low-frequency channel components according to the zigzag form as the compression weight coefficients for the FCA channel attention block in our proposed scheme. Experimental results show that this selection scheme is superior to the midfrequency and high-frequency components when the noise layer is JPEG compression(iii)
We propose a two-branch structure, which concentrates on the information from the lowest-frequency channel feature map and other low-frequency channel feature maps. The results of the experiments indicate that this structure can perform better than other mixed-frequency channel attention structuresThe remainder of the paper is arranged as follows. Section2 introduces the details of the proposed framework. Experiments and comparisons with relative schemes are presented in Section 3. The discussion and analyses are described in Section 4. Section 5 concludes the paper.


## 2. Proposed Framework and Method

### 2.1. Network Architecture

As shown in Figure1, the whole model includes five components: message processor, encoder, noise layer, decoder, and adversary.Figure 1
Overall model architecture. The message processor can learn the method for expanding message and realizing redundancy by the transposed convolutional layer; the encoder includes the mixed-frequency channel attention block which embeds the secret message into the whole cover image, the noise layer changes the kind of noise according to the MBRS method for offering the robustness, and the decoder extracts the secret message from the encoded image. An adversary discriminator is used to distinguish the cover image and the encoded image.


#### 2.1.1. Message Processor MP

The message processor is mainly responsible for processing the message and inputting the processed feature maps into the encoder. MP receives the binary secret messageM of length l that is composed of 0,1 and outputs the message feature maps Men of shape C′×H×W, where C′ is the channel number of the feature map. Specifically, the message M is generated randomly with a length of l and is reshaped to 0,11×h×w. It is then amplified by a 3×3 ConvBNReLU layer, which consists of a convolutional layer, batch normalization, and ReLU activation function and is expanded to C×H×W by several transposed convolution layers. Finally, to expand the message more appropriately, the features of the message are extracted by several SE blocks that maintain the shape.


#### 2.1.2. EncoderE

An encoder with the parameterθE takes a RGB color image Ico of the shape 3×H×W and the message maps Men as input and outputs an encoded image Ien of the shape 3×H×W. For selecting channel features better, we utilize a mixed-frequency channel attention block that includes several SE blocks and an FCA block as shown in Figure 1. The whole encoder consists of several 3×3 ConvBNReLU layers, a mixed-frequency channel attention block, and a 1×1 convolutional layer. Firstly, we amplify the cover image through a 3×3 ConvBNReLU layer and then extract image features of the same shape with the proposed attention block. The feature maps obtained by the attention block are then concentrated through a 3×3 ConvBNReLU layer. We feed the cover image features and message feature maps obtained from the message processor into a 3×3 ConvBNReLU layer for simple fusion. Then, we concatenate the obtained tensor and the cover image into a new tensor and feed it into a 1×1 convolutional layer to obtain the encoded image Ien. Training the encoder is aimed at minimizing the L2 distance between Ico and Ien by updating θE:
(1)LE1=MSEIco,Ien=MSEIco,EθE,Ico,Men.


#### 2.1.3. Noise LayerN

The robustness of the whole model is provided by the noise layer. We select different noises from the appointed noise pool as the noise layer. It receivesIen and outputs the noised image Ino of the same shape. Besides, the end-to-end model requires all noises to join in the process of training. Therefore, we proposed the MBRS method [28] as the training method for the noise layer.


#### 2.1.4. DecoderD

The task of the decoder with parameterθD is to recover the secret message MD of length L from the noised image Ino. The component determines the ability of the whole model to extract watermarking. In the decoding stage, we feed the noised image Ino to a 3×3 ConvBNReLU layer and downsample the obtained feature maps by several SE blocks. Then, we convert the multichannel tensor into a single-channel tensor through a 3×3 convolutional layer and change the shape of the single-channel tensor to obtain the decoded message MD. The objective of decoder training is to minimize the distance between M and MD by updating parameters θD to make them the same:
(2)LD=MSEM,MD=MSEM,DθD,Ino.Since it plays an important role in the bit error rate indicator, the loss function accounts for the largest proportion of the total loss function.


#### 2.1.5. Adversary DiscriminatorA

The adversary discriminator [33] consists of several 3×3 ConvBNReLU layers and a global average pooling layer. Under the influence of the adversarial network, the encoder will try to deceive the adversary as much as possible, so that the adversary cannot make a correct judgment on Ico and Ien. And update parameters θE to minimize LE2 to improve the encoding visual quality of the encoder:
(3)LE2=logAθA,Ien=logAθA,EθE,Ico,Men.The discriminator with parametersθA needs to distinguish between Ico and Ien as a binary classifier. The goal of the adversary is to minimize the loss of classification LA by updating θA:
(4)LA=log1−AθA,EθE,Ico,Men+logAθA,Ico.The total loss function isL=λELE1+λDLD+λALE2, and loss LA is for the adversary discriminator.


### 2.2. Squeeze-and-Excitation Networks

An SE channel attention mechanism focuses on exploring the correlation of channel dimensions by modelling the relationships between channels and adaptively adjusting the feature values of each channel so that the attention network learns global information and reinforces the useful information while suppressing the useless information. The SE channel attention network is divided into two-step operations including squeeze and excitation. Squeeze is specifically a global average pooling operation that compresses the size of feature map fromC×h×w into C×1×1, the result of which can represent global information. The excitation operation can be considered a combination of two fully connected layers. The tensor obtained after the squeeze operation is first fully connected to compress the C dimensional tensor to C/r dimension and activated by the ReLU function and then fully connected again to transform the C/r dimension back to c dimension and activated by the sigmoid function to obtain the weight tensor. Finally, the weight tensor with 1×1 obtained by the excitation operation is scaled by the original tensor with C×h×w.


### 2.3. Frequency Channel Attention

#### 2.3.1. The Basic Principle on FCA

Previous studies have tried to explain the relationship between the DCT and global average pooling (GAP) and hoped to mine the information of the DCT domain to better extract features from channels. In this section, we firstly review the formulas of 2D-DCT and GAP, and then, based on the aforementioned work, we elaborate on the principle of the FCA block and the selection of frequency components.To express the basic functions of the two-dimensional (2D) DCT and the entire 2D-DCT more simply, we removed some constant normalization coefficients, but they did not affect the results, just a principle explanation:(5)bu,vi,j=cosπuHi+12cosπvWj+12,(6)Fu,v2d=∑i=0H−1∑j=0W−1xi,j2dbu,vi,j.F2d is the computed 2D-DCT transform domain matrix, x2d is the input, H is the height of x2d, W is the width of x2d, and u∈0,1,⋯,H−1 and v∈0,1,⋯,W−1. GAP is a special case of 2D-DCT when u=0 and v=0 in equation (6), and its result is proportional to the lowest-frequency component of 2D-DCT and is confirmed in [29]:
(7)F0,02d=∑i=0H−1∑j=0W−1xi,j2dcos0Hi+12cos0Wj+12=gapx2dHW.The input to the channel attention block is divided into many parts along the channel dimension. A corresponding 2D-DCT frequency component is assigned to each part, and the 2D-DCT-transformed results can be used as the compression results of channel attention. All transformed parts are concatenated to produce a complete compressed vector. Finally, the obtained compressed weight tensor with1×1×C and the original input tensor are multiplied to get the final result.


### 2.4. Criteria for Choosing Frequency Components

According to the above proof, the squeezing operation of the SE attention block is equivalent to the lowest-frequency component in the corresponding 2D-DCT coefficients. Usually, this component concentrates on most of the energy information of the image, and the conclusion is also valid for channel features. SEnets are a very effective attention network used in most computer vision tasks, but most of the frequency domain components are discarded, some of which are beneficial to improve the performance of watermarking and should not be excluded. Therefore, in order to better compress the channel and introduce more information, we used the FCA block to expand the GAP to more 2D-DCT frequency components. Specific details of the implementation are shown in Figure2(a). We divide 8×8 blocks according to the principle of JPEG compression and select the lowest-frequency component and 15 other low-frequency components according to the form of zigzag as the coefficients of the squeezing operation in the FCA block, as shown in Figure 2(b).Figure 2
(a) The working theory of the FCA block. (b) The method of selecting frequency components in the DCT domain.
(a)(b)


### 2.5. Noise Layer

#### 2.5.1. JPEG Compression

In the real JPEG compression process, we need to quantize the DCT coefficients according to the quantization tables and round them up to the nearest whole number, but the process is nondifferential, which means that the gradient propagates back and the decoding loss will be zero. To address the above-mentioned problem, we use the MBRS method, which can effectively solve the problem about nondifferential distortions.


#### 2.5.2. Traditional Noise Attack

In the field of blind watermarking, some typical noises are often used to test the robustness of the model. In our work, we train five different models separately on the noises, which include cropp=0.035, crop out p=0.3, drop out p=0.3, Gaussian filter σ=2, and identity. Besides, we train a combined noise model with JPEG-Mask Q=50, JPEG Q=50, crop p=0.0225, and identity, which can resist most of the distortions.


### 2.6. Strength Factor

We useIdiff=Ien−Ico to represent the residual signal between the encoded image and the cover image and adjust the trade-off between the visual quality and the bit error rate by the strength factor S: Ien,s=Ico+S·Idiff. The generated image Ien,s is fed into the noise layer to obtain the noised image Ino. We keep S on 1 in the training process and change the S in the testing process for different applications. Because our method is a blind watermarking, the trick is used only in the encoder.


## 2.1. Network Architecture

As shown in Figure1, the whole model includes five components: message processor, encoder, noise layer, decoder, and adversary.Figure 1
Overall model architecture. The message processor can learn the method for expanding message and realizing redundancy by the transposed convolutional layer; the encoder includes the mixed-frequency channel attention block which embeds the secret message into the whole cover image, the noise layer changes the kind of noise according to the MBRS method for offering the robustness, and the decoder extracts the secret message from the encoded image. An adversary discriminator is used to distinguish the cover image and the encoded image.


### 2.1.1. Message Processor MP

The message processor is mainly responsible for processing the message and inputting the processed feature maps into the encoder. MP receives the binary secret messageM of length l that is composed of 0,1 and outputs the message feature maps Men of shape C′×H×W, where C′ is the channel number of the feature map. Specifically, the message M is generated randomly with a length of l and is reshaped to 0,11×h×w. It is then amplified by a 3×3 ConvBNReLU layer, which consists of a convolutional layer, batch normalization, and ReLU activation function and is expanded to C×H×W by several transposed convolution layers. Finally, to expand the message more appropriately, the features of the message are extracted by several SE blocks that maintain the shape.


### 2.1.2. EncoderE

An encoder with the parameterθE takes a RGB color image Ico of the shape 3×H×W and the message maps Men as input and outputs an encoded image Ien of the shape 3×H×W. For selecting channel features better, we utilize a mixed-frequency channel attention block that includes several SE blocks and an FCA block as shown in Figure 1. The whole encoder consists of several 3×3 ConvBNReLU layers, a mixed-frequency channel attention block, and a 1×1 convolutional layer. Firstly, we amplify the cover image through a 3×3 ConvBNReLU layer and then extract image features of the same shape with the proposed attention block. The feature maps obtained by the attention block are then concentrated through a 3×3 ConvBNReLU layer. We feed the cover image features and message feature maps obtained from the message processor into a 3×3 ConvBNReLU layer for simple fusion. Then, we concatenate the obtained tensor and the cover image into a new tensor and feed it into a 1×1 convolutional layer to obtain the encoded image Ien. Training the encoder is aimed at minimizing the L2 distance between Ico and Ien by updating θE:
(1)LE1=MSEIco,Ien=MSEIco,EθE,Ico,Men.


### 2.1.3. Noise LayerN

The robustness of the whole model is provided by the noise layer. We select different noises from the appointed noise pool as the noise layer. It receivesIen and outputs the noised image Ino of the same shape. Besides, the end-to-end model requires all noises to join in the process of training. Therefore, we proposed the MBRS method [28] as the training method for the noise layer.


### 2.1.4. DecoderD

The task of the decoder with parameterθD is to recover the secret message MD of length L from the noised image Ino. The component determines the ability of the whole model to extract watermarking. In the decoding stage, we feed the noised image Ino to a 3×3 ConvBNReLU layer and downsample the obtained feature maps by several SE blocks. Then, we convert the multichannel tensor into a single-channel tensor through a 3×3 convolutional layer and change the shape of the single-channel tensor to obtain the decoded message MD. The objective of decoder training is to minimize the distance between M and MD by updating parameters θD to make them the same:
(2)LD=MSEM,MD=MSEM,DθD,Ino.Since it plays an important role in the bit error rate indicator, the loss function accounts for the largest proportion of the total loss function.


### 2.1.5. Adversary DiscriminatorA

The adversary discriminator [33] consists of several 3×3 ConvBNReLU layers and a global average pooling layer. Under the influence of the adversarial network, the encoder will try to deceive the adversary as much as possible, so that the adversary cannot make a correct judgment on Ico and Ien. And update parameters θE to minimize LE2 to improve the encoding visual quality of the encoder:
(3)LE2=logAθA,Ien=logAθA,EθE,Ico,Men.The discriminator with parametersθA needs to distinguish between Ico and Ien as a binary classifier. The goal of the adversary is to minimize the loss of classification LA by updating θA:
(4)LA=log1−AθA,EθE,Ico,Men+logAθA,Ico.The total loss function isL=λELE1+λDLD+λALE2, and loss LA is for the adversary discriminator.


## 2.1.1. Message Processor MP

The message processor is mainly responsible for processing the message and inputting the processed feature maps into the encoder. MP receives the binary secret messageM of length l that is composed of 0,1 and outputs the message feature maps Men of shape C′×H×W, where C′ is the channel number of the feature map. Specifically, the message M is generated randomly with a length of l and is reshaped to 0,11×h×w. It is then amplified by a 3×3 ConvBNReLU layer, which consists of a convolutional layer, batch normalization, and ReLU activation function and is expanded to C×H×W by several transposed convolution layers. Finally, to expand the message more appropriately, the features of the message are extracted by several SE blocks that maintain the shape.


## 2.1.2. EncoderE

An encoder with the parameterθE takes a RGB color image Ico of the shape 3×H×W and the message maps Men as input and outputs an encoded image Ien of the shape 3×H×W. For selecting channel features better, we utilize a mixed-frequency channel attention block that includes several SE blocks and an FCA block as shown in Figure 1. The whole encoder consists of several 3×3 ConvBNReLU layers, a mixed-frequency channel attention block, and a 1×1 convolutional layer. Firstly, we amplify the cover image through a 3×3 ConvBNReLU layer and then extract image features of the same shape with the proposed attention block. The feature maps obtained by the attention block are then concentrated through a 3×3 ConvBNReLU layer. We feed the cover image features and message feature maps obtained from the message processor into a 3×3 ConvBNReLU layer for simple fusion. Then, we concatenate the obtained tensor and the cover image into a new tensor and feed it into a 1×1 convolutional layer to obtain the encoded image Ien. Training the encoder is aimed at minimizing the L2 distance between Ico and Ien by updating θE:
(1)LE1=MSEIco,Ien=MSEIco,EθE,Ico,Men.


## 2.1.3. Noise LayerN

The robustness of the whole model is provided by the noise layer. We select different noises from the appointed noise pool as the noise layer. It receivesIen and outputs the noised image Ino of the same shape. Besides, the end-to-end model requires all noises to join in the process of training. Therefore, we proposed the MBRS method [28] as the training method for the noise layer.


## 2.1.4. DecoderD

The task of the decoder with parameterθD is to recover the secret message MD of length L from the noised image Ino. The component determines the ability of the whole model to extract watermarking. In the decoding stage, we feed the noised image Ino to a 3×3 ConvBNReLU layer and downsample the obtained feature maps by several SE blocks. Then, we convert the multichannel tensor into a single-channel tensor through a 3×3 convolutional layer and change the shape of the single-channel tensor to obtain the decoded message MD. The objective of decoder training is to minimize the distance between M and MD by updating parameters θD to make them the same:
(2)LD=MSEM,MD=MSEM,DθD,Ino.Since it plays an important role in the bit error rate indicator, the loss function accounts for the largest proportion of the total loss function.


## 2.1.5. Adversary DiscriminatorA

The adversary discriminator [33] consists of several 3×3 ConvBNReLU layers and a global average pooling layer. Under the influence of the adversarial network, the encoder will try to deceive the adversary as much as possible, so that the adversary cannot make a correct judgment on Ico and Ien. And update parameters θE to minimize LE2 to improve the encoding visual quality of the encoder:
(3)LE2=logAθA,Ien=logAθA,EθE,Ico,Men.The discriminator with parametersθA needs to distinguish between Ico and Ien as a binary classifier. The goal of the adversary is to minimize the loss of classification LA by updating θA:
(4)LA=log1−AθA,EθE,Ico,Men+logAθA,Ico.The total loss function isL=λELE1+λDLD+λALE2, and loss LA is for the adversary discriminator.


## 2.2. Squeeze-and-Excitation Networks

An SE channel attention mechanism focuses on exploring the correlation of channel dimensions by modelling the relationships between channels and adaptively adjusting the feature values of each channel so that the attention network learns global information and reinforces the useful information while suppressing the useless information. The SE channel attention network is divided into two-step operations including squeeze and excitation. Squeeze is specifically a global average pooling operation that compresses the size of feature map fromC×h×w into C×1×1, the result of which can represent global information. The excitation operation can be considered a combination of two fully connected layers. The tensor obtained after the squeeze operation is first fully connected to compress the C dimensional tensor to C/r dimension and activated by the ReLU function and then fully connected again to transform the C/r dimension back to c dimension and activated by the sigmoid function to obtain the weight tensor. Finally, the weight tensor with 1×1 obtained by the excitation operation is scaled by the original tensor with C×h×w.


## 2.3. Frequency Channel Attention

### 2.3.1. The Basic Principle on FCA

Previous studies have tried to explain the relationship between the DCT and global average pooling (GAP) and hoped to mine the information of the DCT domain to better extract features from channels. In this section, we firstly review the formulas of 2D-DCT and GAP, and then, based on the aforementioned work, we elaborate on the principle of the FCA block and the selection of frequency components.To express the basic functions of the two-dimensional (2D) DCT and the entire 2D-DCT more simply, we removed some constant normalization coefficients, but they did not affect the results, just a principle explanation:(5)bu,vi,j=cosπuHi+12cosπvWj+12,(6)Fu,v2d=∑i=0H−1∑j=0W−1xi,j2dbu,vi,j.F2d is the computed 2D-DCT transform domain matrix, x2d is the input, H is the height of x2d, W is the width of x2d, and u∈0,1,⋯,H−1 and v∈0,1,⋯,W−1. GAP is a special case of 2D-DCT when u=0 and v=0 in equation (6), and its result is proportional to the lowest-frequency component of 2D-DCT and is confirmed in [29]:
(7)F0,02d=∑i=0H−1∑j=0W−1xi,j2dcos0Hi+12cos0Wj+12=gapx2dHW.The input to the channel attention block is divided into many parts along the channel dimension. A corresponding 2D-DCT frequency component is assigned to each part, and the 2D-DCT-transformed results can be used as the compression results of channel attention. All transformed parts are concatenated to produce a complete compressed vector. Finally, the obtained compressed weight tensor with1×1×C and the original input tensor are multiplied to get the final result.


## 2.3.1. The Basic Principle on FCA

Previous studies have tried to explain the relationship between the DCT and global average pooling (GAP) and hoped to mine the information of the DCT domain to better extract features from channels. In this section, we firstly review the formulas of 2D-DCT and GAP, and then, based on the aforementioned work, we elaborate on the principle of the FCA block and the selection of frequency components.To express the basic functions of the two-dimensional (2D) DCT and the entire 2D-DCT more simply, we removed some constant normalization coefficients, but they did not affect the results, just a principle explanation:(5)bu,vi,j=cosπuHi+12cosπvWj+12,(6)Fu,v2d=∑i=0H−1∑j=0W−1xi,j2dbu,vi,j.F2d is the computed 2D-DCT transform domain matrix, x2d is the input, H is the height of x2d, W is the width of x2d, and u∈0,1,⋯,H−1 and v∈0,1,⋯,W−1. GAP is a special case of 2D-DCT when u=0 and v=0 in equation (6), and its result is proportional to the lowest-frequency component of 2D-DCT and is confirmed in [29]:
(7)F0,02d=∑i=0H−1∑j=0W−1xi,j2dcos0Hi+12cos0Wj+12=gapx2dHW.The input to the channel attention block is divided into many parts along the channel dimension. A corresponding 2D-DCT frequency component is assigned to each part, and the 2D-DCT-transformed results can be used as the compression results of channel attention. All transformed parts are concatenated to produce a complete compressed vector. Finally, the obtained compressed weight tensor with1×1×C and the original input tensor are multiplied to get the final result.


## 2.4. Criteria for Choosing Frequency Components

According to the above proof, the squeezing operation of the SE attention block is equivalent to the lowest-frequency component in the corresponding 2D-DCT coefficients. Usually, this component concentrates on most of the energy information of the image, and the conclusion is also valid for channel features. SEnets are a very effective attention network used in most computer vision tasks, but most of the frequency domain components are discarded, some of which are beneficial to improve the performance of watermarking and should not be excluded. Therefore, in order to better compress the channel and introduce more information, we used the FCA block to expand the GAP to more 2D-DCT frequency components. Specific details of the implementation are shown in Figure2(a). We divide 8×8 blocks according to the principle of JPEG compression and select the lowest-frequency component and 15 other low-frequency components according to the form of zigzag as the coefficients of the squeezing operation in the FCA block, as shown in Figure 2(b).Figure 2
(a) The working theory of the FCA block. (b) The method of selecting frequency components in the DCT domain.
(a)(b)


## 2.5. Noise Layer

### 2.5.1. JPEG Compression

In the real JPEG compression process, we need to quantize the DCT coefficients according to the quantization tables and round them up to the nearest whole number, but the process is nondifferential, which means that the gradient propagates back and the decoding loss will be zero. To address the above-mentioned problem, we use the MBRS method, which can effectively solve the problem about nondifferential distortions.


### 2.5.2. Traditional Noise Attack

In the field of blind watermarking, some typical noises are often used to test the robustness of the model. In our work, we train five different models separately on the noises, which include cropp=0.035, crop out p=0.3, drop out p=0.3, Gaussian filter σ=2, and identity. Besides, we train a combined noise model with JPEG-Mask Q=50, JPEG Q=50, crop p=0.0225, and identity, which can resist most of the distortions.


## 2.5.1. JPEG Compression

In the real JPEG compression process, we need to quantize the DCT coefficients according to the quantization tables and round them up to the nearest whole number, but the process is nondifferential, which means that the gradient propagates back and the decoding loss will be zero. To address the above-mentioned problem, we use the MBRS method, which can effectively solve the problem about nondifferential distortions.


## 2.5.2. Traditional Noise Attack

In the field of blind watermarking, some typical noises are often used to test the robustness of the model. In our work, we train five different models separately on the noises, which include cropp=0.035, crop out p=0.3, drop out p=0.3, Gaussian filter σ=2, and identity. Besides, we train a combined noise model with JPEG-Mask Q=50, JPEG Q=50, crop p=0.0225, and identity, which can resist most of the distortions.


## 2.6. Strength Factor

We useIdiff=Ien−Ico to represent the residual signal between the encoded image and the cover image and adjust the trade-off between the visual quality and the bit error rate by the strength factor S: Ien,s=Ico+S·Idiff. The generated image Ien,s is fed into the noise layer to obtain the noised image Ino. We keep S on 1 in the training process and change the S in the testing process for different applications. Because our method is a blind watermarking, the trick is used only in the encoder.


## 3. Experiments and Results

### 3.1. Experimental Setup, Metrics, and Baselines

To evaluate the effectiveness of the proposed method, we use 10000 random images from the ImageNet dataset [34] for training and 5000 images from the COCO dataset [35] for testing, aiming at ensuring the generation of the trained model. We select the JPEG compression function in the PIL package as testing. The strength factor is set as 1 during training. For the weight factors of the loss function, we choose λE=1, λD=10, and λA=0.0001. For the optimized function, Adam [36] is applied with a learning rate of 10−3 and default hyperparameters. Each model is trained for 100 epochs with a batchsize 16. PSNR and SSIM [37] measure the similarity between Ien and Ico. Robustness is measured by the the difference called BER between the decoded message and secret message. Our baselines for comparison are [26, 27] and [28]. In pursuit of the real results, we realize the MBRS [28] based on the open source of both codes and models. We also try to conduct experiments of [26, 27] but could not reproduce the best performance that they reported. In order to respect the results that they reported, we directly use their published results.


### 3.2. Comparison with SOTA Methods

#### 3.2.1. Robustness

We train a model with JPEG-MaskQ=50, real JPEG Q=50, and identity to demonstrate the robustness of our model against JPEG compression. All the testing processes are performed under real JPEG Q=50. As shown in Table 1, compared to the other method, our model achieves the PSNR that is higher than 38 and the BER that is less than 0.01%, which indicates that our model not only maintains higher image quality for JPEG compression but also achieves lower BER. Figure 3 indicates that the messages are embedded in most areas of the cover images. In addition to JPEG compression distortion, our model is also robust to other image processing distortions, such as Gaussian filter (GF), crop, crop out, and drop out. We also train a combined noise model to embed a 30-bit message into 128×128 images with the noise layer consisting of JPEG-Mask Q=50, real JPEG Q=50, identity, and crop (p=0.0225) and add a diffusion and an inverse-diffusion block mentioned in [28] into the message processor for diffusing a secret message to the whole cover image to resist geometry attacks. As shown in Table 2, our trained model shows robustness against most noises. We also tested some noises not included in the noise layer for the combined noise model, and the experimental results are shown in Figure 4.Table 1
Comparison with the SOTA. We realized the model opening source in [28], while directly using the results included as reported in [26, 27] under quality factor 50. However, SSIM is not reported in [26, 27], for which we empty these items. PSNR is measured for RGB channels, except in [26]; they use the Y channel of the YUV color space.
ModelHiDDeN [26]TSDL [27]MBRS [28]OursImage size128×128128×128128×128128×128Message length30306464Noise layerJPEG-MaskJPEGMixedMixedPSNR30.0933.5136.4938.13SSIM——0.91730.9472BER15%22.3%0.0092%0.0078%Figure 3
Examples of showing the robustness of the model against JPEG compression (Q=50) with experimental results. From top to bottom are the cover images, the encoded images, the noised images, the residual between cover images and encoded image, and the normalization of the residual signal.Table 2
Results of robustness against other distortions. We add an additive diffusion block into the message processor for improving the shortcoming that our original structure is not robust enough against crop, crop out, and drop out and make a comparison with [26–28] trained by a combined noise layer. The strength factor is adjusted for comparison under PSNR=33.5.
NoiseIdentityCrop outp=0.3Drop outp=0.3Cropp=0.035GFσ=2JPEGQ=50HiDDeN [26]0%6%7%12%4%37%TSDL [27]0%2.7%2.6%11%1.4%23.8%MBRS [28]0%0.0027%0.0087%4.15%0.011%4.48%Ours0%0.0013%0.0080%3.24%0.293%2.61%Figure 4
Robustness against different traditional noises. We tested different noises including those that were not included in the trained combined noise layer under strength factors=1. The combined noise layer consists of JPEG-Mask (Q=50), real JPEG (Q=50), identity, and crop (p=0.0225). Top: cover image Ico; second: encoded image Ien; third: noised image Ino; fourth: residual between Ino and Ico; and bottom: the normalization of the residual signal.


#### 3.2.2. Transparency

In order to show that our method can learn more frequency features from cover images, we separately train five models with the noise layer. For GFσ=2 and identity, we embed 64-bit messages into 3×128×128 images without a diffusion block. For crop p=0.035, crop out p=0.3, and drop out p=0.3, we embed 30-bit messages into 3×128×128 images with the diffusion block. Besides, we compare the PSNR and SSIM between Ico and Ien by adjusting S under roughly the same BER. As shown in Table 3, the results of the proposed method perform better than those of other models under most distortions, but our specialized trained model performs worse for the crop attack. Since the information diffusion block we use has more information embedded on a single channel, it has some shortcomings compared to [26] of broadcasting single-bit information on a single channel.Table 3
Results of transparency against other distortions. For revealing the advantage of our proposed method, we make a comparison of PSNR and SSIM with [26, 28] by adjusting strength factor S to keep the BER at 0%. Because the experiment of [27] is not done, we do not list it.
MetricModelCropp=0.035Crop outp=0.3Drop outp=0.3GFσ=2IdentityPSNRHiDDeN [26]35.2047.2442.5240.5544.63MBRS [28]32.1546.7748.5040.7442.81Ours33.0147.2649.4342.3045.71SSIMHiDDeN [26]—————MBRS [28]0.78720.99100.99360.96700.9740Ours0.82250.99240.99450.97600.9867BERHiDDeN [26]0%3%0%0%0%MBRS [28]0.72%0%0%0%0%Ours0.29%0%0%0%0%


### 3.3. Ablation Study

#### 3.3.1. Strength Factor

The strength factor is a parameter used to balance robustness and imperceptibility. We set the value of the strength factorS, from 0.1 to 2.0, with an interval of 0.1, and test the model under different quality factors for JPEG compression. The results are shown in Table 4. With the increment of S, PSNR and SSIM values decrease, the quality of the encoded image becomes worse, and the extraction accuracy becomes higher. In the study, we adjust the value of S to obtain the similar visual quality of different models for fair comparison.Table 4
BER, PSNR, and SSIM values under different strength factorsS and quality factors of JPEG Q.
Strength factor0.40.60.81.01.21.4BERQ=1035.03%27.91%21.80%16.86%13.02%10.06%Q=3015.65%5.94%1.63%0.33%0.047%0.0053%Q=507.74%1.11%0.0778%0.0050%0.0012%0.00%Q=703.87%0.370%0.0170%0.0009%0.00%0.00%Q=902.53%0.202%0.0078%0.0006%0.00%0.00%PSNR45.9242.4039.8937.9536.3735.03SSIM0.98930.97730.96230.94550.92740.9086


#### 3.3.2. Discriminator

To demonstrate that the discriminator can help the encoder generate higher-quality images, we trained the noise-free model with and without the discriminator separately. As can be seen from the normalized watermarking residuals in Figure5, the watermarking model without the discriminator does not produce a uniform distribution of watermarking and produces visual artifacts on the resulting watermarked image. However, the watermarking model with a discriminator generates an even distribution of watermarking, and no aggregation of watermarking occurs.Figure 5
The results of encoding with and without a discriminator: top: cover imageIco; second: encoded image Ien; third: noised image Ino; fourth: residual signal between Ien and Ico; and bottom: the normalization of the residual signal.


#### 3.3.3. Different Mixed-Frequency Channel Attention

To demonstrate that our two-branch structure is superior to other combined mixed-frequency channel attention blocks, we conduct experiments for the encoder with different frequency channel attention structures. We proposed another four kinds of structures to be applied in the encoder. The first is called LFCA, which only consists of several FCA blocks with low-frequency components, the second is called SE&LFCA, in which we insert an FCA block behind the SE blocks, the third is composed of several SE blocks, and the last is called LFCA&SE, in which we insert an FCA block in front of the SE blocks. Their detailed structures are shown in Figure6. We list the results of experiments separately under JPEG compression and combined noises for the above-mentioned four structures in Tables 5 and 6.Figure 6
The other structures of mixed-frequency channel attention, which are proposed in the ablation experiment. The structure called LFCA is only composed of several frequency channel attention blocks with low-frequency components. The structure called SE consists of several SE channel attention blocks. The structure called SE&LFCA inserts an FCA block behind the SE blocks. The structure called LFCA&SE inserts FCA blocks in front of the SE blocks.Table 5
Comparison with the ablation experimental results, with the best results in bold and second bests in blue. We compared the performance with four mixed-frequency channel attention variants under JPEG compressionQ=50.
ModelSELFCASE&LFCALFCA&SEOursImage size128×128128×128128×128128×128128×128Message length6464646464Noise layerMixedMixedMixedMixedMixedPSNR37.7037.3238.2137.4938.13SSIM0.94450.94280.95530.94870.9472BER0.0082%0.0081%0.0059%0.0081%0.0078%Table 6
BER comparison between the proposed scheme and ablation schemes, with the best results in bold, second bests in blue, and the worst results in red. The strength factor is adjusted for comparison underPSNR=33.5.
NoiseIdentityCrop outp=0.3Drop outp=0.3Cropp=0.035GFσ=2JPEGQ=50SE0%0.0027%0.0093%5.07%0.011%3.22%LFCA0%0.0%0.0027%4.23%0.031%5.62%SE&LFCA0%0.0040%0.0179%6.21%0.185%22.87%LFCA&SE0%0.0047%0.0053%18.06%0.006%9.12%Ours0%0.0013%0.0080%3.24%0.293%2.61%The channel attention mechanism assigns weights to the feature maps. SE only selects the lowest-frequency component coefficients of the 2D-DCT to enhance all channel feature maps through multiple SE blocks, while LFCA chooses to divide the feature maps on the channels and select multiple low-frequency component coefficients of the 2D-DCT to enhance through several LFCA blocks. We believe that when the noise layer only includes JPEG compression, the weights of LFCA enhancement are spread over multiple low-frequency components relative to SE, and thus, the performance will be worse than that of SE. However, combining SE blocks and LFCA blocks gives better performance. As can be seen from Table5, the performance of SE&LFCA and LFCA&SE is better than that of SE and LFCA. SE&LFCA firstly allocates the lowest-frequency component coefficients through an SE block and then uses several LFCA blocks to enhance multifrequency component coefficients on the basis of the lowest-frequency component, which has a good effect. Although LFCA&SE is also composed of an SE block and several LFCA blocks, its effect is not as good as that of SE&LFCA. We believed that this is caused by LFCA assigning weights in the first place.Our parallel structure is a better way of feature fusion when the noise layer includes multinoises. We believe that the reason why the experimental results of SE&LFCA and LFCA&SE perform worse is that they have no skip connection. Our proposed method achieves better performance with skip connection of FCA, which is confirmed by the experimental results in Table7.Table 7
The ablation experiment of skip connection. Baseline: no attention model; +SE: add SE channel attention networks; +skip connection: based on SEnets, add a skip connection of LFCA.
(a)ModelMetricJPEGQ=50Cropp=0.035Crop outp=0.3Drop outp=0.3GFσ=2BaselinePSNR32.6032.6032.5932.5732.59+SE32.5532.5832.5732.5632.56+skip connection (LFCA)32.7532.7632.7632.7332.76BaselineSSIM0.81500.81570.81490.81460.8150+SE0.81610.81650.81600.81570.8159+skip connection (LFCA)0.82630.82660.82630.82590.8265BaselineBER3.09%6.96%0%0.0007%0.168%+SE2.19%4.30%0.0013%0.0067%0.0054%+skip connection (LFCA)1.94%3.08%0.0067%0.0040%0.228%(b)Salt&Pepperp=0.1GNp=0.1MFs=332.5932.5932.5832.5532.5632.5532.7532.7632.750.81480.81530.81540.81560.81590.81600.82600.82710.82630.830%1.23%0%0.813%1.51%0%0.665%0.859%0%


#### 3.3.4. Selection Scheme of Frequency Components

To demonstrate that the FCA attention block in our method chooses the low-frequency component coefficients of the DCT to improve the robustness to JPEG compression, we select 16 components of low frequency, 16 components of middle frequency, and 16 components of high frequency as the weight coefficients of FCA from the8×8 coefficients, respectively, and train them under JPEG compression. It can be seen from Table 8 that the selection of frequency domain components has a certain impact on the robustness and imperceptibility of the model. When the low-frequency components are selected, the metrics such as PSNR, SSIM, and BER all reach the highest.Table 8
BER comparison between the proposed scheme and ablation schemes. We train three different ways of frequency domain component selection for the FCA attention block in the mixed-frequency channel attention block under JPEG compression. Low frequency: 16 coefficients of low-frequency components; middle frequency: 16 coefficients of middle-frequency components; and high frequency: 16 coefficients of high-frequency components.
ModelPSNRSSIMBERHigh frequency37.770.94070.6781%Middle frequency38.020.94720.5782%Low frequency38.130.94720.0078%


#### 3.3.5. Skip Connection

To show the important role of introducing frequency analysis and skip connection, we trained three different watermarking models under a mixed-noise layer separately: baseline: without attention networks in the encoder; +SE: with the addition of the SE channel attention blocks in the encoder; and +skip connection: based on +SE, with the addition of the LFCA attention block via skip connection. Table7 shows the results of experiments, where the performance of the model by adding SE attention block is improved compared to baseline under most of the noises. However, we find that the embedding of the watermark information by adding the SE attention block is more concentrated in the low-frequency region which is less affected by the Gaussian filter but will be more affected by the Gaussian noise. In order to further improve the robustness for noises such as JPEG compression, we added the LFCA attention block by skip connection on the basis of the SE attention blocks, and the experimental results show that the quality of the encoded image is improved by skip connection, the best robustness is achieved for most distortions, and our watermark embedding assignment is more reasonable.


## 3.1. Experimental Setup, Metrics, and Baselines

To evaluate the effectiveness of the proposed method, we use 10000 random images from the ImageNet dataset [34] for training and 5000 images from the COCO dataset [35] for testing, aiming at ensuring the generation of the trained model. We select the JPEG compression function in the PIL package as testing. The strength factor is set as 1 during training. For the weight factors of the loss function, we choose λE=1, λD=10, and λA=0.0001. For the optimized function, Adam [36] is applied with a learning rate of 10−3 and default hyperparameters. Each model is trained for 100 epochs with a batchsize 16. PSNR and SSIM [37] measure the similarity between Ien and Ico. Robustness is measured by the the difference called BER between the decoded message and secret message. Our baselines for comparison are [26, 27] and [28]. In pursuit of the real results, we realize the MBRS [28] based on the open source of both codes and models. We also try to conduct experiments of [26, 27] but could not reproduce the best performance that they reported. In order to respect the results that they reported, we directly use their published results.


## 3.2. Comparison with SOTA Methods

### 3.2.1. Robustness

We train a model with JPEG-MaskQ=50, real JPEG Q=50, and identity to demonstrate the robustness of our model against JPEG compression. All the testing processes are performed under real JPEG Q=50. As shown in Table 1, compared to the other method, our model achieves the PSNR that is higher than 38 and the BER that is less than 0.01%, which indicates that our model not only maintains higher image quality for JPEG compression but also achieves lower BER. Figure 3 indicates that the messages are embedded in most areas of the cover images. In addition to JPEG compression distortion, our model is also robust to other image processing distortions, such as Gaussian filter (GF), crop, crop out, and drop out. We also train a combined noise model to embed a 30-bit message into 128×128 images with the noise layer consisting of JPEG-Mask Q=50, real JPEG Q=50, identity, and crop (p=0.0225) and add a diffusion and an inverse-diffusion block mentioned in [28] into the message processor for diffusing a secret message to the whole cover image to resist geometry attacks. As shown in Table 2, our trained model shows robustness against most noises. We also tested some noises not included in the noise layer for the combined noise model, and the experimental results are shown in Figure 4.Table 1
Comparison with the SOTA. We realized the model opening source in [28], while directly using the results included as reported in [26, 27] under quality factor 50. However, SSIM is not reported in [26, 27], for which we empty these items. PSNR is measured for RGB channels, except in [26]; they use the Y channel of the YUV color space.
ModelHiDDeN [26]TSDL [27]MBRS [28]OursImage size128×128128×128128×128128×128Message length30306464Noise layerJPEG-MaskJPEGMixedMixedPSNR30.0933.5136.4938.13SSIM——0.91730.9472BER15%22.3%0.0092%0.0078%Figure 3
Examples of showing the robustness of the model against JPEG compression (Q=50) with experimental results. From top to bottom are the cover images, the encoded images, the noised images, the residual between cover images and encoded image, and the normalization of the residual signal.Table 2
Results of robustness against other distortions. We add an additive diffusion block into the message processor for improving the shortcoming that our original structure is not robust enough against crop, crop out, and drop out and make a comparison with [26–28] trained by a combined noise layer. The strength factor is adjusted for comparison under PSNR=33.5.
NoiseIdentityCrop outp=0.3Drop outp=0.3Cropp=0.035GFσ=2JPEGQ=50HiDDeN [26]0%6%7%12%4%37%TSDL [27]0%2.7%2.6%11%1.4%23.8%MBRS [28]0%0.0027%0.0087%4.15%0.011%4.48%Ours0%0.0013%0.0080%3.24%0.293%2.61%Figure 4
Robustness against different traditional noises. We tested different noises including those that were not included in the trained combined noise layer under strength factors=1. The combined noise layer consists of JPEG-Mask (Q=50), real JPEG (Q=50), identity, and crop (p=0.0225). Top: cover image Ico; second: encoded image Ien; third: noised image Ino; fourth: residual between Ino and Ico; and bottom: the normalization of the residual signal.


### 3.2.2. Transparency

In order to show that our method can learn more frequency features from cover images, we separately train five models with the noise layer. For GFσ=2 and identity, we embed 64-bit messages into 3×128×128 images without a diffusion block. For crop p=0.035, crop out p=0.3, and drop out p=0.3, we embed 30-bit messages into 3×128×128 images with the diffusion block. Besides, we compare the PSNR and SSIM between Ico and Ien by adjusting S under roughly the same BER. As shown in Table 3, the results of the proposed method perform better than those of other models under most distortions, but our specialized trained model performs worse for the crop attack. Since the information diffusion block we use has more information embedded on a single channel, it has some shortcomings compared to [26] of broadcasting single-bit information on a single channel.Table 3
Results of transparency against other distortions. For revealing the advantage of our proposed method, we make a comparison of PSNR and SSIM with [26, 28] by adjusting strength factor S to keep the BER at 0%. Because the experiment of [27] is not done, we do not list it.
MetricModelCropp=0.035Crop outp=0.3Drop outp=0.3GFσ=2IdentityPSNRHiDDeN [26]35.2047.2442.5240.5544.63MBRS [28]32.1546.7748.5040.7442.81Ours33.0147.2649.4342.3045.71SSIMHiDDeN [26]—————MBRS [28]0.78720.99100.99360.96700.9740Ours0.82250.99240.99450.97600.9867BERHiDDeN [26]0%3%0%0%0%MBRS [28]0.72%0%0%0%0%Ours0.29%0%0%0%0%


## 3.2.1. Robustness

We train a model with JPEG-MaskQ=50, real JPEG Q=50, and identity to demonstrate the robustness of our model against JPEG compression. All the testing processes are performed under real JPEG Q=50. As shown in Table 1, compared to the other method, our model achieves the PSNR that is higher than 38 and the BER that is less than 0.01%, which indicates that our model not only maintains higher image quality for JPEG compression but also achieves lower BER. Figure 3 indicates that the messages are embedded in most areas of the cover images. In addition to JPEG compression distortion, our model is also robust to other image processing distortions, such as Gaussian filter (GF), crop, crop out, and drop out. We also train a combined noise model to embed a 30-bit message into 128×128 images with the noise layer consisting of JPEG-Mask Q=50, real JPEG Q=50, identity, and crop (p=0.0225) and add a diffusion and an inverse-diffusion block mentioned in [28] into the message processor for diffusing a secret message to the whole cover image to resist geometry attacks. As shown in Table 2, our trained model shows robustness against most noises. We also tested some noises not included in the noise layer for the combined noise model, and the experimental results are shown in Figure 4.Table 1
Comparison with the SOTA. We realized the model opening source in [28], while directly using the results included as reported in [26, 27] under quality factor 50. However, SSIM is not reported in [26, 27], for which we empty these items. PSNR is measured for RGB channels, except in [26]; they use the Y channel of the YUV color space.
ModelHiDDeN [26]TSDL [27]MBRS [28]OursImage size128×128128×128128×128128×128Message length30306464Noise layerJPEG-MaskJPEGMixedMixedPSNR30.0933.5136.4938.13SSIM——0.91730.9472BER15%22.3%0.0092%0.0078%Figure 3
Examples of showing the robustness of the model against JPEG compression (Q=50) with experimental results. From top to bottom are the cover images, the encoded images, the noised images, the residual between cover images and encoded image, and the normalization of the residual signal.Table 2
Results of robustness against other distortions. We add an additive diffusion block into the message processor for improving the shortcoming that our original structure is not robust enough against crop, crop out, and drop out and make a comparison with [26–28] trained by a combined noise layer. The strength factor is adjusted for comparison under PSNR=33.5.
NoiseIdentityCrop outp=0.3Drop outp=0.3Cropp=0.035GFσ=2JPEGQ=50HiDDeN [26]0%6%7%12%4%37%TSDL [27]0%2.7%2.6%11%1.4%23.8%MBRS [28]0%0.0027%0.0087%4.15%0.011%4.48%Ours0%0.0013%0.0080%3.24%0.293%2.61%Figure 4
Robustness against different traditional noises. We tested different noises including those that were not included in the trained combined noise layer under strength factors=1. The combined noise layer consists of JPEG-Mask (Q=50), real JPEG (Q=50), identity, and crop (p=0.0225). Top: cover image Ico; second: encoded image Ien; third: noised image Ino; fourth: residual between Ino and Ico; and bottom: the normalization of the residual signal.


## 3.2.2. Transparency

In order to show that our method can learn more frequency features from cover images, we separately train five models with the noise layer. For GFσ=2 and identity, we embed 64-bit messages into 3×128×128 images without a diffusion block. For crop p=0.035, crop out p=0.3, and drop out p=0.3, we embed 30-bit messages into 3×128×128 images with the diffusion block. Besides, we compare the PSNR and SSIM between Ico and Ien by adjusting S under roughly the same BER. As shown in Table 3, the results of the proposed method perform better than those of other models under most distortions, but our specialized trained model performs worse for the crop attack. Since the information diffusion block we use has more information embedded on a single channel, it has some shortcomings compared to [26] of broadcasting single-bit information on a single channel.Table 3
Results of transparency against other distortions. For revealing the advantage of our proposed method, we make a comparison of PSNR and SSIM with [26, 28] by adjusting strength factor S to keep the BER at 0%. Because the experiment of [27] is not done, we do not list it.
MetricModelCropp=0.035Crop outp=0.3Drop outp=0.3GFσ=2IdentityPSNRHiDDeN [26]35.2047.2442.5240.5544.63MBRS [28]32.1546.7748.5040.7442.81Ours33.0147.2649.4342.3045.71SSIMHiDDeN [26]—————MBRS [28]0.78720.99100.99360.96700.9740Ours0.82250.99240.99450.97600.9867BERHiDDeN [26]0%3%0%0%0%MBRS [28]0.72%0%0%0%0%Ours0.29%0%0%0%0%


## 3.3. Ablation Study

### 3.3.1. Strength Factor

The strength factor is a parameter used to balance robustness and imperceptibility. We set the value of the strength factorS, from 0.1 to 2.0, with an interval of 0.1, and test the model under different quality factors for JPEG compression. The results are shown in Table 4. With the increment of S, PSNR and SSIM values decrease, the quality of the encoded image becomes worse, and the extraction accuracy becomes higher. In the study, we adjust the value of S to obtain the similar visual quality of different models for fair comparison.Table 4
BER, PSNR, and SSIM values under different strength factorsS and quality factors of JPEG Q.
Strength factor0.40.60.81.01.21.4BERQ=1035.03%27.91%21.80%16.86%13.02%10.06%Q=3015.65%5.94%1.63%0.33%0.047%0.0053%Q=507.74%1.11%0.0778%0.0050%0.0012%0.00%Q=703.87%0.370%0.0170%0.0009%0.00%0.00%Q=902.53%0.202%0.0078%0.0006%0.00%0.00%PSNR45.9242.4039.8937.9536.3735.03SSIM0.98930.97730.96230.94550.92740.9086


### 3.3.2. Discriminator

To demonstrate that the discriminator can help the encoder generate higher-quality images, we trained the noise-free model with and without the discriminator separately. As can be seen from the normalized watermarking residuals in Figure5, the watermarking model without the discriminator does not produce a uniform distribution of watermarking and produces visual artifacts on the resulting watermarked image. However, the watermarking model with a discriminator generates an even distribution of watermarking, and no aggregation of watermarking occurs.Figure 5
The results of encoding with and without a discriminator: top: cover imageIco; second: encoded image Ien; third: noised image Ino; fourth: residual signal between Ien and Ico; and bottom: the normalization of the residual signal.


### 3.3.3. Different Mixed-Frequency Channel Attention

To demonstrate that our two-branch structure is superior to other combined mixed-frequency channel attention blocks, we conduct experiments for the encoder with different frequency channel attention structures. We proposed another four kinds of structures to be applied in the encoder. The first is called LFCA, which only consists of several FCA blocks with low-frequency components, the second is called SE&LFCA, in which we insert an FCA block behind the SE blocks, the third is composed of several SE blocks, and the last is called LFCA&SE, in which we insert an FCA block in front of the SE blocks. Their detailed structures are shown in Figure6. We list the results of experiments separately under JPEG compression and combined noises for the above-mentioned four structures in Tables 5 and 6.Figure 6
The other structures of mixed-frequency channel attention, which are proposed in the ablation experiment. The structure called LFCA is only composed of several frequency channel attention blocks with low-frequency components. The structure called SE consists of several SE channel attention blocks. The structure called SE&LFCA inserts an FCA block behind the SE blocks. The structure called LFCA&SE inserts FCA blocks in front of the SE blocks.Table 5
Comparison with the ablation experimental results, with the best results in bold and second bests in blue. We compared the performance with four mixed-frequency channel attention variants under JPEG compressionQ=50.
ModelSELFCASE&LFCALFCA&SEOursImage size128×128128×128128×128128×128128×128Message length6464646464Noise layerMixedMixedMixedMixedMixedPSNR37.7037.3238.2137.4938.13SSIM0.94450.94280.95530.94870.9472BER0.0082%0.0081%0.0059%0.0081%0.0078%Table 6
BER comparison between the proposed scheme and ablation schemes, with the best results in bold, second bests in blue, and the worst results in red. The strength factor is adjusted for comparison underPSNR=33.5.
NoiseIdentityCrop outp=0.3Drop outp=0.3Cropp=0.035GFσ=2JPEGQ=50SE0%0.0027%0.0093%5.07%0.011%3.22%LFCA0%0.0%0.0027%4.23%0.031%5.62%SE&LFCA0%0.0040%0.0179%6.21%0.185%22.87%LFCA&SE0%0.0047%0.0053%18.06%0.006%9.12%Ours0%0.0013%0.0080%3.24%0.293%2.61%The channel attention mechanism assigns weights to the feature maps. SE only selects the lowest-frequency component coefficients of the 2D-DCT to enhance all channel feature maps through multiple SE blocks, while LFCA chooses to divide the feature maps on the channels and select multiple low-frequency component coefficients of the 2D-DCT to enhance through several LFCA blocks. We believe that when the noise layer only includes JPEG compression, the weights of LFCA enhancement are spread over multiple low-frequency components relative to SE, and thus, the performance will be worse than that of SE. However, combining SE blocks and LFCA blocks gives better performance. As can be seen from Table5, the performance of SE&LFCA and LFCA&SE is better than that of SE and LFCA. SE&LFCA firstly allocates the lowest-frequency component coefficients through an SE block and then uses several LFCA blocks to enhance multifrequency component coefficients on the basis of the lowest-frequency component, which has a good effect. Although LFCA&SE is also composed of an SE block and several LFCA blocks, its effect is not as good as that of SE&LFCA. We believed that this is caused by LFCA assigning weights in the first place.Our parallel structure is a better way of feature fusion when the noise layer includes multinoises. We believe that the reason why the experimental results of SE&LFCA and LFCA&SE perform worse is that they have no skip connection. Our proposed method achieves better performance with skip connection of FCA, which is confirmed by the experimental results in Table7.Table 7
The ablation experiment of skip connection. Baseline: no attention model; +SE: add SE channel attention networks; +skip connection: based on SEnets, add a skip connection of LFCA.
(a)ModelMetricJPEGQ=50Cropp=0.035Crop outp=0.3Drop outp=0.3GFσ=2BaselinePSNR32.6032.6032.5932.5732.59+SE32.5532.5832.5732.5632.56+skip connection (LFCA)32.7532.7632.7632.7332.76BaselineSSIM0.81500.81570.81490.81460.8150+SE0.81610.81650.81600.81570.8159+skip connection (LFCA)0.82630.82660.82630.82590.8265BaselineBER3.09%6.96%0%0.0007%0.168%+SE2.19%4.30%0.0013%0.0067%0.0054%+skip connection (LFCA)1.94%3.08%0.0067%0.0040%0.228%(b)Salt&Pepperp=0.1GNp=0.1MFs=332.5932.5932.5832.5532.5632.5532.7532.7632.750.81480.81530.81540.81560.81590.81600.82600.82710.82630.830%1.23%0%0.813%1.51%0%0.665%0.859%0%


### 3.3.4. Selection Scheme of Frequency Components

To demonstrate that the FCA attention block in our method chooses the low-frequency component coefficients of the DCT to improve the robustness to JPEG compression, we select 16 components of low frequency, 16 components of middle frequency, and 16 components of high frequency as the weight coefficients of FCA from the8×8 coefficients, respectively, and train them under JPEG compression. It can be seen from Table 8 that the selection of frequency domain components has a certain impact on the robustness and imperceptibility of the model. When the low-frequency components are selected, the metrics such as PSNR, SSIM, and BER all reach the highest.Table 8
BER comparison between the proposed scheme and ablation schemes. We train three different ways of frequency domain component selection for the FCA attention block in the mixed-frequency channel attention block under JPEG compression. Low frequency: 16 coefficients of low-frequency components; middle frequency: 16 coefficients of middle-frequency components; and high frequency: 16 coefficients of high-frequency components.
ModelPSNRSSIMBERHigh frequency37.770.94070.6781%Middle frequency38.020.94720.5782%Low frequency38.130.94720.0078%


### 3.3.5. Skip Connection

To show the important role of introducing frequency analysis and skip connection, we trained three different watermarking models under a mixed-noise layer separately: baseline: without attention networks in the encoder; +SE: with the addition of the SE channel attention blocks in the encoder; and +skip connection: based on +SE, with the addition of the LFCA attention block via skip connection. Table7 shows the results of experiments, where the performance of the model by adding SE attention block is improved compared to baseline under most of the noises. However, we find that the embedding of the watermark information by adding the SE attention block is more concentrated in the low-frequency region which is less affected by the Gaussian filter but will be more affected by the Gaussian noise. In order to further improve the robustness for noises such as JPEG compression, we added the LFCA attention block by skip connection on the basis of the SE attention blocks, and the experimental results show that the quality of the encoded image is improved by skip connection, the best robustness is achieved for most distortions, and our watermark embedding assignment is more reasonable.


## 3.3.1. Strength Factor

The strength factor is a parameter used to balance robustness and imperceptibility. We set the value of the strength factorS, from 0.1 to 2.0, with an interval of 0.1, and test the model under different quality factors for JPEG compression. The results are shown in Table 4. With the increment of S, PSNR and SSIM values decrease, the quality of the encoded image becomes worse, and the extraction accuracy becomes higher. In the study, we adjust the value of S to obtain the similar visual quality of different models for fair comparison.Table 4
BER, PSNR, and SSIM values under different strength factorsS and quality factors of JPEG Q.
Strength factor0.40.60.81.01.21.4BERQ=1035.03%27.91%21.80%16.86%13.02%10.06%Q=3015.65%5.94%1.63%0.33%0.047%0.0053%Q=507.74%1.11%0.0778%0.0050%0.0012%0.00%Q=703.87%0.370%0.0170%0.0009%0.00%0.00%Q=902.53%0.202%0.0078%0.0006%0.00%0.00%PSNR45.9242.4039.8937.9536.3735.03SSIM0.98930.97730.96230.94550.92740.9086


## 3.3.2. Discriminator

To demonstrate that the discriminator can help the encoder generate higher-quality images, we trained the noise-free model with and without the discriminator separately. As can be seen from the normalized watermarking residuals in Figure5, the watermarking model without the discriminator does not produce a uniform distribution of watermarking and produces visual artifacts on the resulting watermarked image. However, the watermarking model with a discriminator generates an even distribution of watermarking, and no aggregation of watermarking occurs.Figure 5
The results of encoding with and without a discriminator: top: cover imageIco; second: encoded image Ien; third: noised image Ino; fourth: residual signal between Ien and Ico; and bottom: the normalization of the residual signal.


## 3.3.3. Different Mixed-Frequency Channel Attention

To demonstrate that our two-branch structure is superior to other combined mixed-frequency channel attention blocks, we conduct experiments for the encoder with different frequency channel attention structures. We proposed another four kinds of structures to be applied in the encoder. The first is called LFCA, which only consists of several FCA blocks with low-frequency components, the second is called SE&LFCA, in which we insert an FCA block behind the SE blocks, the third is composed of several SE blocks, and the last is called LFCA&SE, in which we insert an FCA block in front of the SE blocks. Their detailed structures are shown in Figure6. We list the results of experiments separately under JPEG compression and combined noises for the above-mentioned four structures in Tables 5 and 6.Figure 6
The other structures of mixed-frequency channel attention, which are proposed in the ablation experiment. The structure called LFCA is only composed of several frequency channel attention blocks with low-frequency components. The structure called SE consists of several SE channel attention blocks. The structure called SE&LFCA inserts an FCA block behind the SE blocks. The structure called LFCA&SE inserts FCA blocks in front of the SE blocks.Table 5
Comparison with the ablation experimental results, with the best results in bold and second bests in blue. We compared the performance with four mixed-frequency channel attention variants under JPEG compressionQ=50.
ModelSELFCASE&LFCALFCA&SEOursImage size128×128128×128128×128128×128128×128Message length6464646464Noise layerMixedMixedMixedMixedMixedPSNR37.7037.3238.2137.4938.13SSIM0.94450.94280.95530.94870.9472BER0.0082%0.0081%0.0059%0.0081%0.0078%Table 6
BER comparison between the proposed scheme and ablation schemes, with the best results in bold, second bests in blue, and the worst results in red. The strength factor is adjusted for comparison underPSNR=33.5.
NoiseIdentityCrop outp=0.3Drop outp=0.3Cropp=0.035GFσ=2JPEGQ=50SE0%0.0027%0.0093%5.07%0.011%3.22%LFCA0%0.0%0.0027%4.23%0.031%5.62%SE&LFCA0%0.0040%0.0179%6.21%0.185%22.87%LFCA&SE0%0.0047%0.0053%18.06%0.006%9.12%Ours0%0.0013%0.0080%3.24%0.293%2.61%The channel attention mechanism assigns weights to the feature maps. SE only selects the lowest-frequency component coefficients of the 2D-DCT to enhance all channel feature maps through multiple SE blocks, while LFCA chooses to divide the feature maps on the channels and select multiple low-frequency component coefficients of the 2D-DCT to enhance through several LFCA blocks. We believe that when the noise layer only includes JPEG compression, the weights of LFCA enhancement are spread over multiple low-frequency components relative to SE, and thus, the performance will be worse than that of SE. However, combining SE blocks and LFCA blocks gives better performance. As can be seen from Table5, the performance of SE&LFCA and LFCA&SE is better than that of SE and LFCA. SE&LFCA firstly allocates the lowest-frequency component coefficients through an SE block and then uses several LFCA blocks to enhance multifrequency component coefficients on the basis of the lowest-frequency component, which has a good effect. Although LFCA&SE is also composed of an SE block and several LFCA blocks, its effect is not as good as that of SE&LFCA. We believed that this is caused by LFCA assigning weights in the first place.Our parallel structure is a better way of feature fusion when the noise layer includes multinoises. We believe that the reason why the experimental results of SE&LFCA and LFCA&SE perform worse is that they have no skip connection. Our proposed method achieves better performance with skip connection of FCA, which is confirmed by the experimental results in Table7.Table 7
The ablation experiment of skip connection. Baseline: no attention model; +SE: add SE channel attention networks; +skip connection: based on SEnets, add a skip connection of LFCA.
(a)ModelMetricJPEGQ=50Cropp=0.035Crop outp=0.3Drop outp=0.3GFσ=2BaselinePSNR32.6032.6032.5932.5732.59+SE32.5532.5832.5732.5632.56+skip connection (LFCA)32.7532.7632.7632.7332.76BaselineSSIM0.81500.81570.81490.81460.8150+SE0.81610.81650.81600.81570.8159+skip connection (LFCA)0.82630.82660.82630.82590.8265BaselineBER3.09%6.96%0%0.0007%0.168%+SE2.19%4.30%0.0013%0.0067%0.0054%+skip connection (LFCA)1.94%3.08%0.0067%0.0040%0.228%(b)Salt&Pepperp=0.1GNp=0.1MFs=332.5932.5932.5832.5532.5632.5532.7532.7632.750.81480.81530.81540.81560.81590.81600.82600.82710.82630.830%1.23%0%0.813%1.51%0%0.665%0.859%0%


## 3.3.4. Selection Scheme of Frequency Components

To demonstrate that the FCA attention block in our method chooses the low-frequency component coefficients of the DCT to improve the robustness to JPEG compression, we select 16 components of low frequency, 16 components of middle frequency, and 16 components of high frequency as the weight coefficients of FCA from the8×8 coefficients, respectively, and train them under JPEG compression. It can be seen from Table 8 that the selection of frequency domain components has a certain impact on the robustness and imperceptibility of the model. When the low-frequency components are selected, the metrics such as PSNR, SSIM, and BER all reach the highest.Table 8
BER comparison between the proposed scheme and ablation schemes. We train three different ways of frequency domain component selection for the FCA attention block in the mixed-frequency channel attention block under JPEG compression. Low frequency: 16 coefficients of low-frequency components; middle frequency: 16 coefficients of middle-frequency components; and high frequency: 16 coefficients of high-frequency components.
ModelPSNRSSIMBERHigh frequency37.770.94070.6781%Middle frequency38.020.94720.5782%Low frequency38.130.94720.0078%


## 3.3.5. Skip Connection

To show the important role of introducing frequency analysis and skip connection, we trained three different watermarking models under a mixed-noise layer separately: baseline: without attention networks in the encoder; +SE: with the addition of the SE channel attention blocks in the encoder; and +skip connection: based on +SE, with the addition of the LFCA attention block via skip connection. Table7 shows the results of experiments, where the performance of the model by adding SE attention block is improved compared to baseline under most of the noises. However, we find that the embedding of the watermark information by adding the SE attention block is more concentrated in the low-frequency region which is less affected by the Gaussian filter but will be more affected by the Gaussian noise. In order to further improve the robustness for noises such as JPEG compression, we added the LFCA attention block by skip connection on the basis of the SE attention blocks, and the experimental results show that the quality of the encoded image is improved by skip connection, the best robustness is achieved for most distortions, and our watermark embedding assignment is more reasonable.


## 4. Discussion and Analysis

According to Figures2, 3, and 6 and Tables 5 and 6, some analyses are given as follows.
(1)
Our scheme significantly improved visual quality compared with relative schemes. We can find that the secret messages are embedded in most areas of the cover image including low-frequency and high-frequency components from Figure3(2)
To further reflect our scheme, we calculated the indicators SSIM and PSNR. SSIM can show the overall structure of images. PSNR is calculated based on the discrepancy between the corresponding two pixel values. PSNR and SSIM are utilized jointly to evaluate the visual quality of the encoded image(3)
A frequency channel attention block with selected low-frequency channel components can effectively improve the robustness and imperceptibility of the proposed watermarking model under JPEG compression and combined noise layer, as shown in Tables5 and 6. However, the performance of the variants suggests that the balance of robustness and invisibility is very challenging. Our scheme chose the two-branch structure to concentrate on the features from the LFCA block and SE blocks. Experimental results demonstrate that skip connections provide better performance gains for the whole model(4)
The performance of the watermarking algorithm depends largely on the selection of frequency channel components. We chose 16 low-frequency channel components according to the zigzag form. Compared to the lowest-frequency channel components extracted by the SE block and medium-high-frequency channel components, the multi-low-frequency channel components include the information that is beneficial to embedding messages and defence distortions(5)
Although the method we proposed at the current stage has good performance in robustness and imperceptibility, we believe that it will also cause computational costs to a certain extent. Therefore, we hope to explore more concise and effective selection methods of channel feature components in the future


## 5. Conclusions

In the paper, we proposed a novel mixed-frequency channel attention block to improve the robustness and imperceptibility of existing deep robust image watermarking algorithms for JPEG compression. We divide the 2D-DCT frequency space into8×8 parts according the principle of JPEG compression and utilize the SE block to obtain the lowest-frequency component in 2D-DCT domain, which is equal to GAP operation, as the weight coefficient for input. Then, we select the 16 low-frequency components in the 2D-DCT domain as the weight coefficients by the FCA block according to the zigzag form. Finally, we concentrate on the feature maps by skip connection in the channel dimension. Besides, we use an optional diffusion block in [28] for robustness against geometric attack. The comprehensive experiments have proven that the proposed method performs better in not only robustness but also image quality. Skip connection and the selection scheme of frequency components prove to be effective. In the future, we will also explore a more suitable channel selection method for watermarking embedding.



---

*Source: 9880038-2022-07-14.xml*