# Visual Security Assessment via Saliency-Weighted Structure and Orientation Similarity for Selective Encrypted Images

Authors: Zhengguo Wu; Kai Zhang; Yannan Ren; Jing Li; Jiande Sun; Wenbo Wan
Journal: Security and Communication Networks
 (2021)

---

## Abstract

Selective encryption has been widely used in image privacy protection. Visual security assessment is necessary for the effectiveness and practicability of image encryption methods, and there have been a series of research studies on this aspect. However, these methods do not take into account perceptual factors. In this paper, we propose a new visual security assessment (VSA) by saliency-weighted structure and orientation similarity. Considering that the human visual perception is sensitive to the characteristics of selective encrypted images, we extract the structure and orientation feature maps, and then similarity measurements are conducted on these feature maps to generate the structure and orientation similarity maps. Next, we compute the saliency map of the original image. Then, a simple saliency-based pooling strategy is subsequently used to combine these measurements and generate the final visual security score. Extensive experiments are conducted on two public encryption databases, and the results demonstrate the superiority and robustness of our proposed VSA compared with the existing most advanced work.


---

## Body

## 1. Introduction

Nowadays, with the widely pervasive usage of interaction devices, such as cameras, cloud storage devices, and the explosive growth of digital images, privacy protection has attracted a lot of attention from researchers [1–5]. Various security schemes, such as digital watermarking [6–8], steganography [9], and encryption [10], have been developed to protect copyright, and encryption is the mostly accepted approach which can ensure the security and integrity of data all the time. Roughly, the existing image encryption methods can be divided into two categories: full encryption and selective encryption. Full encryption refers to encrypting the entire image; therefore, we cannot get any information about the original image from the encrypted image. However, the content information of the image cannot be revealed if several kinds of redundant information are unencrypted. For this reason, traditional full encryption methods such as AES are not suitable for image data, because these methods always encrypt all the information of the image which cost a lot of time. Therefore, researchers proposed the selective encryption algorithm that has been widely used to protect the visual content of multimedia by only encrypting the specified parts of the multimedia data. A great variety of selective encryption algorithms [10–16] have been proposed in recent decades. Compared with full encryption algorithms, selective encryption has two main advantages as follows. First, it can be extremely fast on encryption and decryption because only a portion of the data needs to be encrypted. Second, the selective encrypted multimedia data can prevent the abuse of the essential visual property of the original data. These advantages make selective encryption highly desirable for protecting more and more image and video data which hide a large amount of personal privacy on the network.The purpose of security analysis for selective image encryption is to measure the degree of visual security of selective encrypted images. Visual security analysis can measure the performance of the selective encryption methods and then help us to optimize the encryption methods. Since humans are the ultimate receivers of images, subjective tests conducted by human viewers are the most suitable and accurate way to evaluate the visual security of selective encrypted images. However, such tests are too time-consuming and laborious to accomplish real-time applications. For this reason, visual security assessment [17] (VSA) is proposed to evaluate the visual security of selective encrypted images by measuring the unintelligibility or unrecognizability of the image automatically, which indicates the amount of useful information about the original image that an attacker can obtain from its selective encrypted image via visual perception. The higher the unrecognizability degree of the selective encrypted image is, the less visual information the attacker can obtain. In such a situation, it becomes more difficult for an attacker to obtain information about the original image and the selective encryption method is more secure.In the past decades, many efforts have been conducted to design VSAs. At the beginning, researchers believe that visual security of images has a strong relationship with image quality. Therefore, they directly used the well-known image quality assessment [18] (IQA) methods to evaluate the visual security degree of selective encrypted images, such as peak signal-to-noise ratio (PSNR), structural similarity (SSIM) [19] and visual information fidelity (VIF) [20], and the images with lower quality tend to have higher security. However, these metrics may be inconsistent with the concept of security strength. For example, an image with a lower PSNR value may be even more recognizable than one with a higher PSNR value. These IQA methods do not take full account of the characteristics of the selective encrypted images. For selective encrypted images, an important feature is that the skeleton of the image is still intelligible but the details are almost unintelligible [21]. On the other hand, the structure information can express the skeleton of an image which plays a more important role in selective encrypted images. Subsequently, several VSAs have been developed based on some visual features of selective encrypted images, e.g., edge similarity score (ESS) [22] based on the edge, luminance similarity score (LSS) [22] utilizing luminance feature, local feature-based visual security (LFBVS) [23] using luminance and localized gradient, and the visual security index-based Canny (VSI-Canny) [21] which extracted edge and texture features. However, these VSAs do not fully consider the role of visual perception [24–26] factor in VSAs, because the visual perception of each region differs from another according to the principle of the human visual system (HVS), which also have different impacts on visual security evaluation. Additionally, HVS presents an obvious visual saliency mechanism. HVS focuses only on these important regions for detailed perception and withdraws the other regions. The regions have high saliency values play more important roles than the other regions for visual perception, and the information leakage on the high saliency regions has a larger influence on the visual security assessment.Motivated by the problems mentioned above, in this paper, we propose a visual security assessment via saliency-weighted structure and orientation similarity. Structure is the basic element that conveys important visual information, and selective encryption can cause obvious structure changes of an image [21]. Therefore, we can measure the visual security of selective encrypted images by the change of structure. The gradient magnitude (GM) and the phase congruence [27] (PC) are widely used to extract the image structure information. However, GM and PC cannot effectively reflect the structure degradation in the selective encryption images. GM is sensitive to luminance and it can well reflect the changes of image luminance [28]. However, this characteristic of GM also makes it is not effective to extract the structure information of the areas with similar grayscale values. Compared with GM, PC is not affected by luminance [27]. However, PC cannot extract the clear structure information of the areas with similar frequencies as it is calculated based on frequency [27]. Therefore, we integrated PC with GM to obtain the structure features of the selective encrypted images. Studies show that HVS is highly adapted to extract orientation information [29] and selective encryption can cause obvious orientation changes of an image. Therefore, we can extract the orientation information of a selective encrypted image to measure its security. The structure and the orientation feature maps are extracted from both original and selective encrypted images. Finally, an image saliency-based pooling strategy is introduced to combine these measurements and generate a visual security score. Our main contributions can be summarized as follows:(1)
We propose to extract the structure and orientation features for the visual security evaluation of the selective encrypted images, because selective encryption can cause obvious changes in structure and orientation of an image and the HVS is highly sensitive to the change of structure and orientation. We combine GM and PC to extract the structure information to measure the structure similarity of original images and selective encrypted images and utilize the change of image orientation to measure the orientation similarity.(2)
Considering that different regions of an image have different effects on visual security assessment of selective encrypted images, we combine the saliency map with the structure and orientation similarity maps to generate the final VSA.(3)
We conduct comparative experiments on two common encryption image databases to evaluate the performance of our proposed VSA. The experimental results show that the proposed method achieves superior and robust performance compared with other state-of-the-art VSAs, especially on the images in low- and moderate-quality ranges.The structure of the rest of the paper is as follows. Section2 reviews the related work. The details of our proposed VSA method are in Section 3. Then, we describe the experimental evaluation of our proposed VSA and existing VSAs in Section 4. Finally, Section 5 concludes this paper.


## 2. Related Work

A variety of methods have been proposed to estimate the visual security of selective encrypted images. The initial solutions usually employ well-known IQAs to evaluate the visual security. Subsequently, several VSAs have been proposed to evaluate the visual security.


### 2.1. Image Quality Assessment

Many researchers believe that the images with higher visual security tend to have lower visual quality, so many IQA methods designed for the assessment of image visual quality have been employed to measure the visual security of selective encrypted images. For instance, PSNR is the simplest and the most widely used method [30, 31]. PSNR, which evaluates visual security by calculating the Euclidean distance between the original image and distorted image, is the simplest and most popular visual quality assessment metric. SSIM [19] is also adopted for visual security evaluation [30, 31] by measuring the similarities of luminance, contrast, and structure between two images in consideration of the HVS. VIF [20] is another IQA method used to estimate the visual security of selective encrypted images. It measures the amount of information contained in original and selective encrypted images, respectively, and then measures the relationship between image information and visual quality. However, these IQA metrics often exhibit unsatisfactory performance when they are used to estimate the visual security of selective encrypted images of low quality. Since the task of IQA is inconsistent with that of VSA, an image with poor visual quality may not indicate its visual security [21].For example, an image with a higher VIF, PSNR, or SSIM may even be more visually secure than one with a lower value of one of these indices. Figures1(a)–1(c) show the performance of the PSNR, VIF, and SSIM indices on several images from the PEID database [33]. Figure 1(a) shows an original image, and Figures 1(b) and 1(c) show two encrypted images. It is clear that Figure 1(c) has a higher visual security, but this image is found to have better visual quality as assessed using the PSNR, VIF, and SSIM.Figure 1
The performances of IQAs on images from the IVC-SelectEncrypt database [32] and PEID [33] database. (a) Original image from the PEID database. (b) Encrypted image with PSNR = 14.5 and SSIM = 0.19. (c) Encrypted image with PSNR = 20.3 and SSIM = 0.51. (d) Original image from the IVC-SelectEncrypt database. (e) Encrypted image with LSS = −0.007 and VSI-Canny = 0.0752. (f) Encrypted image with LSS = −0.02 and VSI-Canny = 0.1893.
(a)(b)(c)(d)(e)(f)We can find that many IQAs cannot achieve excellent performance on visual security assessment because the targets of image quality assessment and visual security assessment are different: image quality assessment focuses on the fidelity of an image, but visual security assessment is concerned with the leakage degree of an image’s content.


### 2.2. Visual Security Assessment

Several VSAs have been proposed to evaluate the visual security of selective encrypted images. They are usually more accurate and effective than IQA methods because they are specifically designed for the visual security evaluation of selective encrypted images. Mao and Wu [22] proposed the ESS and LSS to compute the edge similarity and the luminance similarity between original and selective encrypted images. However, the ESS and LSS focus only on local information of the images, which may not cover the various types of distortions that appear in selective encrypted images. Tong et al. [23] presented the LFBVS by considering various types of distortions present in selective encrypted images and measured the similarities of luminance and the localized gradient between original and selective encrypted images. Although the LFBVS utilizes more visual information compared with the ESS and LSS, its performance is still unsatisfactory when tested on various encrypted image databases. Xiang et al. [21] proposed the VSI-Canny by calculating the edge and texture similarities between original and selective encrypted images. VSI-Canny considers more visual features of selective encrypted images and has relatively good performance, but it does not consider the image’s visual saliency, which is a critical property of the HVS.For example, Figures1(d)–1(f) illustrate the performance of different VSA indices on an image from the IVC-SelectEncrypt database. Figure 1(d) shows the original image, and two encrypted versions of which are shown in Figures 1(e) and 1(f). It is clear that Figure 1(f) is more visually secure than Figure 1(e). However, Figure 1(f) has higher LSS and VSI-Canny values than Figure 1(e).As mentioned above, the problems of the existing visual security metrics exhibit many aspects. These questions will lead to the inaccurate evaluation of image security by visual security indicators. We consider and address these issues in our proposed scheme, as described in the following section.


## 2.1. Image Quality Assessment

Many researchers believe that the images with higher visual security tend to have lower visual quality, so many IQA methods designed for the assessment of image visual quality have been employed to measure the visual security of selective encrypted images. For instance, PSNR is the simplest and the most widely used method [30, 31]. PSNR, which evaluates visual security by calculating the Euclidean distance between the original image and distorted image, is the simplest and most popular visual quality assessment metric. SSIM [19] is also adopted for visual security evaluation [30, 31] by measuring the similarities of luminance, contrast, and structure between two images in consideration of the HVS. VIF [20] is another IQA method used to estimate the visual security of selective encrypted images. It measures the amount of information contained in original and selective encrypted images, respectively, and then measures the relationship between image information and visual quality. However, these IQA metrics often exhibit unsatisfactory performance when they are used to estimate the visual security of selective encrypted images of low quality. Since the task of IQA is inconsistent with that of VSA, an image with poor visual quality may not indicate its visual security [21].For example, an image with a higher VIF, PSNR, or SSIM may even be more visually secure than one with a lower value of one of these indices. Figures1(a)–1(c) show the performance of the PSNR, VIF, and SSIM indices on several images from the PEID database [33]. Figure 1(a) shows an original image, and Figures 1(b) and 1(c) show two encrypted images. It is clear that Figure 1(c) has a higher visual security, but this image is found to have better visual quality as assessed using the PSNR, VIF, and SSIM.Figure 1
The performances of IQAs on images from the IVC-SelectEncrypt database [32] and PEID [33] database. (a) Original image from the PEID database. (b) Encrypted image with PSNR = 14.5 and SSIM = 0.19. (c) Encrypted image with PSNR = 20.3 and SSIM = 0.51. (d) Original image from the IVC-SelectEncrypt database. (e) Encrypted image with LSS = −0.007 and VSI-Canny = 0.0752. (f) Encrypted image with LSS = −0.02 and VSI-Canny = 0.1893.
(a)(b)(c)(d)(e)(f)We can find that many IQAs cannot achieve excellent performance on visual security assessment because the targets of image quality assessment and visual security assessment are different: image quality assessment focuses on the fidelity of an image, but visual security assessment is concerned with the leakage degree of an image’s content.


## 2.2. Visual Security Assessment

Several VSAs have been proposed to evaluate the visual security of selective encrypted images. They are usually more accurate and effective than IQA methods because they are specifically designed for the visual security evaluation of selective encrypted images. Mao and Wu [22] proposed the ESS and LSS to compute the edge similarity and the luminance similarity between original and selective encrypted images. However, the ESS and LSS focus only on local information of the images, which may not cover the various types of distortions that appear in selective encrypted images. Tong et al. [23] presented the LFBVS by considering various types of distortions present in selective encrypted images and measured the similarities of luminance and the localized gradient between original and selective encrypted images. Although the LFBVS utilizes more visual information compared with the ESS and LSS, its performance is still unsatisfactory when tested on various encrypted image databases. Xiang et al. [21] proposed the VSI-Canny by calculating the edge and texture similarities between original and selective encrypted images. VSI-Canny considers more visual features of selective encrypted images and has relatively good performance, but it does not consider the image’s visual saliency, which is a critical property of the HVS.For example, Figures1(d)–1(f) illustrate the performance of different VSA indices on an image from the IVC-SelectEncrypt database. Figure 1(d) shows the original image, and two encrypted versions of which are shown in Figures 1(e) and 1(f). It is clear that Figure 1(f) is more visually secure than Figure 1(e). However, Figure 1(f) has higher LSS and VSI-Canny values than Figure 1(e).As mentioned above, the problems of the existing visual security metrics exhibit many aspects. These questions will lead to the inaccurate evaluation of image security by visual security indicators. We consider and address these issues in our proposed scheme, as described in the following section.


## 3. Proposed Visual Security Assessment

In this work, we describe our proposed VSA and the flowchart of the proposed VSA is shown in Figure2. First, we combine GM and PC to extract the structure information and we can compute the structure similarity map of the original and selective encrypted image. Secondly, based on the fact that the HVS is sensitive to the change of orientation, we extract the orientation information by the GM and we can compute the orientation similarity map. Next, considering that the security of a selective encrypted image depends on the degree of disclosure of its visual content, which is obtained by comparing it with the original image, we only compute the saliency map of the original image. At last, the generated structure and orientation similarity maps are further fused by saliency-based polling method to obtain the final score.Figure 2
Flowchart illustration of the proposed security model for selective encrypted images.


### 3.1. Structure Similarity

The structure of an image has important information which is highly sensitive to the visual perception. Both GM and PC can extract structural information of images and we found that they can complement each other. Therefore, we integrated the GM map with the PC map to generate the structure features of selective encrypted images.


#### 3.1.1. Gradient Magnitude

Image gradient magnitude can be defined as a transition in intensity. The GM of an image is represented by a vector which consists of gradient in the horizontal and vertical directions at each pixel, and it reflects the maximum strength of structure variation. The gradient magnitude of an image is defined as(1)GMi,j=Gh2i,j+Gv2i,j,where (i, j) is the index of an image I. In this work, for image I, Gh and Gv are calculated as(2)Gh=F∗I,Gv=FT∗I,where ∗ and T denote the convolution and transpose, respectively, and F is the gradient operator:(3)F=10−110−110−1.As shown in Figure3(b), it can be seen that the GM maps of the encrypted images have obvious changes. However, there are no obvious changes in the GM values in some areas with similar grayscale values. GM is sensitive to luminance; therefore, it can well reflect the changes of image luminance [28]. However, this characteristic of GM also makes it is not effective to extract the structure information of the areas with similar grayscale values.Figure 3
Illustrations of different feature maps. (a) First column is the input images. (b) Second column is the GM maps of the images in (a). (c) Third column is the PC maps of the images in (a). (d) Fourth column is the ST maps of the images in (a). (e) Fifth column is the orientation maps of the images in (a). (f) Sixth column is the saliency maps of the images in (a). And the images in second, fourth, and sixth rows in (a) are the encrypted images of first, third, and fifth rows in (a), respectively.
(a)(b)(c)(d)(e)(f)


#### 3.1.2. Phase Congruency

The phase congruency model [27], which is based on frequency domain processing of an image, means that features with similar edges appear more frequently at the same stage. It assumes that the visual system is more competent in performing operations using the phase and amplitude of the individual frequency components in an image than handing of image information spatially. Compared with GM, PC is invariant to local smooth luminance changes. Given an image I, its PC is computed as(4)PCi,j=∑s∑θWi,jAsθi,j⋅Δφsθi,j−T∑s∑θAsθi,j+ε,with(5)Δφsθi,j=cosφsθi,j−φsθ¯i,j−sinφsθi,j−φsθ¯i,j,Wi,j=11+eγc−vi,j,Asθ=Esθ2i,j+Osθ2i,j,φsθi,j=atanEsθi,j,Osθi,j,Esθi,j,Osθi,j=Ii,j∗MsθE,Ii,j∗MsθO,where W(i, j) denotes the weighting parameter to reduce the effect of frequency spread at position (i, j); vi,j=1/N∑s∑θAsθi,jAmaxi,j+lnormi,j−1 denotes the manipulating function by weighting; N is the scale number; c offers a cutoff value for penalizing low PC values under it; lnorm(i, j)is the normalized luminance at (i, j) to avoid the effect of luminance. γ, as the gain variable, controls the cutoff sharpness; and symbol ⌊⋅⌋ aims to set negative value to zero. To determine two-dimensional phase congruency of a given image, the image is first convoluted with a Log-Gabor filters bank s and θ are the scale and orientation of the Log-Gabor filter. And the even symmetric filter and odd-symmetric filter at scale s and orientation θ are MsθE and MsθO, respectively. Asθ(i, j) and φsθ(i, j) represent the amplitude and phase at position (i, j), respectively; T is a quantity introduced to compensate image noise; ε is a small positive constant to preserve stability; φsθ¯ (i, j)represents the mean value of phase. Since it is out of scope to investigate these parameters’ influence on the PC map, in this study, we directly set them according to [27].The effectiveness of PC can be demonstrated in Figure3(c), and the PC maps of the encrypted images have obvious changes. However, there are no obvious changes in the PC values in some areas with similar frequencies. Compared with GM, PC is not affected by luminance [27]. However, PC cannot extract the clear structure information of the areas with similar frequencies as it is calculated based on frequency [28].


#### 3.1.3. Structure Map Integrating GM with PC

As mentioned above, GM and PC cannot effectively reflect the structure degradation in the selective encryption images. Therefore, after extracting the GM and PC of the image, we proposed to reflect the structure map of the image by integrating GM with PC, and the structure information ST of the image can be obtained as(6)STi,j=maxGMi,jGMmax,PCi,j,where (i, j) is the index of the pixel, GMmax represents the maximum value of GM, the maximum value of the corresponding positions of GM and PC is used to form ST, and GM uses the maximum value for normalization. If one of the two values has a larger value, the pixel is considered to be a structural feature point, and the maximum fusion strategy can comprehensively extract the structural features of the image. GM and PC can play complementary roles in extracting structural information. Here, we give an example in Figure 3 to demonstrate the effectiveness of our structure extraction method. Figure 3 shows different feature maps from the reference images and their selective encrypted images. Apparently, as compared with the other feature map, the proposed structure map has a remarkable effect on structure degradation of a selective encrypted image. From Figure 3(d), we can observe the more accurate and clearer structure information in the selective encrypted image.Analogous to the practice exercised in [19, 34], the structure similarity map SST of the original and the encrypted images can be measured as(7)SSTi,j=2⋅STOi,j⋅STEi,j+RSTOi,j2⋅STEi,j2+R,where SST(i, j)ε(0, 1], SO(i, j) and SE(i, j) are the structure maps of the original image O and the encrypted image E, respectively, and R is a positive constant used to avoid instability when the denominator converges to zero.


### 3.2. Orientation Similarity

In addition to the structure similarity, we also consider the orientation similarity between the original and encrypted images because the orientation information is an indispensable element for human visual perception. The orientation of an image, which has been widely used to the image quality assessment [35], conveys important information [29], which has an important effect on the visual security evaluation of selective encrypted images. The orientation change of each pixel can reflect the degradation of the selective encrypted image details.A visual pattern was built by orientation information in [35], which can be used for IQA. However, this pattern ignores some intuitive visual information; therefore, this pattern does not fully apply to VSA.Considering the above question, we design a new algorithm to compute the orientation similarity. In this work, for an imageI, the preferred orientation of each pixel is calculated as its gradient direction θI:(8)θIi,j=arctanGvi,jGhi,j⋅180π,where Gh(i, j) and Gv(i, j) are the gradient magnitudes along the horizontal and vertical directions, respectively, which can be obtained from equation (2). And (i, j) is the index of the pixel in I. So, we can obtain the quantitative orientation information. We give an example in Figure 3, and we can find that the orientation information of the image has obvious changes.Then, we compute the orientation changeDO of the original image O and its encrypted image E by calculating their distance:(9)DOi,j=θOi,j−θEi,j,where |.| denotes the absolute operation, (i, j) is the index of the pixel, and θO and θE are the orientation maps of the original image O and its encrypted image E, respectively.Because the range ofθO and θE is [−180, 180], the range of DO is [0, 360]. Considering that HVS has a similar perception of relative orientation (such as 90o and 270o), we set the range of DO to [0, 180] by setting DODO>180=360−DODO>180.Then, the orientation similarity mapSO of the original and the encrypted images can be measured as(10)SOi,j=1+log2DOi,jlog21/180.


### 3.3. Saliency-Weighted Pooling

It is observed that different regions have drastically different effects on the visual understanding of an image. Most of the contribution to visual perception is provided by the information loss and distortion in important regions. An image importance map refers to the important regions that provide a greater contribution to the visual perception, and such maps have been studied extensively in recent years. So, we highlight these important regions and suppress the other regions with a salient map for visual content extraction. To this end, the salient value of each pixel is required. As illustrated in Figure3(f), visual saliency map highlights the important regions in an image, and the visual saliency map can extract the important areas of an image and then get a better VSA. In the past decades, a large number of saliency models [36–40] have been proposed and these models can help us complete a better VSA.SST(i, j) and SO(i, j), obtained by equations (7) and (10), respectively, are two feature similarity maps with the same size as the image. However, we need a VSA score to represent the visual security. Therefore, we need a pooling method to compress the two feature maps into two scores to represent the feature similarities. In our work, we take the simple and classic saliency-weighted pooling method. Considering that the security of a selective encrypted image depends on the degree of disclosure of its visual content, which is obtained by comparing it with the original image. So, we select the original image’s saliency map SMO(i, j) to combine with the structure similarity map SST(i, j) and orientation similarity map SO(i, j), respectively:(11)VSST=∑i,jSSTi,j⋅SMOi,j∑i,jSMOi,j,VSO=∑i,jSOi,j⋅SMOi,j∑i,jSMOi,j.Considering that different saliency models affect the performance and communicational cost of our proposed VSA, we calculate the performance and running time of different saliency models. To eliminate the possible bias due to specific image selection, we randomly choose 100 images from the IVC-SelectEncrypt database and then calculate the average running time as the computation cost of each saliency model. Table1 shows the results. From Table 1, we can find that the more appropriate saliency model is GBVS. Therefore, as a simple but powerful saliency model, graph-based visual saliency [36] (GBVS) is employed. A saliency map of an original image generated by GBVS can be seen in Figure 3(f).Table 1
Performance comparison employing different saliency models: graph-based visual saliency (GBVS), context-aware visual saliency (CAVS), saliency estimation using region covariance (CovSal), saliency through adaptive whitening of color and scale features (AWS), and Boolean map saliency (BMS).
MetricPLCCSRCCKRCCRMSETime(s)Without saliency0.9430.9350.7690.4350.356GBVS [36]0.9560.9540.8150.3880.476CAVS [37]0.9580.9560.8150.38621.43CovSal [38]0.9490.9430.8090.3951.256AWS [39]0.9530.9470.8170.3960.789BMS [40]0.9510.9480.8130.4016.579After performing the similarity measurements on the structure and the orientation features between the original and encrypted images, respectively, the generated structure similarity VSST and orientation similarity VSO are combined together to calculate the visual security:(12)VSA=α⋅VSST+β⋅VSO,where α and β are two parameters used to adjust the relative importance of VSST and VSO. The structure and orientation features of an image are important which are highly sensitive to the visual perception. For selective encrypted images, an important feature is that the skeleton of the image is still intelligible but the details are almost unintelligible. Therefore, structure obviously plays a more important role than orientation, and we explore the effect of the structure information and orientation information, respectively, in Table 2. So that the value of α should be greater than β. In our experiments, α and β are set to 0.8 and 0.2, respectively, because this setting was found to be optimal.Table 2
Performance of structure similarity VSST, orientation similarity VSO, and final VSA.
MetricPLCCSRCCKRCCRMSEVSST0.9510.9470.8330.398VSO0.9430.9390.8020.418VSA0.9580.9560.8150.386


## 3.1. Structure Similarity

The structure of an image has important information which is highly sensitive to the visual perception. Both GM and PC can extract structural information of images and we found that they can complement each other. Therefore, we integrated the GM map with the PC map to generate the structure features of selective encrypted images.


### 3.1.1. Gradient Magnitude

Image gradient magnitude can be defined as a transition in intensity. The GM of an image is represented by a vector which consists of gradient in the horizontal and vertical directions at each pixel, and it reflects the maximum strength of structure variation. The gradient magnitude of an image is defined as(1)GMi,j=Gh2i,j+Gv2i,j,where (i, j) is the index of an image I. In this work, for image I, Gh and Gv are calculated as(2)Gh=F∗I,Gv=FT∗I,where ∗ and T denote the convolution and transpose, respectively, and F is the gradient operator:(3)F=10−110−110−1.As shown in Figure3(b), it can be seen that the GM maps of the encrypted images have obvious changes. However, there are no obvious changes in the GM values in some areas with similar grayscale values. GM is sensitive to luminance; therefore, it can well reflect the changes of image luminance [28]. However, this characteristic of GM also makes it is not effective to extract the structure information of the areas with similar grayscale values.Figure 3
Illustrations of different feature maps. (a) First column is the input images. (b) Second column is the GM maps of the images in (a). (c) Third column is the PC maps of the images in (a). (d) Fourth column is the ST maps of the images in (a). (e) Fifth column is the orientation maps of the images in (a). (f) Sixth column is the saliency maps of the images in (a). And the images in second, fourth, and sixth rows in (a) are the encrypted images of first, third, and fifth rows in (a), respectively.
(a)(b)(c)(d)(e)(f)


### 3.1.2. Phase Congruency

The phase congruency model [27], which is based on frequency domain processing of an image, means that features with similar edges appear more frequently at the same stage. It assumes that the visual system is more competent in performing operations using the phase and amplitude of the individual frequency components in an image than handing of image information spatially. Compared with GM, PC is invariant to local smooth luminance changes. Given an image I, its PC is computed as(4)PCi,j=∑s∑θWi,jAsθi,j⋅Δφsθi,j−T∑s∑θAsθi,j+ε,with(5)Δφsθi,j=cosφsθi,j−φsθ¯i,j−sinφsθi,j−φsθ¯i,j,Wi,j=11+eγc−vi,j,Asθ=Esθ2i,j+Osθ2i,j,φsθi,j=atanEsθi,j,Osθi,j,Esθi,j,Osθi,j=Ii,j∗MsθE,Ii,j∗MsθO,where W(i, j) denotes the weighting parameter to reduce the effect of frequency spread at position (i, j); vi,j=1/N∑s∑θAsθi,jAmaxi,j+lnormi,j−1 denotes the manipulating function by weighting; N is the scale number; c offers a cutoff value for penalizing low PC values under it; lnorm(i, j)is the normalized luminance at (i, j) to avoid the effect of luminance. γ, as the gain variable, controls the cutoff sharpness; and symbol ⌊⋅⌋ aims to set negative value to zero. To determine two-dimensional phase congruency of a given image, the image is first convoluted with a Log-Gabor filters bank s and θ are the scale and orientation of the Log-Gabor filter. And the even symmetric filter and odd-symmetric filter at scale s and orientation θ are MsθE and MsθO, respectively. Asθ(i, j) and φsθ(i, j) represent the amplitude and phase at position (i, j), respectively; T is a quantity introduced to compensate image noise; ε is a small positive constant to preserve stability; φsθ¯ (i, j)represents the mean value of phase. Since it is out of scope to investigate these parameters’ influence on the PC map, in this study, we directly set them according to [27].The effectiveness of PC can be demonstrated in Figure3(c), and the PC maps of the encrypted images have obvious changes. However, there are no obvious changes in the PC values in some areas with similar frequencies. Compared with GM, PC is not affected by luminance [27]. However, PC cannot extract the clear structure information of the areas with similar frequencies as it is calculated based on frequency [28].


### 3.1.3. Structure Map Integrating GM with PC

As mentioned above, GM and PC cannot effectively reflect the structure degradation in the selective encryption images. Therefore, after extracting the GM and PC of the image, we proposed to reflect the structure map of the image by integrating GM with PC, and the structure information ST of the image can be obtained as(6)STi,j=maxGMi,jGMmax,PCi,j,where (i, j) is the index of the pixel, GMmax represents the maximum value of GM, the maximum value of the corresponding positions of GM and PC is used to form ST, and GM uses the maximum value for normalization. If one of the two values has a larger value, the pixel is considered to be a structural feature point, and the maximum fusion strategy can comprehensively extract the structural features of the image. GM and PC can play complementary roles in extracting structural information. Here, we give an example in Figure 3 to demonstrate the effectiveness of our structure extraction method. Figure 3 shows different feature maps from the reference images and their selective encrypted images. Apparently, as compared with the other feature map, the proposed structure map has a remarkable effect on structure degradation of a selective encrypted image. From Figure 3(d), we can observe the more accurate and clearer structure information in the selective encrypted image.Analogous to the practice exercised in [19, 34], the structure similarity map SST of the original and the encrypted images can be measured as(7)SSTi,j=2⋅STOi,j⋅STEi,j+RSTOi,j2⋅STEi,j2+R,where SST(i, j)ε(0, 1], SO(i, j) and SE(i, j) are the structure maps of the original image O and the encrypted image E, respectively, and R is a positive constant used to avoid instability when the denominator converges to zero.


## 3.1.1. Gradient Magnitude

Image gradient magnitude can be defined as a transition in intensity. The GM of an image is represented by a vector which consists of gradient in the horizontal and vertical directions at each pixel, and it reflects the maximum strength of structure variation. The gradient magnitude of an image is defined as(1)GMi,j=Gh2i,j+Gv2i,j,where (i, j) is the index of an image I. In this work, for image I, Gh and Gv are calculated as(2)Gh=F∗I,Gv=FT∗I,where ∗ and T denote the convolution and transpose, respectively, and F is the gradient operator:(3)F=10−110−110−1.As shown in Figure3(b), it can be seen that the GM maps of the encrypted images have obvious changes. However, there are no obvious changes in the GM values in some areas with similar grayscale values. GM is sensitive to luminance; therefore, it can well reflect the changes of image luminance [28]. However, this characteristic of GM also makes it is not effective to extract the structure information of the areas with similar grayscale values.Figure 3
Illustrations of different feature maps. (a) First column is the input images. (b) Second column is the GM maps of the images in (a). (c) Third column is the PC maps of the images in (a). (d) Fourth column is the ST maps of the images in (a). (e) Fifth column is the orientation maps of the images in (a). (f) Sixth column is the saliency maps of the images in (a). And the images in second, fourth, and sixth rows in (a) are the encrypted images of first, third, and fifth rows in (a), respectively.
(a)(b)(c)(d)(e)(f)


## 3.1.2. Phase Congruency

The phase congruency model [27], which is based on frequency domain processing of an image, means that features with similar edges appear more frequently at the same stage. It assumes that the visual system is more competent in performing operations using the phase and amplitude of the individual frequency components in an image than handing of image information spatially. Compared with GM, PC is invariant to local smooth luminance changes. Given an image I, its PC is computed as(4)PCi,j=∑s∑θWi,jAsθi,j⋅Δφsθi,j−T∑s∑θAsθi,j+ε,with(5)Δφsθi,j=cosφsθi,j−φsθ¯i,j−sinφsθi,j−φsθ¯i,j,Wi,j=11+eγc−vi,j,Asθ=Esθ2i,j+Osθ2i,j,φsθi,j=atanEsθi,j,Osθi,j,Esθi,j,Osθi,j=Ii,j∗MsθE,Ii,j∗MsθO,where W(i, j) denotes the weighting parameter to reduce the effect of frequency spread at position (i, j); vi,j=1/N∑s∑θAsθi,jAmaxi,j+lnormi,j−1 denotes the manipulating function by weighting; N is the scale number; c offers a cutoff value for penalizing low PC values under it; lnorm(i, j)is the normalized luminance at (i, j) to avoid the effect of luminance. γ, as the gain variable, controls the cutoff sharpness; and symbol ⌊⋅⌋ aims to set negative value to zero. To determine two-dimensional phase congruency of a given image, the image is first convoluted with a Log-Gabor filters bank s and θ are the scale and orientation of the Log-Gabor filter. And the even symmetric filter and odd-symmetric filter at scale s and orientation θ are MsθE and MsθO, respectively. Asθ(i, j) and φsθ(i, j) represent the amplitude and phase at position (i, j), respectively; T is a quantity introduced to compensate image noise; ε is a small positive constant to preserve stability; φsθ¯ (i, j)represents the mean value of phase. Since it is out of scope to investigate these parameters’ influence on the PC map, in this study, we directly set them according to [27].The effectiveness of PC can be demonstrated in Figure3(c), and the PC maps of the encrypted images have obvious changes. However, there are no obvious changes in the PC values in some areas with similar frequencies. Compared with GM, PC is not affected by luminance [27]. However, PC cannot extract the clear structure information of the areas with similar frequencies as it is calculated based on frequency [28].


## 3.1.3. Structure Map Integrating GM with PC

As mentioned above, GM and PC cannot effectively reflect the structure degradation in the selective encryption images. Therefore, after extracting the GM and PC of the image, we proposed to reflect the structure map of the image by integrating GM with PC, and the structure information ST of the image can be obtained as(6)STi,j=maxGMi,jGMmax,PCi,j,where (i, j) is the index of the pixel, GMmax represents the maximum value of GM, the maximum value of the corresponding positions of GM and PC is used to form ST, and GM uses the maximum value for normalization. If one of the two values has a larger value, the pixel is considered to be a structural feature point, and the maximum fusion strategy can comprehensively extract the structural features of the image. GM and PC can play complementary roles in extracting structural information. Here, we give an example in Figure 3 to demonstrate the effectiveness of our structure extraction method. Figure 3 shows different feature maps from the reference images and their selective encrypted images. Apparently, as compared with the other feature map, the proposed structure map has a remarkable effect on structure degradation of a selective encrypted image. From Figure 3(d), we can observe the more accurate and clearer structure information in the selective encrypted image.Analogous to the practice exercised in [19, 34], the structure similarity map SST of the original and the encrypted images can be measured as(7)SSTi,j=2⋅STOi,j⋅STEi,j+RSTOi,j2⋅STEi,j2+R,where SST(i, j)ε(0, 1], SO(i, j) and SE(i, j) are the structure maps of the original image O and the encrypted image E, respectively, and R is a positive constant used to avoid instability when the denominator converges to zero.


## 3.2. Orientation Similarity

In addition to the structure similarity, we also consider the orientation similarity between the original and encrypted images because the orientation information is an indispensable element for human visual perception. The orientation of an image, which has been widely used to the image quality assessment [35], conveys important information [29], which has an important effect on the visual security evaluation of selective encrypted images. The orientation change of each pixel can reflect the degradation of the selective encrypted image details.A visual pattern was built by orientation information in [35], which can be used for IQA. However, this pattern ignores some intuitive visual information; therefore, this pattern does not fully apply to VSA.Considering the above question, we design a new algorithm to compute the orientation similarity. In this work, for an imageI, the preferred orientation of each pixel is calculated as its gradient direction θI:(8)θIi,j=arctanGvi,jGhi,j⋅180π,where Gh(i, j) and Gv(i, j) are the gradient magnitudes along the horizontal and vertical directions, respectively, which can be obtained from equation (2). And (i, j) is the index of the pixel in I. So, we can obtain the quantitative orientation information. We give an example in Figure 3, and we can find that the orientation information of the image has obvious changes.Then, we compute the orientation changeDO of the original image O and its encrypted image E by calculating their distance:(9)DOi,j=θOi,j−θEi,j,where |.| denotes the absolute operation, (i, j) is the index of the pixel, and θO and θE are the orientation maps of the original image O and its encrypted image E, respectively.Because the range ofθO and θE is [−180, 180], the range of DO is [0, 360]. Considering that HVS has a similar perception of relative orientation (such as 90o and 270o), we set the range of DO to [0, 180] by setting DODO>180=360−DODO>180.Then, the orientation similarity mapSO of the original and the encrypted images can be measured as(10)SOi,j=1+log2DOi,jlog21/180.


## 3.3. Saliency-Weighted Pooling

It is observed that different regions have drastically different effects on the visual understanding of an image. Most of the contribution to visual perception is provided by the information loss and distortion in important regions. An image importance map refers to the important regions that provide a greater contribution to the visual perception, and such maps have been studied extensively in recent years. So, we highlight these important regions and suppress the other regions with a salient map for visual content extraction. To this end, the salient value of each pixel is required. As illustrated in Figure3(f), visual saliency map highlights the important regions in an image, and the visual saliency map can extract the important areas of an image and then get a better VSA. In the past decades, a large number of saliency models [36–40] have been proposed and these models can help us complete a better VSA.SST(i, j) and SO(i, j), obtained by equations (7) and (10), respectively, are two feature similarity maps with the same size as the image. However, we need a VSA score to represent the visual security. Therefore, we need a pooling method to compress the two feature maps into two scores to represent the feature similarities. In our work, we take the simple and classic saliency-weighted pooling method. Considering that the security of a selective encrypted image depends on the degree of disclosure of its visual content, which is obtained by comparing it with the original image. So, we select the original image’s saliency map SMO(i, j) to combine with the structure similarity map SST(i, j) and orientation similarity map SO(i, j), respectively:(11)VSST=∑i,jSSTi,j⋅SMOi,j∑i,jSMOi,j,VSO=∑i,jSOi,j⋅SMOi,j∑i,jSMOi,j.Considering that different saliency models affect the performance and communicational cost of our proposed VSA, we calculate the performance and running time of different saliency models. To eliminate the possible bias due to specific image selection, we randomly choose 100 images from the IVC-SelectEncrypt database and then calculate the average running time as the computation cost of each saliency model. Table1 shows the results. From Table 1, we can find that the more appropriate saliency model is GBVS. Therefore, as a simple but powerful saliency model, graph-based visual saliency [36] (GBVS) is employed. A saliency map of an original image generated by GBVS can be seen in Figure 3(f).Table 1
Performance comparison employing different saliency models: graph-based visual saliency (GBVS), context-aware visual saliency (CAVS), saliency estimation using region covariance (CovSal), saliency through adaptive whitening of color and scale features (AWS), and Boolean map saliency (BMS).
MetricPLCCSRCCKRCCRMSETime(s)Without saliency0.9430.9350.7690.4350.356GBVS [36]0.9560.9540.8150.3880.476CAVS [37]0.9580.9560.8150.38621.43CovSal [38]0.9490.9430.8090.3951.256AWS [39]0.9530.9470.8170.3960.789BMS [40]0.9510.9480.8130.4016.579After performing the similarity measurements on the structure and the orientation features between the original and encrypted images, respectively, the generated structure similarity VSST and orientation similarity VSO are combined together to calculate the visual security:(12)VSA=α⋅VSST+β⋅VSO,where α and β are two parameters used to adjust the relative importance of VSST and VSO. The structure and orientation features of an image are important which are highly sensitive to the visual perception. For selective encrypted images, an important feature is that the skeleton of the image is still intelligible but the details are almost unintelligible. Therefore, structure obviously plays a more important role than orientation, and we explore the effect of the structure information and orientation information, respectively, in Table 2. So that the value of α should be greater than β. In our experiments, α and β are set to 0.8 and 0.2, respectively, because this setting was found to be optimal.Table 2
Performance of structure similarity VSST, orientation similarity VSO, and final VSA.
MetricPLCCSRCCKRCCRMSEVSST0.9510.9470.8330.398VSO0.9430.9390.8020.418VSA0.9580.9560.8150.386


## 4. Experiments

In this section, the performance of our proposed VSA is analyzed by comparing with other IQAs and VSAs. We evaluate the performance from confidence, monotonicity, linearity, and accuracy and provide comparisons with other IQAs and VSAs.


### 4.1. Experimental Protocol

#### 4.1.1. Test Database

To verify the performance of our proposed method, experiments are conducted on two common encrypted databases: IVC-SelectEncrypt [32] and PEID [33].The IVC-SelectEncrypt can be downloaded fromhttp://www.polytech.univ-nantes.fr/autrusseau-f/Databases/SelectiveEncryption/. The PEID is from https://sites.google.com/site/xiangtaooo/. Their detailed statistical information is summarized in Table 3.Table 3
Statistical information of the test databases.
DatabaseIVC-SelectEncryptPEIDReference image820Encrypted image2001080Encryption type510Subjective scoreMOSMOSScore range[1, 5][0, 6]Low quality[1, 2.5][0, 2]Moderate quality(2.5, 3.5](2, 4]High quality(3.5, 5](4, 6]The IVC-SelectEncrypt database consists of 8 original images, 200 encrypted images are generated from them using 5 different encrypted algorithms with 5 different encryption degrees. The range of its mean opinion scores (MOS) is [1, 5].The PEID database has 1080 encrypted images obtained from 20 original images by using 10 encryption schemes. It has two subjective scores: the visual quality score and visual security score. We use only the visual security score here because our task is visual security assessment, and the range of its mean opinion scores (MOS) is [0, 6].


#### 4.1.2. Evaluation Methodology

We evaluate the performance from confidence, monotonicity, linearity, and accuracy.Confidence is utilized to establish how well a VSA actually reflects the human judgment [17]. Given a subjective score x(x ε MOS) on a database D, and for this score x, each image I has a subjective score VI. We define Vmax(x) as the maximum of the objective scores of those images on D and define Vmin(x) as the minimum of the objective scores. Confidence Cx = | Vmax(x) − Vmin(x)| measures the difference between these two extrema. The normalized mean confidence µD, the normalized standard deviation σD, and the normalized maximum confidence maxD are the evaluation criteria which are generated based on Cx.To ascertain the correlation between the subject VSA scores and object scores MOS, we compute the Spearman rank correlation coefficient (SRCC), the Kendall rank correlation coefficient (KRCC), the Person linear correlation coefficient (PLCC), and the root mean-squared error (RMSE). SRCC and KRCC can evaluate performance monotonicity, PLCC can evaluate linearity, and RMSE can evaluate accuracy. Before the calculation of the correlation between the subject VSA scores and object scores MOS, a five-parameter logistic regression function is applied to reduce the nonlinearity of the subject VSA scores [33], which is defined as(13)S′=β112−1eβ2S−β3+β4S+β5,where S′ is the fitted VSA score, S is the objective VSA score, and βi (i = 1, 2, 3, 4, 5) denotes the parameters determined via curve fitting.A better VSA should have lowerµD, σD, maxD, and RMSE values but have higher SRCC, KRCC, and PLCC values.


### 4.2. Comparative Analysis

We compare our proposed VSA with other IQAs and VSAs by the evaluation criterions mentioned above from the following three aspects. And these IQAs and VSAs include the PSNR, SSIM [19], VIF [20], ESS [22], LSS [22], LFBVS [23], and VSI-Canny [21].


#### 4.2.1. Overall Evaluation

The results of the confidence evaluation of all IQAs and VSAs on the IVC-SelectEncrypt database are shown in Figure4. A better VSA should have lower and more stable Cx values. From Figure 4, we can find that the Cx values of VIF, VSI-Canny, and our VSA are more stable.Figure 4
Confidence evaluation of all the VSAs on the IVC-SelectEncrypt database, where the pairs of the MOS values and the objective scores are plotted as black dots. The nonlinear fitting results are plotted as red dashed lines, andVmax(x), Vmin(x), and Cx are plotted as blue, green, and black solid lines, respectively. (a) PSNR. (b) SSIM. (c) VIF. (d) LSS. (e) ESS. (f) LFBVS. (g) VSI-Canny. (h) The proposed VSA.
(a)(b)(c)(d)(e)(f)(g)(h)Table4 lists the overall performance of all IQAs and VSAs on the IVC-SelectEncrypt and PEID databases, and the best is marked in bold. Obviously, our proposed VSA performs best on IVC-SelectEncrypt. On the PEID database, VIF achieves the best monotonicity (the highest SRCC and KRCC) and the lowest RMSE, LSS has the lowest σD, ESS achieves the lowest maxD, and our proposed VSA achieves the best μD and PLCC. Although our proposed VSA is not the best in some values, it is very close to the best one. Compared with other methods, the extracted structure and orientation features of our proposed VSA are more consistent with HVS because HVS is very sensitive to structure and orientation changes caused by selective encryption. In addition, we also considered the visual saliency that was not considered by other IQAs and VSAs. Therefore, it is clear and reasonable that our proposed VSA exhibits the better overall performance.Table 4
Overall performance comparison on different databases.
DatabaseEvaluationPSNRSSIMVIFLSSESSLFBVSVSI-CannyProposedIVC-SelectEncryptµD0.1560.1220.1700.2220.3430.2540.1540.098σD0.1010.1840.1010.1880.1880.1740.0890.099maxD0.3940.6890.4040.5200.3990.4790.3530.343PLCC0.9150.8910.9390.9190.9010.8910.9420.956SRCC0.9110.8690.9390.9320.9090.8910.9350.954KRCC0.7430.7020.7910.7860.7470.7120.7760.815RMSE0.5430.6450.4550.6580.5060.6010.4460.388PEIDµD0.1890.4310.1690.0980.1580.1160.3330.096σD0.1440.2390.0770.0440.1370.1050.2100.059maxD0.5380.8530.5340.7710.4790.5300.8820.774PLCC0.8250.8450.85408230.8080.7240.8500.886SRCC0.7960.8310.9100.7960.7720.6180.7260.870KRCC0.6160.6570.7680.6160.5920.4580.5570.691RMSE1.0560.9990.6451.0611.0651.2480.9840.866


#### 4.2.2. Evaluation on Different Quality Ranges

The selective encrypted images usually have low and moderate visual quality [21, 26]. Therefore, to evaluate the performance of these VSAs more comprehensively, we should evaluate the performance of these VSAs on different image quality ranges (i.e., low, moderate, and high). The detailed division information can be found in Table 2. Considering that the selective encrypted images are typically in the low- or moderate-quality ranges, it is more important to evaluate the performance of VSAs in the low and moderate image quality ranges than in the high-quality ranges [21, 26].The comparison results of different VSAs in different image quality ranges on the two test databases are shown in Table5. We can find that our proposed VSA has better performance compared with the other VSAs in the low and moderate image quality ranges. In the low image quality range, on the IVC-SelectEncrypt database, VIF shows superior performance in maxD, LSS shows the best performance on PLCC, SRCC, KRCC, and RMSE, our proposed VSA achieves the best value on µD and σD, and other values are very close to the best one. On the PEID database, VIF shows superior performance in confidence evaluation (lowest μD, σD, and maxD), and our proposed VSA achieves the best performance on PLCC, SRCC, KRCC, and RMSE. In the moderate image quality range, our proposed VSA achieves the best performance on monotonicity, linearity, and accuracy evaluation on the two databases. In the high image quality range, SSIM obtains the best performance on IVC-SelectEncrypt database; on the PEID database, various VSAs exhibit satisfactory performance in different aspects. In summary, our proposed VSA exhibits better performance in low and moderate image quality ranges on the two databases. In the low and moderate image quality ranges, the structure and orientation changes caused by selective encryption are more obvious. And the saliency of the original image can extract the more important areas of the images which is important for the visual security assessment of the selective encrypted images. Therefore, it is rational that our proposed VSA shows the better performance in low and moderate image quality ranges.Table 5
Overall performance comparison on different ranges.
DatabaseEvaluationPSNRSSIMVIFLSSESSLFBVSVSI-CannyProposedIVC-SelectEncryptLowµD0.2160.3670.2830.3110.2180.1790.1830.155σD0.1120.1990.1370.2110.3070.2560.0990.092maxD0.3940.6890.2830.8760.3120.4790.3530.306PLCC0.6840.6800.9000.9350.7600.5860.9000.912SRCC0.7030.5280.8890.9020.6730.6620.8260.899KRCC0.5240.3890.7110.7460.5820.4790.6350.733RMSE0.3810.3840.2280.1850.5740.6200.2280.246ModerateµD0.1290.1680.2220.2900.1360.1410.1830.154σD0.1740.1250.1380.1160.2580.2730.0900.094maxD0.2550.3930.4040.4230.3760.3120.2770.306PLCC0.5140.4740.5830.5030.6020.3880.5030.713SRCC0.5310.3480.5210.4950.5770.3210.4730.629KRCC0.3670.2440.3670.3450.3240.2530.3240.422RMSE0.2860.2940.2710.2890.6020.6390.2880.234HighµD0.1300.0380.1860.1140.2200.1980.1190.164σD0.0760.0440.0930.1170.3310.2770.0680.080maxD0.3010.1780.3680.4240.3990.3090.2590.334PLCC0.6940.7400.5960.2530.5880.6460.6050.650SRCC0.6940.7320.6060.4940.5990.6460.6060.638KRCC0.5200.5580.4450.3600.4420.4660.4370.469RMSE0.3160.2950.3530.4250.5640.5490.3520.333PEIDLowµD0.1520.3000.0300.3890.1700.0490.1970.097σD0.1700.1590.0190.1520.0950.0390.1050.041maxD0.5270.7160.1420.7530.3760.2830.6430.225PLCC0.5930.5770.7930.6410.5730.3150.4320.804SRCC0.5740.6170.6980.5310.4840.2870.1960.711KRCC0.4050.4520.5320.3800.3550.2040.1350.533RMSE0.5340.5420.4490.5100.5730.6190.5990.403ModerateµD0.1870.5270.0910.4320.3980.1160.4410.290σD0.1450.1610.0540.1920.1950.0380.1610.127maxD0.4990.8500.3870.7710.3940.2760.7250.565PLCC0.3550.4070.6170.2950.3870.3760.4740.623SRCC0.3200.3680.6040.4180.2890.2570.4750.601KRCC0.2210.2560.4180.2880.1980.1700.3330.430RMSE0.5310.5190.5030.5430.5340.5390.5000.519HighµD0.2100.5110.1590.2950.2780.2860.3930.317σD0.1720.3210.1330.1800.1630.1720.2640.198maxD0.5380.8530.5340.6280.4790.5300.8820.774PLCC0.8500.8490.8380.5180.8400.6930.8600.480SRCC0.7950.7810.7850.6320.7660.6290.8200.459KRCC0.6080.5930.5840.4660.5680.4330.6390.321RMSE0.4730.3290.3120.5320.3100.4120.3170.498


#### 4.2.3. Evaluation on Different Encryption Types

We also evaluated the different VSAs on various types of encryption on the two test databases to more comprehensively evaluate the performance of all VSAs. There are 15 different encryption types in the test on the two databases. Tables6 and 7 report the performance results of all encryption types that appear in the test databases, respectively.Table 6
Overall performance comparison (SRCC and KRCC) on different encryptions.
SRCCKRCCDatabaseMetricPSNRSSIMVIFLSSESSLFBVSVSI-CannyProposedPSNRSSIMVIFLSSESSLFBVSVSI-CannyProposedIVC-SelectEncrypttrad0.9650.9680.9640.9480.9330.9600.9510.9680.8640.8640.8540.8200.7990.8520.8180.854trunc0.8880.8790.9530.9480.9130.9070.9530.9210.7310.7100.8030.8290.8010.7980.8260.764iwind_ec0.9090.9240.9410.8870.8590.8860.9270.9650.7380.7660.8000.7510.7240.7530.7790.811iwind_nec0.9100.9530.9510.9070.9460.9460.9340.9530.7700.8370.8290.8200.8270.8220.7980.837res0.9150.8860.9550.9550.9260.9160.9440.9600.7780.7170.8390.8420.7870.7650.8250.846PEIDenc010.9060.9270.8970.9510.5400.4520.8640.9370.7350.7740.7590.8070.4730.3860.7090.759enc020.9510.9110.9630.8350.8460.6270.8870.9380.8090.7600.7850.6500.6680.4530.7110.764enc030.9230.8980.9770.9440.9660.8620.9750.9780.7490.7410.8250.7950.8130.7240.8720.873enc040.9110.9160.9160.9080.8280.9070.7380.9620.7420.7500.7410.7360.6940.7130.5380.769enc050.9290.9360.9400.9240.7890.8210.9270.9460.7740.7900.7830.7670.6130.6890.7830.830enc060.9700.9650.9720.9680.9610.9140.9720.9760.8470.8330.8520.8450.8440.8250.8530.853enc070.9560.9770.9830.9580.9750.9590.9820.9710.8050.8690.8850.8180.8760.8590.8840.862enc080.0830.6660.1910.2000.3250.3600.7100.2460.0660.5280.1440.1450.2560.2590.5580.184enc090.2350.7600.5820.1870.3640.3390.1550.5210.1620.5820.4930.1260.2890.2760.0820.380enc100.7600.9430.9800.6310.8460.6780.6320.9300.5890.8070.8230.4730.7230.6490.4680.788Table 7
Overall performance comparison (PLCC and RMSE) on different encryptions.
PLCCRMSEDatabaseMetricPSNRSSIMVIFLSSESSLFBVSVSI-cannyProposedPSNRSSIMVIFLSSESSLFBVSVSI-cannyProposedIVC-SelectEncrypttrad0.9550.9690.9690.9550.9220.9570.9550.9680.2500.2080.2070.2500.2640.2480.2490.211trunc0.9160.9070.9800.9700.9460.9490.9740.9580.5100.5360.2460.3080.3160.2930.2870.365iwind_ec0.9570.9810.9830.9410.9160.9670.9820.9840.4340.2800.2640.4830.5120.4910.2740.258iwind_nec0.9310.9650.9600.9070.9240.9480.9500.9710.4350.3120.3340.5020.4930.3750.3700.312res0.9050.9310.9780.9550.9490.9390.9610.9610.4800.4120.2340.2110.5010.4870.1980.311PEIDenc010.9040.9570.9480.9520.7960.5830.9440.9460.5090.3460.3280.3630.7160.9030.3930.418enc020.9670.9500.9700.8880.8610.6360.8880.9390.4240.5160.4100.7600.7490.8560.7620.517enc030.9640.9560.9850.9700.9790.9170.9840.9850.4810.5330.3260.4420.3830.4120.3240.312enc040.9400.9280.9500.9390.8160.9110.8830.9670.5260.5730.4420.5310.4730.4590.7220.419enc050.9170.9470.9550.9210.8120.8530.9560.9560.5200.4200.4020.5080.5290.5130.3850.402enc060.9820.9790.9880.9820.9780.9260.9840.9890.3630.3910.3620.3580.3710.4020.3450.339enc070.9800.9820.9870.9680.9820.9640.9790.9870.3860.3660.3670.4830.3750.3960.3910.415enc080.1080.6100.2720.2730.3870.3640.7170.2290.4280.3410.7960.4140.3690.4830.2300.418enc090.4260.7600.7150.2960.6190.4750.6560.6981.4081.0111.1291.4871.2451.3781.1741.115enc100.8240.9660.9870.7500.9330.8090.8040.9691.0670.4890.4631.2460.5740.7961.1210.463From Table6, we can find that our proposed VSA has better monotonicity performance (the higher SRCC and KRCC values) than other IQAs and VSAs on the two databases. More specifically, our proposed VSA achieves the highest SRCC hit-count (8 times) and KRCC hit-count (7 times), and this value is higher than those of the other metrics. We can also find that our proposed VSA still has the highest PLCC hit-count (7 times) and RMSE hit-count (7 times) from Table 7.From Tables6 and 7, we can see that all of the involved VSAs obtain relatively inferior performance on encryption types enc08 and enc09 in the PEID database, and our proposed VSA is still relatively great on enc09 but relatively poor on enc08. As shown in Figure 5, the distortions caused by these two encryption methods are different from other methods that they make the images warping. Therefore, the features extracted from these encrypted images cannot match the features of the original images and these VSAs and IQAs have not good performance on the two encryption methods. This situation also results the overall performance of our method at PEID being worse than IVC-SelectEncrypt. Except for enc08 and enc09, other encryption methods cause obvious structure and orientation changes of images. Therefore, it is reasonable that our proposed method shows the better performance because the features of our proposed VSA are more relevant to the content leakage caused by most encryption methods.Figure 5
The encrypted images by enc08 and enc09 from the PEID (29) database. (a) Original image. (b) Encrypted image of (a) by enc08. (c) Encrypted image of (a) by enc09.
(a)(b)(c)


### 4.3. Computational Complexity

Finally, considering that the running time is important in many practical applications, we analyze the computational cost of all VSAs. In our test, we measure the computational cost of a VSA on 512 × 512 images. We perform experiments using the original code in the MATLAB R2016b on a 64-bit Windows 7 operating system at 16 GB memory and 3.20 GHz frequency of Intel processors. To avoid the possible bias caused by selecting the specific images, we randomly choose 100 images from the PEID database and then calculate the average running time as the computation cost of each VSA.It is known from Table8 that most of the metrics are fast to compute. By contrast, PSNR and SSIM are the fastest methods but they are mainly used for image quality assessment, and their performance is not excellent. VIF is also an IQA method which has a relatively good performance, but its running time is much higher than other methods because its computational model is more complex. Compared with other VSAs, our proposed VSA has a faster running speed. In implementation, our method takes up most of the time in feature extraction procedure. In the future, we will try to explore more efficient feature extraction techniques to reduce the computational cost of the proposed method.Table 8
Computational cost of all involved metrics.
MetricPSNRSSIMVIFLSSESSLFBVSVSI-CannyProposedTime(s)0.0340.1361.9830.1730.5180.7890.7780.476


## 4.1. Experimental Protocol

### 4.1.1. Test Database

To verify the performance of our proposed method, experiments are conducted on two common encrypted databases: IVC-SelectEncrypt [32] and PEID [33].The IVC-SelectEncrypt can be downloaded fromhttp://www.polytech.univ-nantes.fr/autrusseau-f/Databases/SelectiveEncryption/. The PEID is from https://sites.google.com/site/xiangtaooo/. Their detailed statistical information is summarized in Table 3.Table 3
Statistical information of the test databases.
DatabaseIVC-SelectEncryptPEIDReference image820Encrypted image2001080Encryption type510Subjective scoreMOSMOSScore range[1, 5][0, 6]Low quality[1, 2.5][0, 2]Moderate quality(2.5, 3.5](2, 4]High quality(3.5, 5](4, 6]The IVC-SelectEncrypt database consists of 8 original images, 200 encrypted images are generated from them using 5 different encrypted algorithms with 5 different encryption degrees. The range of its mean opinion scores (MOS) is [1, 5].The PEID database has 1080 encrypted images obtained from 20 original images by using 10 encryption schemes. It has two subjective scores: the visual quality score and visual security score. We use only the visual security score here because our task is visual security assessment, and the range of its mean opinion scores (MOS) is [0, 6].


### 4.1.2. Evaluation Methodology

We evaluate the performance from confidence, monotonicity, linearity, and accuracy.Confidence is utilized to establish how well a VSA actually reflects the human judgment [17]. Given a subjective score x(x ε MOS) on a database D, and for this score x, each image I has a subjective score VI. We define Vmax(x) as the maximum of the objective scores of those images on D and define Vmin(x) as the minimum of the objective scores. Confidence Cx = | Vmax(x) − Vmin(x)| measures the difference between these two extrema. The normalized mean confidence µD, the normalized standard deviation σD, and the normalized maximum confidence maxD are the evaluation criteria which are generated based on Cx.To ascertain the correlation between the subject VSA scores and object scores MOS, we compute the Spearman rank correlation coefficient (SRCC), the Kendall rank correlation coefficient (KRCC), the Person linear correlation coefficient (PLCC), and the root mean-squared error (RMSE). SRCC and KRCC can evaluate performance monotonicity, PLCC can evaluate linearity, and RMSE can evaluate accuracy. Before the calculation of the correlation between the subject VSA scores and object scores MOS, a five-parameter logistic regression function is applied to reduce the nonlinearity of the subject VSA scores [33], which is defined as(13)S′=β112−1eβ2S−β3+β4S+β5,where S′ is the fitted VSA score, S is the objective VSA score, and βi (i = 1, 2, 3, 4, 5) denotes the parameters determined via curve fitting.A better VSA should have lowerµD, σD, maxD, and RMSE values but have higher SRCC, KRCC, and PLCC values.


## 4.1.1. Test Database

To verify the performance of our proposed method, experiments are conducted on two common encrypted databases: IVC-SelectEncrypt [32] and PEID [33].The IVC-SelectEncrypt can be downloaded fromhttp://www.polytech.univ-nantes.fr/autrusseau-f/Databases/SelectiveEncryption/. The PEID is from https://sites.google.com/site/xiangtaooo/. Their detailed statistical information is summarized in Table 3.Table 3
Statistical information of the test databases.
DatabaseIVC-SelectEncryptPEIDReference image820Encrypted image2001080Encryption type510Subjective scoreMOSMOSScore range[1, 5][0, 6]Low quality[1, 2.5][0, 2]Moderate quality(2.5, 3.5](2, 4]High quality(3.5, 5](4, 6]The IVC-SelectEncrypt database consists of 8 original images, 200 encrypted images are generated from them using 5 different encrypted algorithms with 5 different encryption degrees. The range of its mean opinion scores (MOS) is [1, 5].The PEID database has 1080 encrypted images obtained from 20 original images by using 10 encryption schemes. It has two subjective scores: the visual quality score and visual security score. We use only the visual security score here because our task is visual security assessment, and the range of its mean opinion scores (MOS) is [0, 6].


## 4.1.2. Evaluation Methodology

We evaluate the performance from confidence, monotonicity, linearity, and accuracy.Confidence is utilized to establish how well a VSA actually reflects the human judgment [17]. Given a subjective score x(x ε MOS) on a database D, and for this score x, each image I has a subjective score VI. We define Vmax(x) as the maximum of the objective scores of those images on D and define Vmin(x) as the minimum of the objective scores. Confidence Cx = | Vmax(x) − Vmin(x)| measures the difference between these two extrema. The normalized mean confidence µD, the normalized standard deviation σD, and the normalized maximum confidence maxD are the evaluation criteria which are generated based on Cx.To ascertain the correlation between the subject VSA scores and object scores MOS, we compute the Spearman rank correlation coefficient (SRCC), the Kendall rank correlation coefficient (KRCC), the Person linear correlation coefficient (PLCC), and the root mean-squared error (RMSE). SRCC and KRCC can evaluate performance monotonicity, PLCC can evaluate linearity, and RMSE can evaluate accuracy. Before the calculation of the correlation between the subject VSA scores and object scores MOS, a five-parameter logistic regression function is applied to reduce the nonlinearity of the subject VSA scores [33], which is defined as(13)S′=β112−1eβ2S−β3+β4S+β5,where S′ is the fitted VSA score, S is the objective VSA score, and βi (i = 1, 2, 3, 4, 5) denotes the parameters determined via curve fitting.A better VSA should have lowerµD, σD, maxD, and RMSE values but have higher SRCC, KRCC, and PLCC values.


## 4.2. Comparative Analysis

We compare our proposed VSA with other IQAs and VSAs by the evaluation criterions mentioned above from the following three aspects. And these IQAs and VSAs include the PSNR, SSIM [19], VIF [20], ESS [22], LSS [22], LFBVS [23], and VSI-Canny [21].


### 4.2.1. Overall Evaluation

The results of the confidence evaluation of all IQAs and VSAs on the IVC-SelectEncrypt database are shown in Figure4. A better VSA should have lower and more stable Cx values. From Figure 4, we can find that the Cx values of VIF, VSI-Canny, and our VSA are more stable.Figure 4
Confidence evaluation of all the VSAs on the IVC-SelectEncrypt database, where the pairs of the MOS values and the objective scores are plotted as black dots. The nonlinear fitting results are plotted as red dashed lines, andVmax(x), Vmin(x), and Cx are plotted as blue, green, and black solid lines, respectively. (a) PSNR. (b) SSIM. (c) VIF. (d) LSS. (e) ESS. (f) LFBVS. (g) VSI-Canny. (h) The proposed VSA.
(a)(b)(c)(d)(e)(f)(g)(h)Table4 lists the overall performance of all IQAs and VSAs on the IVC-SelectEncrypt and PEID databases, and the best is marked in bold. Obviously, our proposed VSA performs best on IVC-SelectEncrypt. On the PEID database, VIF achieves the best monotonicity (the highest SRCC and KRCC) and the lowest RMSE, LSS has the lowest σD, ESS achieves the lowest maxD, and our proposed VSA achieves the best μD and PLCC. Although our proposed VSA is not the best in some values, it is very close to the best one. Compared with other methods, the extracted structure and orientation features of our proposed VSA are more consistent with HVS because HVS is very sensitive to structure and orientation changes caused by selective encryption. In addition, we also considered the visual saliency that was not considered by other IQAs and VSAs. Therefore, it is clear and reasonable that our proposed VSA exhibits the better overall performance.Table 4
Overall performance comparison on different databases.
DatabaseEvaluationPSNRSSIMVIFLSSESSLFBVSVSI-CannyProposedIVC-SelectEncryptµD0.1560.1220.1700.2220.3430.2540.1540.098σD0.1010.1840.1010.1880.1880.1740.0890.099maxD0.3940.6890.4040.5200.3990.4790.3530.343PLCC0.9150.8910.9390.9190.9010.8910.9420.956SRCC0.9110.8690.9390.9320.9090.8910.9350.954KRCC0.7430.7020.7910.7860.7470.7120.7760.815RMSE0.5430.6450.4550.6580.5060.6010.4460.388PEIDµD0.1890.4310.1690.0980.1580.1160.3330.096σD0.1440.2390.0770.0440.1370.1050.2100.059maxD0.5380.8530.5340.7710.4790.5300.8820.774PLCC0.8250.8450.85408230.8080.7240.8500.886SRCC0.7960.8310.9100.7960.7720.6180.7260.870KRCC0.6160.6570.7680.6160.5920.4580.5570.691RMSE1.0560.9990.6451.0611.0651.2480.9840.866


### 4.2.2. Evaluation on Different Quality Ranges

The selective encrypted images usually have low and moderate visual quality [21, 26]. Therefore, to evaluate the performance of these VSAs more comprehensively, we should evaluate the performance of these VSAs on different image quality ranges (i.e., low, moderate, and high). The detailed division information can be found in Table 2. Considering that the selective encrypted images are typically in the low- or moderate-quality ranges, it is more important to evaluate the performance of VSAs in the low and moderate image quality ranges than in the high-quality ranges [21, 26].The comparison results of different VSAs in different image quality ranges on the two test databases are shown in Table5. We can find that our proposed VSA has better performance compared with the other VSAs in the low and moderate image quality ranges. In the low image quality range, on the IVC-SelectEncrypt database, VIF shows superior performance in maxD, LSS shows the best performance on PLCC, SRCC, KRCC, and RMSE, our proposed VSA achieves the best value on µD and σD, and other values are very close to the best one. On the PEID database, VIF shows superior performance in confidence evaluation (lowest μD, σD, and maxD), and our proposed VSA achieves the best performance on PLCC, SRCC, KRCC, and RMSE. In the moderate image quality range, our proposed VSA achieves the best performance on monotonicity, linearity, and accuracy evaluation on the two databases. In the high image quality range, SSIM obtains the best performance on IVC-SelectEncrypt database; on the PEID database, various VSAs exhibit satisfactory performance in different aspects. In summary, our proposed VSA exhibits better performance in low and moderate image quality ranges on the two databases. In the low and moderate image quality ranges, the structure and orientation changes caused by selective encryption are more obvious. And the saliency of the original image can extract the more important areas of the images which is important for the visual security assessment of the selective encrypted images. Therefore, it is rational that our proposed VSA shows the better performance in low and moderate image quality ranges.Table 5
Overall performance comparison on different ranges.
DatabaseEvaluationPSNRSSIMVIFLSSESSLFBVSVSI-CannyProposedIVC-SelectEncryptLowµD0.2160.3670.2830.3110.2180.1790.1830.155σD0.1120.1990.1370.2110.3070.2560.0990.092maxD0.3940.6890.2830.8760.3120.4790.3530.306PLCC0.6840.6800.9000.9350.7600.5860.9000.912SRCC0.7030.5280.8890.9020.6730.6620.8260.899KRCC0.5240.3890.7110.7460.5820.4790.6350.733RMSE0.3810.3840.2280.1850.5740.6200.2280.246ModerateµD0.1290.1680.2220.2900.1360.1410.1830.154σD0.1740.1250.1380.1160.2580.2730.0900.094maxD0.2550.3930.4040.4230.3760.3120.2770.306PLCC0.5140.4740.5830.5030.6020.3880.5030.713SRCC0.5310.3480.5210.4950.5770.3210.4730.629KRCC0.3670.2440.3670.3450.3240.2530.3240.422RMSE0.2860.2940.2710.2890.6020.6390.2880.234HighµD0.1300.0380.1860.1140.2200.1980.1190.164σD0.0760.0440.0930.1170.3310.2770.0680.080maxD0.3010.1780.3680.4240.3990.3090.2590.334PLCC0.6940.7400.5960.2530.5880.6460.6050.650SRCC0.6940.7320.6060.4940.5990.6460.6060.638KRCC0.5200.5580.4450.3600.4420.4660.4370.469RMSE0.3160.2950.3530.4250.5640.5490.3520.333PEIDLowµD0.1520.3000.0300.3890.1700.0490.1970.097σD0.1700.1590.0190.1520.0950.0390.1050.041maxD0.5270.7160.1420.7530.3760.2830.6430.225PLCC0.5930.5770.7930.6410.5730.3150.4320.804SRCC0.5740.6170.6980.5310.4840.2870.1960.711KRCC0.4050.4520.5320.3800.3550.2040.1350.533RMSE0.5340.5420.4490.5100.5730.6190.5990.403ModerateµD0.1870.5270.0910.4320.3980.1160.4410.290σD0.1450.1610.0540.1920.1950.0380.1610.127maxD0.4990.8500.3870.7710.3940.2760.7250.565PLCC0.3550.4070.6170.2950.3870.3760.4740.623SRCC0.3200.3680.6040.4180.2890.2570.4750.601KRCC0.2210.2560.4180.2880.1980.1700.3330.430RMSE0.5310.5190.5030.5430.5340.5390.5000.519HighµD0.2100.5110.1590.2950.2780.2860.3930.317σD0.1720.3210.1330.1800.1630.1720.2640.198maxD0.5380.8530.5340.6280.4790.5300.8820.774PLCC0.8500.8490.8380.5180.8400.6930.8600.480SRCC0.7950.7810.7850.6320.7660.6290.8200.459KRCC0.6080.5930.5840.4660.5680.4330.6390.321RMSE0.4730.3290.3120.5320.3100.4120.3170.498


### 4.2.3. Evaluation on Different Encryption Types

We also evaluated the different VSAs on various types of encryption on the two test databases to more comprehensively evaluate the performance of all VSAs. There are 15 different encryption types in the test on the two databases. Tables6 and 7 report the performance results of all encryption types that appear in the test databases, respectively.Table 6
Overall performance comparison (SRCC and KRCC) on different encryptions.
SRCCKRCCDatabaseMetricPSNRSSIMVIFLSSESSLFBVSVSI-CannyProposedPSNRSSIMVIFLSSESSLFBVSVSI-CannyProposedIVC-SelectEncrypttrad0.9650.9680.9640.9480.9330.9600.9510.9680.8640.8640.8540.8200.7990.8520.8180.854trunc0.8880.8790.9530.9480.9130.9070.9530.9210.7310.7100.8030.8290.8010.7980.8260.764iwind_ec0.9090.9240.9410.8870.8590.8860.9270.9650.7380.7660.8000.7510.7240.7530.7790.811iwind_nec0.9100.9530.9510.9070.9460.9460.9340.9530.7700.8370.8290.8200.8270.8220.7980.837res0.9150.8860.9550.9550.9260.9160.9440.9600.7780.7170.8390.8420.7870.7650.8250.846PEIDenc010.9060.9270.8970.9510.5400.4520.8640.9370.7350.7740.7590.8070.4730.3860.7090.759enc020.9510.9110.9630.8350.8460.6270.8870.9380.8090.7600.7850.6500.6680.4530.7110.764enc030.9230.8980.9770.9440.9660.8620.9750.9780.7490.7410.8250.7950.8130.7240.8720.873enc040.9110.9160.9160.9080.8280.9070.7380.9620.7420.7500.7410.7360.6940.7130.5380.769enc050.9290.9360.9400.9240.7890.8210.9270.9460.7740.7900.7830.7670.6130.6890.7830.830enc060.9700.9650.9720.9680.9610.9140.9720.9760.8470.8330.8520.8450.8440.8250.8530.853enc070.9560.9770.9830.9580.9750.9590.9820.9710.8050.8690.8850.8180.8760.8590.8840.862enc080.0830.6660.1910.2000.3250.3600.7100.2460.0660.5280.1440.1450.2560.2590.5580.184enc090.2350.7600.5820.1870.3640.3390.1550.5210.1620.5820.4930.1260.2890.2760.0820.380enc100.7600.9430.9800.6310.8460.6780.6320.9300.5890.8070.8230.4730.7230.6490.4680.788Table 7
Overall performance comparison (PLCC and RMSE) on different encryptions.
PLCCRMSEDatabaseMetricPSNRSSIMVIFLSSESSLFBVSVSI-cannyProposedPSNRSSIMVIFLSSESSLFBVSVSI-cannyProposedIVC-SelectEncrypttrad0.9550.9690.9690.9550.9220.9570.9550.9680.2500.2080.2070.2500.2640.2480.2490.211trunc0.9160.9070.9800.9700.9460.9490.9740.9580.5100.5360.2460.3080.3160.2930.2870.365iwind_ec0.9570.9810.9830.9410.9160.9670.9820.9840.4340.2800.2640.4830.5120.4910.2740.258iwind_nec0.9310.9650.9600.9070.9240.9480.9500.9710.4350.3120.3340.5020.4930.3750.3700.312res0.9050.9310.9780.9550.9490.9390.9610.9610.4800.4120.2340.2110.5010.4870.1980.311PEIDenc010.9040.9570.9480.9520.7960.5830.9440.9460.5090.3460.3280.3630.7160.9030.3930.418enc020.9670.9500.9700.8880.8610.6360.8880.9390.4240.5160.4100.7600.7490.8560.7620.517enc030.9640.9560.9850.9700.9790.9170.9840.9850.4810.5330.3260.4420.3830.4120.3240.312enc040.9400.9280.9500.9390.8160.9110.8830.9670.5260.5730.4420.5310.4730.4590.7220.419enc050.9170.9470.9550.9210.8120.8530.9560.9560.5200.4200.4020.5080.5290.5130.3850.402enc060.9820.9790.9880.9820.9780.9260.9840.9890.3630.3910.3620.3580.3710.4020.3450.339enc070.9800.9820.9870.9680.9820.9640.9790.9870.3860.3660.3670.4830.3750.3960.3910.415enc080.1080.6100.2720.2730.3870.3640.7170.2290.4280.3410.7960.4140.3690.4830.2300.418enc090.4260.7600.7150.2960.6190.4750.6560.6981.4081.0111.1291.4871.2451.3781.1741.115enc100.8240.9660.9870.7500.9330.8090.8040.9691.0670.4890.4631.2460.5740.7961.1210.463From Table6, we can find that our proposed VSA has better monotonicity performance (the higher SRCC and KRCC values) than other IQAs and VSAs on the two databases. More specifically, our proposed VSA achieves the highest SRCC hit-count (8 times) and KRCC hit-count (7 times), and this value is higher than those of the other metrics. We can also find that our proposed VSA still has the highest PLCC hit-count (7 times) and RMSE hit-count (7 times) from Table 7.From Tables6 and 7, we can see that all of the involved VSAs obtain relatively inferior performance on encryption types enc08 and enc09 in the PEID database, and our proposed VSA is still relatively great on enc09 but relatively poor on enc08. As shown in Figure 5, the distortions caused by these two encryption methods are different from other methods that they make the images warping. Therefore, the features extracted from these encrypted images cannot match the features of the original images and these VSAs and IQAs have not good performance on the two encryption methods. This situation also results the overall performance of our method at PEID being worse than IVC-SelectEncrypt. Except for enc08 and enc09, other encryption methods cause obvious structure and orientation changes of images. Therefore, it is reasonable that our proposed method shows the better performance because the features of our proposed VSA are more relevant to the content leakage caused by most encryption methods.Figure 5
The encrypted images by enc08 and enc09 from the PEID (29) database. (a) Original image. (b) Encrypted image of (a) by enc08. (c) Encrypted image of (a) by enc09.
(a)(b)(c)


## 4.2.1. Overall Evaluation

The results of the confidence evaluation of all IQAs and VSAs on the IVC-SelectEncrypt database are shown in Figure4. A better VSA should have lower and more stable Cx values. From Figure 4, we can find that the Cx values of VIF, VSI-Canny, and our VSA are more stable.Figure 4
Confidence evaluation of all the VSAs on the IVC-SelectEncrypt database, where the pairs of the MOS values and the objective scores are plotted as black dots. The nonlinear fitting results are plotted as red dashed lines, andVmax(x), Vmin(x), and Cx are plotted as blue, green, and black solid lines, respectively. (a) PSNR. (b) SSIM. (c) VIF. (d) LSS. (e) ESS. (f) LFBVS. (g) VSI-Canny. (h) The proposed VSA.
(a)(b)(c)(d)(e)(f)(g)(h)Table4 lists the overall performance of all IQAs and VSAs on the IVC-SelectEncrypt and PEID databases, and the best is marked in bold. Obviously, our proposed VSA performs best on IVC-SelectEncrypt. On the PEID database, VIF achieves the best monotonicity (the highest SRCC and KRCC) and the lowest RMSE, LSS has the lowest σD, ESS achieves the lowest maxD, and our proposed VSA achieves the best μD and PLCC. Although our proposed VSA is not the best in some values, it is very close to the best one. Compared with other methods, the extracted structure and orientation features of our proposed VSA are more consistent with HVS because HVS is very sensitive to structure and orientation changes caused by selective encryption. In addition, we also considered the visual saliency that was not considered by other IQAs and VSAs. Therefore, it is clear and reasonable that our proposed VSA exhibits the better overall performance.Table 4
Overall performance comparison on different databases.
DatabaseEvaluationPSNRSSIMVIFLSSESSLFBVSVSI-CannyProposedIVC-SelectEncryptµD0.1560.1220.1700.2220.3430.2540.1540.098σD0.1010.1840.1010.1880.1880.1740.0890.099maxD0.3940.6890.4040.5200.3990.4790.3530.343PLCC0.9150.8910.9390.9190.9010.8910.9420.956SRCC0.9110.8690.9390.9320.9090.8910.9350.954KRCC0.7430.7020.7910.7860.7470.7120.7760.815RMSE0.5430.6450.4550.6580.5060.6010.4460.388PEIDµD0.1890.4310.1690.0980.1580.1160.3330.096σD0.1440.2390.0770.0440.1370.1050.2100.059maxD0.5380.8530.5340.7710.4790.5300.8820.774PLCC0.8250.8450.85408230.8080.7240.8500.886SRCC0.7960.8310.9100.7960.7720.6180.7260.870KRCC0.6160.6570.7680.6160.5920.4580.5570.691RMSE1.0560.9990.6451.0611.0651.2480.9840.866


## 4.2.2. Evaluation on Different Quality Ranges

The selective encrypted images usually have low and moderate visual quality [21, 26]. Therefore, to evaluate the performance of these VSAs more comprehensively, we should evaluate the performance of these VSAs on different image quality ranges (i.e., low, moderate, and high). The detailed division information can be found in Table 2. Considering that the selective encrypted images are typically in the low- or moderate-quality ranges, it is more important to evaluate the performance of VSAs in the low and moderate image quality ranges than in the high-quality ranges [21, 26].The comparison results of different VSAs in different image quality ranges on the two test databases are shown in Table5. We can find that our proposed VSA has better performance compared with the other VSAs in the low and moderate image quality ranges. In the low image quality range, on the IVC-SelectEncrypt database, VIF shows superior performance in maxD, LSS shows the best performance on PLCC, SRCC, KRCC, and RMSE, our proposed VSA achieves the best value on µD and σD, and other values are very close to the best one. On the PEID database, VIF shows superior performance in confidence evaluation (lowest μD, σD, and maxD), and our proposed VSA achieves the best performance on PLCC, SRCC, KRCC, and RMSE. In the moderate image quality range, our proposed VSA achieves the best performance on monotonicity, linearity, and accuracy evaluation on the two databases. In the high image quality range, SSIM obtains the best performance on IVC-SelectEncrypt database; on the PEID database, various VSAs exhibit satisfactory performance in different aspects. In summary, our proposed VSA exhibits better performance in low and moderate image quality ranges on the two databases. In the low and moderate image quality ranges, the structure and orientation changes caused by selective encryption are more obvious. And the saliency of the original image can extract the more important areas of the images which is important for the visual security assessment of the selective encrypted images. Therefore, it is rational that our proposed VSA shows the better performance in low and moderate image quality ranges.Table 5
Overall performance comparison on different ranges.
DatabaseEvaluationPSNRSSIMVIFLSSESSLFBVSVSI-CannyProposedIVC-SelectEncryptLowµD0.2160.3670.2830.3110.2180.1790.1830.155σD0.1120.1990.1370.2110.3070.2560.0990.092maxD0.3940.6890.2830.8760.3120.4790.3530.306PLCC0.6840.6800.9000.9350.7600.5860.9000.912SRCC0.7030.5280.8890.9020.6730.6620.8260.899KRCC0.5240.3890.7110.7460.5820.4790.6350.733RMSE0.3810.3840.2280.1850.5740.6200.2280.246ModerateµD0.1290.1680.2220.2900.1360.1410.1830.154σD0.1740.1250.1380.1160.2580.2730.0900.094maxD0.2550.3930.4040.4230.3760.3120.2770.306PLCC0.5140.4740.5830.5030.6020.3880.5030.713SRCC0.5310.3480.5210.4950.5770.3210.4730.629KRCC0.3670.2440.3670.3450.3240.2530.3240.422RMSE0.2860.2940.2710.2890.6020.6390.2880.234HighµD0.1300.0380.1860.1140.2200.1980.1190.164σD0.0760.0440.0930.1170.3310.2770.0680.080maxD0.3010.1780.3680.4240.3990.3090.2590.334PLCC0.6940.7400.5960.2530.5880.6460.6050.650SRCC0.6940.7320.6060.4940.5990.6460.6060.638KRCC0.5200.5580.4450.3600.4420.4660.4370.469RMSE0.3160.2950.3530.4250.5640.5490.3520.333PEIDLowµD0.1520.3000.0300.3890.1700.0490.1970.097σD0.1700.1590.0190.1520.0950.0390.1050.041maxD0.5270.7160.1420.7530.3760.2830.6430.225PLCC0.5930.5770.7930.6410.5730.3150.4320.804SRCC0.5740.6170.6980.5310.4840.2870.1960.711KRCC0.4050.4520.5320.3800.3550.2040.1350.533RMSE0.5340.5420.4490.5100.5730.6190.5990.403ModerateµD0.1870.5270.0910.4320.3980.1160.4410.290σD0.1450.1610.0540.1920.1950.0380.1610.127maxD0.4990.8500.3870.7710.3940.2760.7250.565PLCC0.3550.4070.6170.2950.3870.3760.4740.623SRCC0.3200.3680.6040.4180.2890.2570.4750.601KRCC0.2210.2560.4180.2880.1980.1700.3330.430RMSE0.5310.5190.5030.5430.5340.5390.5000.519HighµD0.2100.5110.1590.2950.2780.2860.3930.317σD0.1720.3210.1330.1800.1630.1720.2640.198maxD0.5380.8530.5340.6280.4790.5300.8820.774PLCC0.8500.8490.8380.5180.8400.6930.8600.480SRCC0.7950.7810.7850.6320.7660.6290.8200.459KRCC0.6080.5930.5840.4660.5680.4330.6390.321RMSE0.4730.3290.3120.5320.3100.4120.3170.498


## 4.2.3. Evaluation on Different Encryption Types

We also evaluated the different VSAs on various types of encryption on the two test databases to more comprehensively evaluate the performance of all VSAs. There are 15 different encryption types in the test on the two databases. Tables6 and 7 report the performance results of all encryption types that appear in the test databases, respectively.Table 6
Overall performance comparison (SRCC and KRCC) on different encryptions.
SRCCKRCCDatabaseMetricPSNRSSIMVIFLSSESSLFBVSVSI-CannyProposedPSNRSSIMVIFLSSESSLFBVSVSI-CannyProposedIVC-SelectEncrypttrad0.9650.9680.9640.9480.9330.9600.9510.9680.8640.8640.8540.8200.7990.8520.8180.854trunc0.8880.8790.9530.9480.9130.9070.9530.9210.7310.7100.8030.8290.8010.7980.8260.764iwind_ec0.9090.9240.9410.8870.8590.8860.9270.9650.7380.7660.8000.7510.7240.7530.7790.811iwind_nec0.9100.9530.9510.9070.9460.9460.9340.9530.7700.8370.8290.8200.8270.8220.7980.837res0.9150.8860.9550.9550.9260.9160.9440.9600.7780.7170.8390.8420.7870.7650.8250.846PEIDenc010.9060.9270.8970.9510.5400.4520.8640.9370.7350.7740.7590.8070.4730.3860.7090.759enc020.9510.9110.9630.8350.8460.6270.8870.9380.8090.7600.7850.6500.6680.4530.7110.764enc030.9230.8980.9770.9440.9660.8620.9750.9780.7490.7410.8250.7950.8130.7240.8720.873enc040.9110.9160.9160.9080.8280.9070.7380.9620.7420.7500.7410.7360.6940.7130.5380.769enc050.9290.9360.9400.9240.7890.8210.9270.9460.7740.7900.7830.7670.6130.6890.7830.830enc060.9700.9650.9720.9680.9610.9140.9720.9760.8470.8330.8520.8450.8440.8250.8530.853enc070.9560.9770.9830.9580.9750.9590.9820.9710.8050.8690.8850.8180.8760.8590.8840.862enc080.0830.6660.1910.2000.3250.3600.7100.2460.0660.5280.1440.1450.2560.2590.5580.184enc090.2350.7600.5820.1870.3640.3390.1550.5210.1620.5820.4930.1260.2890.2760.0820.380enc100.7600.9430.9800.6310.8460.6780.6320.9300.5890.8070.8230.4730.7230.6490.4680.788Table 7
Overall performance comparison (PLCC and RMSE) on different encryptions.
PLCCRMSEDatabaseMetricPSNRSSIMVIFLSSESSLFBVSVSI-cannyProposedPSNRSSIMVIFLSSESSLFBVSVSI-cannyProposedIVC-SelectEncrypttrad0.9550.9690.9690.9550.9220.9570.9550.9680.2500.2080.2070.2500.2640.2480.2490.211trunc0.9160.9070.9800.9700.9460.9490.9740.9580.5100.5360.2460.3080.3160.2930.2870.365iwind_ec0.9570.9810.9830.9410.9160.9670.9820.9840.4340.2800.2640.4830.5120.4910.2740.258iwind_nec0.9310.9650.9600.9070.9240.9480.9500.9710.4350.3120.3340.5020.4930.3750.3700.312res0.9050.9310.9780.9550.9490.9390.9610.9610.4800.4120.2340.2110.5010.4870.1980.311PEIDenc010.9040.9570.9480.9520.7960.5830.9440.9460.5090.3460.3280.3630.7160.9030.3930.418enc020.9670.9500.9700.8880.8610.6360.8880.9390.4240.5160.4100.7600.7490.8560.7620.517enc030.9640.9560.9850.9700.9790.9170.9840.9850.4810.5330.3260.4420.3830.4120.3240.312enc040.9400.9280.9500.9390.8160.9110.8830.9670.5260.5730.4420.5310.4730.4590.7220.419enc050.9170.9470.9550.9210.8120.8530.9560.9560.5200.4200.4020.5080.5290.5130.3850.402enc060.9820.9790.9880.9820.9780.9260.9840.9890.3630.3910.3620.3580.3710.4020.3450.339enc070.9800.9820.9870.9680.9820.9640.9790.9870.3860.3660.3670.4830.3750.3960.3910.415enc080.1080.6100.2720.2730.3870.3640.7170.2290.4280.3410.7960.4140.3690.4830.2300.418enc090.4260.7600.7150.2960.6190.4750.6560.6981.4081.0111.1291.4871.2451.3781.1741.115enc100.8240.9660.9870.7500.9330.8090.8040.9691.0670.4890.4631.2460.5740.7961.1210.463From Table6, we can find that our proposed VSA has better monotonicity performance (the higher SRCC and KRCC values) than other IQAs and VSAs on the two databases. More specifically, our proposed VSA achieves the highest SRCC hit-count (8 times) and KRCC hit-count (7 times), and this value is higher than those of the other metrics. We can also find that our proposed VSA still has the highest PLCC hit-count (7 times) and RMSE hit-count (7 times) from Table 7.From Tables6 and 7, we can see that all of the involved VSAs obtain relatively inferior performance on encryption types enc08 and enc09 in the PEID database, and our proposed VSA is still relatively great on enc09 but relatively poor on enc08. As shown in Figure 5, the distortions caused by these two encryption methods are different from other methods that they make the images warping. Therefore, the features extracted from these encrypted images cannot match the features of the original images and these VSAs and IQAs have not good performance on the two encryption methods. This situation also results the overall performance of our method at PEID being worse than IVC-SelectEncrypt. Except for enc08 and enc09, other encryption methods cause obvious structure and orientation changes of images. Therefore, it is reasonable that our proposed method shows the better performance because the features of our proposed VSA are more relevant to the content leakage caused by most encryption methods.Figure 5
The encrypted images by enc08 and enc09 from the PEID (29) database. (a) Original image. (b) Encrypted image of (a) by enc08. (c) Encrypted image of (a) by enc09.
(a)(b)(c)


## 4.3. Computational Complexity

Finally, considering that the running time is important in many practical applications, we analyze the computational cost of all VSAs. In our test, we measure the computational cost of a VSA on 512 × 512 images. We perform experiments using the original code in the MATLAB R2016b on a 64-bit Windows 7 operating system at 16 GB memory and 3.20 GHz frequency of Intel processors. To avoid the possible bias caused by selecting the specific images, we randomly choose 100 images from the PEID database and then calculate the average running time as the computation cost of each VSA.It is known from Table8 that most of the metrics are fast to compute. By contrast, PSNR and SSIM are the fastest methods but they are mainly used for image quality assessment, and their performance is not excellent. VIF is also an IQA method which has a relatively good performance, but its running time is much higher than other methods because its computational model is more complex. Compared with other VSAs, our proposed VSA has a faster running speed. In implementation, our method takes up most of the time in feature extraction procedure. In the future, we will try to explore more efficient feature extraction techniques to reduce the computational cost of the proposed method.Table 8
Computational cost of all involved metrics.
MetricPSNRSSIMVIFLSSESSLFBVSVSI-CannyProposedTime(s)0.0340.1361.9830.1730.5180.7890.7780.476


## 5. Conclusions

In this paper, we have presented a novel visual security assessment (VSA) that makes use of the structure and orientation information. First, we extract the structure of the original and the encrypted images by combining PC and GM. Then, we extract the orientation information by the GM, and we can obtain similarity measurements by calculating the structure and orientation similarity maps. Meanwhile, we compute the saliency map of original image. Then, we utilize a saliency-based polling strategy to combine these two similarity maps and generate the final VSA score. We conduct extensive experiments to evaluate the performance of our proposed VSA and compare it with other IQAs and VSAs which are widely used for the visual security assessment for encrypted images on two encryption image databases. The experimental results show that our proposed VSA has better performance and stronger robustness than all existing IQAs and VSAs, especially in the range of low and moderate image quality.



---

*Source: 6675354-2021-01-23.xml*