# Local Privacy Protection for Sensitive Areas in Multiface Images

Authors: Chao Liu; Jing Yang; Xuan Zhang; Yining Zhang; Weinan Zhao; Fengjuan Miao; Yukun Shao
Journal: Computational Intelligence and Neuroscience
 (2022)

---

## Abstract

The privacy protection for face images aims to prevent attackers from accurately identifying target persons through face recognition. Inspired by goal-driven reasoning (reverse reasoning), this paper designs a goal-driven algorithm of local privacy protection for sensitive areas in multiface images (face areas) under the interactive framework of face recognition algorithm, regional growth, and differential privacy. The designed algorithm, named privacy protection for sensitive areas (PPSA), is realized in the following manner: Firstly, the multitask cascaded convolutional network (MTCNN) was adopted to recognize the region and landmark of each face. If the landmark overlaps a subgraph divided from the original image, the subgraph will be taken as the seed for regional growth in the face area, following the growth criterion of the fusion similarity measurement mechanism (FSMM). Different from single-face privacy protection, multiface privacy protection needs to deal with an unknown number of faces. Thus, the allocation of the privacy budgetε directly affects the operation effect of the PPSA algorithm. In our scheme, the total privacy budget ε is divided into two parts: ε_1 and ε_2. The former is evenly allocated to each seed, according to the estimated number of faces ρ contained in the image, while the latter is allocated to the other areas that may consume the privacy budget through dichotomization. Unlike the Laplacian (LAP) algorithm, the noise error of the PPSA algorithm will not change with the image size, for the privacy protection is limited to the face area. The results show that the PPSA algorithm meets the requirements ε-Differential privacy, and image classification is realized by using different image privacy protection algorithms in different human face databases. The verification results show that the accuracy of the PPSA algorithm is improved by at least 16.1%, the recall rate is improved by at least 2.3%, and F1-score is improved by at least 15.2%.


---

## Body

## 1. Introduction

In the information age, the importance of data becomes more and more prominent. The value of data is highlighted in various fields in our society. Data-based services are widely applied in different industries. In particular, the application of image data has been developing rapidly. Thanks to the fast development of information technology and multimedia technology, it is now easier to acquire and share digital face images. The users can publish their own photos on social network platforms and other channels. Statistics show that more than 3.2 billion face images are shared by users of major social network platforms around the world. These digital images usually contain a wealth of personally sensitive information. If the information is collected and analyzed by attackers, unmeasurable losses will occur, in addition to the leak of personal privacy.The privacy protection of image data often relies on techniques likek-anonymity, access control, and privacy encryption. Fung et al. [1] and Xiao and Tao [2] proposed the k-same method based on the anonymization mechanism. The method anonymizes each published gray image, reducing the probability of attackers deriving user identity from the published image to less than 1/k. Li et al. [3] applied access control to restrict user access to social network images. The transfer and visitor volumes of these images are reduced to protect privacy. This approach, which protects images via access settings, is not a fundamental privacy protection method for images. If an attacker has a certain background knowledge, he/she may bypass the access control and acquire the user images and privacy information. To prevent the communication process from being eavesdropped, Terrovitis et al. [4] implemented homomorphic encryption on gray images. However, every data encryption technique makes some assumptions and designs the corresponding encryption algorithm based on these assumptions. However new it is, the encryption algorithm will soon be breached by attackers. Focusing on anonymized images on Facebook, Sweeney, and Anonymity [5] revealed that attackers can deduce the social security number (SSN) of the person in each anonymized image, based on the unique Friendster feature of Facebook, and identify the sensitive information (e.g., disease and address) from the SSN.In 2006, Dwork [6] proposed differential probability, which disturbs sensitive data by adding noise to the output. Differential privacy can hide the impact of a single record: whether a record is in the dataset or not, the output probability of the same result will not change significantly. In this way, attackers are hampered from further reasoning. Compared with other privacy protection techniques, the differential probability is superior in that it makes no assumption about the background knowledge of attackers. Dwork further investigated differential privacy in a series of theses [7–11] and proposed its implementation mechanism [12, 13]. McSherry [14] pointed out that the differential privacy algorithm for complex privacy problems needs to satisfy two composition properties: sequence combination and parallel combination. In recent years, differential privacy has been mainly applied to the field of data publication. The differential privacy protection for data publication mainly attempts to ensure the accuracy of published data or query results while satisfying differential privacy. The relevant research focuses on adjusting the publication mechanism and algorithm complexity. The primary research method is a quantitative analysis based on calculation and learning theories.By the realization environment, the differential privacy protection for data publication can be divided into interactive data publication and noninteractive data publication [12]. On interactive data publication, Roth and Roughgarden [15] presented a median algorithm that can respond to more queries. Hardt and Rothblum [16] developed a pulse wave modulation (PMW) mechanism capable of increasing the number of queries. Gupta et al. [17] proposed a universal iterative dataset generation mechanism. Fan and Xiong [18] designed a novel approach with filtering and adaptive sampling for releasing time series under differential privacy (FAST). Kellaris et al. [19] put forward a flow data publication algorithm with no limit on the number of publications. On noninteractive data publication, Xiao et al. [20] came up with the Privelet algorithm. Xu et al. [21] created the noise first and structure first algorithms. Li et al. [22] proposed the matrix mechanism. Li et al. [23] put forward a data- and workload-aware algorithm (DAWA) algorithm. Owing to the complexity of image data, the current research is still in the exploratory stage concerning the differential privacy protection for sensitive information in images.Images are often represented as matrices in the real number field. Any pixel in an image can be mapped to a value at the corresponding location in a two-dimensional (2D) matrix. To satisfyε-differential privacy, the most direct approach is to add a Laplace noise to all the values in the matrix. However, the disturbed image may be over distorted and become useless. Fourier transform and wavelet transform are common ways to compress images. Zhang et al. [24] proposed an image compression method based on discrete Fourier transform (DFT), which adds a unique Laplace noise to each target image. Despite suppressing the noise error, their approach introduces the reconstruction error to image compression. Considering the uncorrelation between the values in the image matrix, Liu et al. [25] converted the image gray matrix into a one-dimensional (1D) ordered data flow and modeled the data flow with the sliding window model. To protect the privacy of images, the privacy budget was allocated dynamically based on the similarity between the data of adjacent sliding windows. This highly available strategy is restricted to the 1D space. Liu et al. [26] utilized regional growth to expand the comparison between adjacent subgraphs to the 2D space and further optimized the privacy protection for face images.The noises of the above methods cover the entire image. For face images, the sensitive information of the face only concentrates in the face area and even in some specific areas. The exposure of other nonsensitive areas will not lead to the leak of privacy. Therefore, this paper proposes a local privacy protection method for the sensitive areas in multiface images.


## 2. Background

### 2.1. Differential Privacy

Definition 1.
Adjacent datasets of the face image
For a given imageX, the gray matrix Xmn can be obtained by normalizing the image. Then, there exists X|Xmn=x11,x12,⋯x1n⋮⋱⋮xm1,xm1,⋯xmn, where xij in matrix Xmn represents the gray of the corresponding element. If there exists an X′ with only one element difference from X, X−X′=xij1≤i≤m,1≤j≤n, then X and X′ are adjacent datasets.Definition 2.
Differential privacy
For a given random algorithmM of image data publication, with the output range of RangeM, the algorithm can provide ε-differential privacy, if its arbitrary outputs on two adjacent gray images X and X′ satisfy the following:(1)PrMX∈S≤expε×PrMX′∈S,where ε is typically a small positive number that balances privacy with accuracy. If ε is small, the privacy is high and accuracy is low. The inverse is also true. Normally, ε is selected by the user by executing a privacy policy. When the adjacent datasets vary by only one record, the algorithm satisfies ε-differential privacy. When the adjacent datasets vary by k records, the algorithm satisfies kε-differential privacy.Definition 3.
Global sensitivity
LetQ be a random query function meeting Q:D⟶Rn. Then, the global sensitivity of Q can be expressed as follows:(2)ΔQGS=maxX,X′QD−QD′ρ.Theorem 1.
Laplace mechanism
LetQ be a query series of the length d. The random algorithm M receives database D and outputs the following vector that satisfies ε-differential privacy:(3)MD=QD+<Lap1ΔQLSε,…,LapdΔQLSε.As the most common noise addition mechanism, the Laplace mechanism disturbs the real output by adding the noise generated by Laplace distribution, thereby achieving differential privacy. The probability density function (PDF) of its noise distribution satisfiesfx|μ,b=1/2bex−μ/b.Property 1.
Differential privacy-serial combination property
For a given datasetX and a set of differential privacy algorithms M1X,M2X,…,MmX related to X, if the algorithm MiD satisfies εi- differential privacy, and the random processes of any two algorithms are independent of each other, then the algorithm combined from these algorithms satisfies ∑i=1mεi- differential privacy.


### 2.2. Multitask Cascaded Convolutional Network (MTCNN)

With the development of CNN [27], Sun et al. [28] suggested that the CNN can be applied to localization face landmarks by virtue of its strong feature extraction ability. A three-layer deep CNN (DCNN) was designed to solve a thorny problem in landmark extraction: the inability to obtain the global optimal solution, due to the improper setting of initial values. Zhang et al. introduced multitask learning to face landmark localization and proposed a task-driven DCNN for face landmark localization (TCDCN). The TCDCN, a multitask learning model of four subtasks, has a smaller time complexity than traditional CNN [29]. Two years later, Zhang et al. developed the MTCNN, which effectively integrates face area detection with face landmark localization. The three cascading neural networks in MTCNN are responsible for face classification, bounding box regression, and key point localization, respectively [30]. Wu et al. [31] presented a tweaked CNN (TCNN), which relies on a mixed Gaussian model to cluster the features on different layers, and concluded that a deeper network layer can more accurately mirror face landmarks. In addition, many other methods have been adopted to locate face landmarks, namely, principal component analysis (PCA) [32], support vector machine [33], Bayesian probabilistic network (BPN) [34], dynamic link architecture (DLA) [35], and Gabor wavelet network (GWN) [36].The MTCNN is a multitask parallel face recognition method based on deep learning. This method has been widely recognized in the industry because it operates rapidly, and its accuracy meets the general requirements of face detection. The multitask parallel capacity of the MTCNN algorithm manifests as the simultaneous detection of multiple faces (detecting whether an image contains faces and finding the face locations in the image, as shown in Figure1) and the localization of face landmarks (locating the five landmarks, namely, two eyes, nose tip, and mouth corners, as shown in Figure 2).Figure 1
Face detection.Figure 2
Landmark localization.The MTCNN algorithm can be realized as a three-stage cascaded CNN. The first stage is called a proposal network (P-Net), which mainly obtains the candidate windows for the face area and the regression vector of the bounding box, carries out regression with the bounding box, calibrates the candidate windows, and merges highly overlapped candidate boxes through nonmaximum suppression (NMS). The structure of the P-Net is shown in Figure3.Figure 3
Structure of P-net.The second stage is called a refine network (R-Net), which eliminates the false positive areas through bounding box regression and NMS. That is, lots of nonface windows are denied by a more complex CNN in order to refine the face window. The structure of the R-Net is shown in Figure4.Figure 4
Structure of R-net.The third stage is called an output network (O-Net). With one more convolutional layer than the R-Net, the O-Net outputs more refined results. The functions of the O-Net are the same as those of the R-Net. But the O-Net carries out more supervision of the face area and output five landmarks. The structure of the O-Net is shown in Figure5.Figure 5
Structure of O-net.Each stage of the MTCNN is a multitask network, responsible for handling tasks like face/nonface judgement, face box regression, and landmark localization. The face/nonface judgement adopts the cross-entropy loss function:(4)Lidet=−yidetlogpi+1−yidet1−logpi,where pi is the probability for the image to contain faces; yidet∈0,1 is the true background label.The regression loss of face box can be calculated by Euclidean distance:(5)Libox=y^ibox−yibox22,where y^ is the background coordinates predicted by the network; y is the true background coordinates; yibox∈ℝ4ℝ4 is a 4-tuple composed of x of the upper left corner, y of the upper left corner, length, and width).Landmark localization is calculated similarly as face box regression. The Euclidean distance between the predicted landmark location and the true landmark location is calculated and minimized:(6)Lilandmark=y^ilandmark−yilandmark22,where yilandmark∈ℝ10 is a 10-tuple, for five landmarks are being localized.The simplest function of the training process can be expressed as follows:(7)min∑i=1N∑jεdet,box,landmarkαjβijLij,where N is the number of training samples; αj is the importance of a task; βij∈0,1 is a sample label. In the P-Net and R-Net, αdet = 1, αbox = 0.5, and αlandmark = 0.5. In the O-Net, αdet = 1, αbox = 0.5, and αlandmark = 1.


## 2.1. Differential Privacy

Definition 1.
Adjacent datasets of the face image
For a given imageX, the gray matrix Xmn can be obtained by normalizing the image. Then, there exists X|Xmn=x11,x12,⋯x1n⋮⋱⋮xm1,xm1,⋯xmn, where xij in matrix Xmn represents the gray of the corresponding element. If there exists an X′ with only one element difference from X, X−X′=xij1≤i≤m,1≤j≤n, then X and X′ are adjacent datasets.Definition 2.
Differential privacy
For a given random algorithmM of image data publication, with the output range of RangeM, the algorithm can provide ε-differential privacy, if its arbitrary outputs on two adjacent gray images X and X′ satisfy the following:(1)PrMX∈S≤expε×PrMX′∈S,where ε is typically a small positive number that balances privacy with accuracy. If ε is small, the privacy is high and accuracy is low. The inverse is also true. Normally, ε is selected by the user by executing a privacy policy. When the adjacent datasets vary by only one record, the algorithm satisfies ε-differential privacy. When the adjacent datasets vary by k records, the algorithm satisfies kε-differential privacy.Definition 3.
Global sensitivity
LetQ be a random query function meeting Q:D⟶Rn. Then, the global sensitivity of Q can be expressed as follows:(2)ΔQGS=maxX,X′QD−QD′ρ.Theorem 1.
Laplace mechanism
LetQ be a query series of the length d. The random algorithm M receives database D and outputs the following vector that satisfies ε-differential privacy:(3)MD=QD+<Lap1ΔQLSε,…,LapdΔQLSε.As the most common noise addition mechanism, the Laplace mechanism disturbs the real output by adding the noise generated by Laplace distribution, thereby achieving differential privacy. The probability density function (PDF) of its noise distribution satisfiesfx|μ,b=1/2bex−μ/b.Property 1.
Differential privacy-serial combination property
For a given datasetX and a set of differential privacy algorithms M1X,M2X,…,MmX related to X, if the algorithm MiD satisfies εi- differential privacy, and the random processes of any two algorithms are independent of each other, then the algorithm combined from these algorithms satisfies ∑i=1mεi- differential privacy.


## 2.2. Multitask Cascaded Convolutional Network (MTCNN)

With the development of CNN [27], Sun et al. [28] suggested that the CNN can be applied to localization face landmarks by virtue of its strong feature extraction ability. A three-layer deep CNN (DCNN) was designed to solve a thorny problem in landmark extraction: the inability to obtain the global optimal solution, due to the improper setting of initial values. Zhang et al. introduced multitask learning to face landmark localization and proposed a task-driven DCNN for face landmark localization (TCDCN). The TCDCN, a multitask learning model of four subtasks, has a smaller time complexity than traditional CNN [29]. Two years later, Zhang et al. developed the MTCNN, which effectively integrates face area detection with face landmark localization. The three cascading neural networks in MTCNN are responsible for face classification, bounding box regression, and key point localization, respectively [30]. Wu et al. [31] presented a tweaked CNN (TCNN), which relies on a mixed Gaussian model to cluster the features on different layers, and concluded that a deeper network layer can more accurately mirror face landmarks. In addition, many other methods have been adopted to locate face landmarks, namely, principal component analysis (PCA) [32], support vector machine [33], Bayesian probabilistic network (BPN) [34], dynamic link architecture (DLA) [35], and Gabor wavelet network (GWN) [36].The MTCNN is a multitask parallel face recognition method based on deep learning. This method has been widely recognized in the industry because it operates rapidly, and its accuracy meets the general requirements of face detection. The multitask parallel capacity of the MTCNN algorithm manifests as the simultaneous detection of multiple faces (detecting whether an image contains faces and finding the face locations in the image, as shown in Figure1) and the localization of face landmarks (locating the five landmarks, namely, two eyes, nose tip, and mouth corners, as shown in Figure 2).Figure 1
Face detection.Figure 2
Landmark localization.The MTCNN algorithm can be realized as a three-stage cascaded CNN. The first stage is called a proposal network (P-Net), which mainly obtains the candidate windows for the face area and the regression vector of the bounding box, carries out regression with the bounding box, calibrates the candidate windows, and merges highly overlapped candidate boxes through nonmaximum suppression (NMS). The structure of the P-Net is shown in Figure3.Figure 3
Structure of P-net.The second stage is called a refine network (R-Net), which eliminates the false positive areas through bounding box regression and NMS. That is, lots of nonface windows are denied by a more complex CNN in order to refine the face window. The structure of the R-Net is shown in Figure4.Figure 4
Structure of R-net.The third stage is called an output network (O-Net). With one more convolutional layer than the R-Net, the O-Net outputs more refined results. The functions of the O-Net are the same as those of the R-Net. But the O-Net carries out more supervision of the face area and output five landmarks. The structure of the O-Net is shown in Figure5.Figure 5
Structure of O-net.Each stage of the MTCNN is a multitask network, responsible for handling tasks like face/nonface judgement, face box regression, and landmark localization. The face/nonface judgement adopts the cross-entropy loss function:(4)Lidet=−yidetlogpi+1−yidet1−logpi,where pi is the probability for the image to contain faces; yidet∈0,1 is the true background label.The regression loss of face box can be calculated by Euclidean distance:(5)Libox=y^ibox−yibox22,where y^ is the background coordinates predicted by the network; y is the true background coordinates; yibox∈ℝ4ℝ4 is a 4-tuple composed of x of the upper left corner, y of the upper left corner, length, and width).Landmark localization is calculated similarly as face box regression. The Euclidean distance between the predicted landmark location and the true landmark location is calculated and minimized:(6)Lilandmark=y^ilandmark−yilandmark22,where yilandmark∈ℝ10 is a 10-tuple, for five landmarks are being localized.The simplest function of the training process can be expressed as follows:(7)min∑i=1N∑jεdet,box,landmarkαjβijLij,where N is the number of training samples; αj is the importance of a task; βij∈0,1 is a sample label. In the P-Net and R-Net, αdet = 1, αbox = 0.5, and αlandmark = 0.5. In the O-Net, αdet = 1, αbox = 0.5, and αlandmark = 1.


## 3. Methodology

### 3.1. Laplacian (LAP) Algorithm

This paper proposes the LAP algorithm based on the Laplace mechanism. In the LAP algorithm, every elementxij in the gray matrix Xm×n is regarded as an independent individual. This division method is the basis for applying the interactive mechanism to privacy protection. Each xij (1 ≤ i ≤ m, 1 ≤ j ≤ n) consumes a privacy budget of ε/m×n. The LAP algorithm can be realized in the steps in Algorithm 1.Algorithm 1: LAP.
Input: original imageX, privacy budget ε, parameters m and n, subgraph similarity expectation ThOutput: ImageX′ satisfying differential privacy(1)
Read the original imageX, convert the image into gray matrix and store it in matrix Xm×n(2)
fori = 1 to m(3)
fori = 1 to n(4)
Xi,j′ = Xj,i + lapΔQ×m×n/ε(5)
end for(6)
end for(7)
Output privacy protected pictureX′Compared with the original imageX, the privacy protected image X′ contains an additive noise of 2m×n×ΔQ×m×n/ε2. Admittedly, the LAP algorithm satisfies ε-differential privacy. However, the noise results show that the LAP algorithm will have a huge error if it is applied to protect the privacy of an excessively large image. If so, the noisy image will be of low availability.


### 3.2. Fusion Similarity Measurement Mechanism (FSMM)

This paper divides a face imageXm×n into multiple nonintersecting subgraphs Tij1≤i≤I,1≤j≤J, each of which contains multiple pixels:(8)Xm×n=x11,x12,⋯x1n⋮⋱⋮xm1,xm1,⋯xmn=T11,T12,⋯T1J⋮⋱⋮TI1,TI1,⋯TIJ.Through the division, the nonintersecting subgraphsTij carry as much information of the original image as possible.The key to regional growth is to determine the criterion. Due to the natural complexity of face images, the criterion should be designed by judging the various features of the original image. The traditional criterion for regional growth only focuses on the difference between gray values. The gray value of a single pixel cannot provide rich information about the image. By contrast, subgraphs as the basic units of regional growth can retain the luminance, contrast, structure, color, texture, and spatial distribution features of the image. This paper presents a new regional growth criterion called FSMM, which further improves the accuracy of area merging with subgraphs as the basic units. The FSMM can be realized in the steps in Algorithm2.Algorithm 2: FSMM.
Input: imageX, image YOutput: graph similarityθ(1)
Convert the image to a grayscale image,X′=X1⋯⋯⋯⋱⋯⋯⋯Xc,Y′=Y1⋯⋯⋯⋱⋯⋯⋯Yc(2)
Calculate the Euclidean distance betweenX′ and Y′, θ=∑i=1nXi−Yi22(3)
Calculate the Jaccard similarity index betweenX′ and Y′, ρX′,Y′=X′∩​Y′/X′∪​Y′(4)
θ=θ/ρX′,Y′+σ(5)
Combined with Hamming distance and perceptual hash algorithmϕ, ϕ=θ×ρX′,Y′+σ/dX′,Y′μX′,Y′<0.078×c1,0.078×c≤μX′,Y′≤0.156×c1/lg×m×n−μX′,Y′,μX′,Y′>0.156×c(6)
θ=ϕ×θ(7)
Calculate the luminance relationship betweenX′ and Y′, lX′,Y′=2uX′uY′+C1/uX′2+uY′2+C1(8)
Calculate the contrast ratio relationship betweenX′ and Y′, cX′,Y′=2σX′σY+C2/σX′2+σY′2+C2(9)
Calculate the structural relationship betweenX′ and Y′, sX′,Y′=σX′Y′+C3/σX′σY′+C3(10)
SSIMX′,Y′=lX′,Y′α·cX′,Y′β·sX′,Y′γ(11)
θ=SSIMX′,Y′×θ(12)
OutputθAs its name suggests, the new regional growth criterion relies on the FSMM, a brand-new similarity measure between images. In rws 3-4 of the algorithm, parameterσ, a very small positive number, appears to ensure the denominator is nonzero. In rows 7–9, uXanduY are, respectively, the mean values of images X and Y, reflecting the luminance features of the images; σXandσY are, respectively, the variances of images X and Y, reflecting the contrast features; C1,C2andC3 are very small positive numbers that ensure the denominator is nonzero. In Column 10, α,β,andγ are parameters adjusting the proportion of different eigenvalues: if α=β=γ=1,thenC1=K1L2,C2=K2L2,andC3=C2/2,withK1≪1,K2≪1,andL being the dynamic range of the image. The final calculation method for FSMMX,Y can be expressed as follows:(9)FSMMX,Y=ϕ×dX,Y/ρX,Y+σ×2uXuY+C1×2σXσY+C2uX2+uY2+C1×σX2+σY2+C2.


### 3.3. Privacy Protection for Sensitive Areas (PPSA) Algorithm

To achieve privacy protection of sensitive information (face area) in multiface images and reduce noise impact on the privacy protected images, this paper proposes a local privacy protection method for face images, which combines face recognition, regional growth, and differential privacy. The proposed algorithm is called the PPSA.Inspired by goal-driven reasoning (reverse reasoning), the PPSA was designed with goal as the starting point. The privacy protection for face images aims to prevent attackers from recognizing the target persons through face detection. Therefore, the first step of PPSA design is to understand how attackers identify faces. Next, a privacy protection method should be provided for face images, targeting the face identification technique of attackers. In this way, the designed privacy protection algorithm can effectively curb privacy leak and mitigate the effects of noise on the availability of the privacy protected image.The design of the PPSA needs to solve three key problems: (1) locate the sensitive area in the multiface image and recognize the landmarks in the area; (2) formulate the criterion for regional growth; (3) allocate the privacy budget reasonably to areas requiring different levels of protection.The PPSA algorithm is designed to protect the sensitive information (face information) in multiface images in batches. Different from single-face privacy protection, multiface privacy protection needs to deal with an unknown number of faces. Thus, the allocation of the privacy budget is crucial to PPSA design. The privacy protection should be tailored to areas with different protection requirements. In our scheme, the total privacy budgetε is divided into two parts: ε1 and ε2. The ε1 is evenly allocated to each seed Seedn (any subgraph containing a landmark is a seed), according to the estimated number of faces ρ contained in the image. During regional growth, Seedn′ (noisy seed) replaces the Ti,j (adjacent subgraph) meeting the growth criterion, thereby protecting the privacy of Ti,j. Note that the privacy protection of Ti,j does not consume any privacy budget. Meanwhile, the ε2 is allocated through dichotomization. This part of the privacy budget is consumed by the Seedn outside the estimated number of faces ρ, the Ti,j failing to meet the growth criterion, and the T^i,j belonging to the sensitive area but not involved in regional growth. The PPSA algorithm can be realized in the steps in Algorithm 3.Algorithm 3: PPSA.
Input: Original imageX, privacy budget ε, estimated number of faces ρ, preset parameters β, estimate the number of faces in the image ω, subgraph similarity expectation ThOutput: ImageX′ satisfying differential privacy.(1)
Read the original imageX, convert the image into gray matrix and store it in matrix Xm×n(2)
MTCNN is used to extract the face regionS=S1,S2,…Sω and face feature point KS=K1,K2,…K5×ω in face image X(3)
Set the gray matrixX according to the preset parameters β It is divided into subgraph sets TI,J with the same structure(4)
IfKS belongs to a subgraph Ti,j, set Ti,j as Seedn(5)
Find all backupSeedN=Seed1,Seed2,…Seed5×ω(6)
Adjust the face image areaS as S′ according to the sub image size(7)
ε=ε1+ε2(8)
ε1left=ε1(9)
ε2left=ε2/2(10)
SetSeed˜=0,Seed˜ used to record whether this seed has been used during regional growth(11)
Create a linkedlist to record the status of the current seed region merging process(12)
whileSeed˜≤5×ω)(13)
ifε1left>0(14)
iflist_head==NULL(15)
Pick unmarkedSeedn adds noise to it as Seedn′, Seedn′=Seedn+lapΔQ×5×ρ/ε1(16)
ε1left=ε1left−ε1/5ρ(17)
Attempt to merge adjacentTi,j start regional growth(18)
ifTi,j∉S′‖Euclidean_DistanceTi,j,Seedn≥MinEuclidean_DistanceSeedn,Seedm/2(19)
brake(20)
else(21)
ifFSMMSeedn′,Ti,j≤Th(22)
Ti,j=Seedn′(23)
else(24)
Ti,j′=Ti,j+lapΔQ/ε2left(25)
ε2left=ε2left/2(26)
Seedn is set to marked, Seed˜=Seed˜+1(27)
Release all data in the list, and the consolidation of this subregion is completed(28)
elseifε1left=0(29)
Execute line 15–line 28, line 16ΔQ×5×ρ/ε1 adjusted to ΔQ/ε2left, line 17 is adjusted to ε2left=ε2left/2(30)
T^i,j′=T^i,j+lapΔQ/ε2leftThe image preprocessing, privacy budget allocation, and regional growth are described in rows 1–6, rows 7–9, and rows 10–29, respectively. In row 30,T^i,j is the addition of Laplace noise to an unprotected subgraph in S′. In row 18, the PPSA limits the range of regional growth: if a Ti,j meeting the growth criterion but lying beyond the coverage of the current Si′ or the growth range of the current Seedn, it will not be able to complete this area merging; the growth range of Seedn will be limited to half of the Euclidean distance to the nearest adjacent Seedm. If ω≤ρ, only rows 10–27 will be executed for regional growth. If ω>ρ, all the rows from 10–29 will be executed for regional growth.Theorem 2.
The PPSA consumes a privacy budget smaller thanε and satisfies ε-differential privacy.Proof.
In the PPSA, the total privacy budgetε is divided into two parts. Among them, ε1 is used to add noise to seeds. If ω<ρ, there exists ε1left>0 after the end of the algorithm; if ω≥ρ, there exists ε1left=0 after the end of the algorithm. Meanwhile, ε2 is allocated by dichotomy:(10)ε2×121+122+123+⋯12N<ε2N⟶+∞.
Thus, there exists the following:(11)ε−ε1left+ε2left>0.
Therefore, the PPSA consumes a privacy budget smaller thanε. According to Property 1, the PPSA satisfies ε-differential privacy.
Q.E.D.Theorem 3.
The error of the PPSA is no greater than that of the LAP, i.e.,(12)ErrorDPLP<ErrorLAP.Proof.
In the PPSA, the privacy protected image contains four kinds of subgraphs, namely, noise-free subgraphPi,j, seed Seedn, subgraph merged in regional growth Ti,j, and subgraph belonging to a sensitive area but not involved in regional growth T^i,j.(1)
ForPi,j:(13)Pi,j_PPSA′=0Pi,j_LAP'=Pi,j+lapΔQ×I×Jε.Subtracting the two formulas above, we have the following:ErrorPPSAPi,j<ErrorLAPPi,j.(2)
ForT^i,j:(14)T^i,j_LAP′=T^i,j+lapΔQ×I×JεT^i,j_PPSA′=T^i,j+lapΔQ×2Mε2.If2M≤I×J, there exists ErrorPPSAPi,j<ErrorLAPPi,j.If2M>I×J, there exists ErrorPPSAPi,j≈ErrorLAPPi,j.The validity ofErrorPPSAPi,j≈ErrorLAPPi,j depends on the uniqueness of the gray image. In the gray image matrix, the gray value falls between 0 and 255. Under the Laplace mechanism, the smaller the ε, the greater the noise. For any other type of data, the noise effect will be huge, if ε is sufficiently small. For gray value, however, the fluctuation induced by noise is limited. Take T^i,j_PPSA′=T^i,j+lapΔQ×2M/ε2 for example. When lapΔQ×2M/ε2⟶+∞ 时, T^i,j_PPSA′ value in the gray matrix will become 255. When lapΔQ×2M/ε2⟶−∞, that value will change to 0. Thus, ErrorPPSAPi,j≈ErrorLAPPi,j holds, as long as I×J is sufficiently great.(3)
ForSeedn:There are two possible scenarios:ω≤ρ and ω>ρ.Whenω≤ρ, Seedn_DPLP′ only consumes ε1. Then, there exists the following:(15)Seedn_PPSA′=Seedn+lapΔQ×5×ρε1Seedn_LAP′=Seedn+lapΔQ×I×Jε.If5×ρ≪I×J, the numerical gap between ε1 and ε is so small as to be negligible. Subtracting the above two formulas, we have the following: ErrorPPSASeedn<ErrorLAPSeedn.Whenω>ρ, the first ρ face areas consume ε1 until no budget in this part is left. In this case, there exists ErrorPPSASeedn<ErrorLAPSeedn. For the lack of space, the proof is omitted.In the followingω−ρ face images, Seedn_DPLP′ needs to consume ε2. According to the proof of T^i,j, there exists ErrorPPSASeedn≈ErrorLAPSeedn during the consumption of ε2. Thus, it can be deduced that, in any scenario, there exists the following:(16)ErrorPPSASeednε1+ErrorPPSASeednε2<ErrorLAPSeednε.(4)
ForTi,j:There are also two possible scenarios:ω≤ρ and ω>ρ. When ω≤ρ, the proof is as follows:(17)ErrorPPSATi,j=Ti,j_PPSA′=Seedn+lapΔQ×5×ρε1−Ti,jErrorLAPTi,j=Ti,j_LAP′=lapΔQ×I×Jε.Subtracting the above two formulas, we have the following:(18)ErrorPPSATi,j−ErrorLAPTi,j=Seedn+lapΔΔQ×5×ρε1−Ti,j−lapΔQ×I×Jε=Seedn+lapΔQ×5×ρε1−Ti,j+lapΔQ×I×Jε=Seedn_PPSA′−Ti,j_LAP′.For a gray image, when the privacy budget is sufficiently small under the Laplace mechanism,ErrorTi,j_LAP′ is equivalent to ErrorSeedn_LAP′. Thus, we have the following:(19)Seedn_PPSA′−Ti,j_LAP′=Seedn_PPSA′−Seedn_LAP′=Seedn+lapΔQ×5×ρε1−Seedn−lapΔQ×I×Jε=lapΔQ×5×ρε1−lapΔQ×I×Jε<0.Thus,ErrorPPSATi,j<ErrorLAPTi,j.Similarly, whenω>ρ, there exists ErrorPPSATi,j≈ErrorLAPTi,j. For the lack of space, the proof is omitted.Q.E.D.


## 3.1. Laplacian (LAP) Algorithm

This paper proposes the LAP algorithm based on the Laplace mechanism. In the LAP algorithm, every elementxij in the gray matrix Xm×n is regarded as an independent individual. This division method is the basis for applying the interactive mechanism to privacy protection. Each xij (1 ≤ i ≤ m, 1 ≤ j ≤ n) consumes a privacy budget of ε/m×n. The LAP algorithm can be realized in the steps in Algorithm 1.Algorithm 1: LAP.
Input: original imageX, privacy budget ε, parameters m and n, subgraph similarity expectation ThOutput: ImageX′ satisfying differential privacy(1)
Read the original imageX, convert the image into gray matrix and store it in matrix Xm×n(2)
fori = 1 to m(3)
fori = 1 to n(4)
Xi,j′ = Xj,i + lapΔQ×m×n/ε(5)
end for(6)
end for(7)
Output privacy protected pictureX′Compared with the original imageX, the privacy protected image X′ contains an additive noise of 2m×n×ΔQ×m×n/ε2. Admittedly, the LAP algorithm satisfies ε-differential privacy. However, the noise results show that the LAP algorithm will have a huge error if it is applied to protect the privacy of an excessively large image. If so, the noisy image will be of low availability.


## 3.2. Fusion Similarity Measurement Mechanism (FSMM)

This paper divides a face imageXm×n into multiple nonintersecting subgraphs Tij1≤i≤I,1≤j≤J, each of which contains multiple pixels:(8)Xm×n=x11,x12,⋯x1n⋮⋱⋮xm1,xm1,⋯xmn=T11,T12,⋯T1J⋮⋱⋮TI1,TI1,⋯TIJ.Through the division, the nonintersecting subgraphsTij carry as much information of the original image as possible.The key to regional growth is to determine the criterion. Due to the natural complexity of face images, the criterion should be designed by judging the various features of the original image. The traditional criterion for regional growth only focuses on the difference between gray values. The gray value of a single pixel cannot provide rich information about the image. By contrast, subgraphs as the basic units of regional growth can retain the luminance, contrast, structure, color, texture, and spatial distribution features of the image. This paper presents a new regional growth criterion called FSMM, which further improves the accuracy of area merging with subgraphs as the basic units. The FSMM can be realized in the steps in Algorithm2.Algorithm 2: FSMM.
Input: imageX, image YOutput: graph similarityθ(1)
Convert the image to a grayscale image,X′=X1⋯⋯⋯⋱⋯⋯⋯Xc,Y′=Y1⋯⋯⋯⋱⋯⋯⋯Yc(2)
Calculate the Euclidean distance betweenX′ and Y′, θ=∑i=1nXi−Yi22(3)
Calculate the Jaccard similarity index betweenX′ and Y′, ρX′,Y′=X′∩​Y′/X′∪​Y′(4)
θ=θ/ρX′,Y′+σ(5)
Combined with Hamming distance and perceptual hash algorithmϕ, ϕ=θ×ρX′,Y′+σ/dX′,Y′μX′,Y′<0.078×c1,0.078×c≤μX′,Y′≤0.156×c1/lg×m×n−μX′,Y′,μX′,Y′>0.156×c(6)
θ=ϕ×θ(7)
Calculate the luminance relationship betweenX′ and Y′, lX′,Y′=2uX′uY′+C1/uX′2+uY′2+C1(8)
Calculate the contrast ratio relationship betweenX′ and Y′, cX′,Y′=2σX′σY+C2/σX′2+σY′2+C2(9)
Calculate the structural relationship betweenX′ and Y′, sX′,Y′=σX′Y′+C3/σX′σY′+C3(10)
SSIMX′,Y′=lX′,Y′α·cX′,Y′β·sX′,Y′γ(11)
θ=SSIMX′,Y′×θ(12)
OutputθAs its name suggests, the new regional growth criterion relies on the FSMM, a brand-new similarity measure between images. In rws 3-4 of the algorithm, parameterσ, a very small positive number, appears to ensure the denominator is nonzero. In rows 7–9, uXanduY are, respectively, the mean values of images X and Y, reflecting the luminance features of the images; σXandσY are, respectively, the variances of images X and Y, reflecting the contrast features; C1,C2andC3 are very small positive numbers that ensure the denominator is nonzero. In Column 10, α,β,andγ are parameters adjusting the proportion of different eigenvalues: if α=β=γ=1,thenC1=K1L2,C2=K2L2,andC3=C2/2,withK1≪1,K2≪1,andL being the dynamic range of the image. The final calculation method for FSMMX,Y can be expressed as follows:(9)FSMMX,Y=ϕ×dX,Y/ρX,Y+σ×2uXuY+C1×2σXσY+C2uX2+uY2+C1×σX2+σY2+C2.


## 3.3. Privacy Protection for Sensitive Areas (PPSA) Algorithm

To achieve privacy protection of sensitive information (face area) in multiface images and reduce noise impact on the privacy protected images, this paper proposes a local privacy protection method for face images, which combines face recognition, regional growth, and differential privacy. The proposed algorithm is called the PPSA.Inspired by goal-driven reasoning (reverse reasoning), the PPSA was designed with goal as the starting point. The privacy protection for face images aims to prevent attackers from recognizing the target persons through face detection. Therefore, the first step of PPSA design is to understand how attackers identify faces. Next, a privacy protection method should be provided for face images, targeting the face identification technique of attackers. In this way, the designed privacy protection algorithm can effectively curb privacy leak and mitigate the effects of noise on the availability of the privacy protected image.The design of the PPSA needs to solve three key problems: (1) locate the sensitive area in the multiface image and recognize the landmarks in the area; (2) formulate the criterion for regional growth; (3) allocate the privacy budget reasonably to areas requiring different levels of protection.The PPSA algorithm is designed to protect the sensitive information (face information) in multiface images in batches. Different from single-face privacy protection, multiface privacy protection needs to deal with an unknown number of faces. Thus, the allocation of the privacy budget is crucial to PPSA design. The privacy protection should be tailored to areas with different protection requirements. In our scheme, the total privacy budgetε is divided into two parts: ε1 and ε2. The ε1 is evenly allocated to each seed Seedn (any subgraph containing a landmark is a seed), according to the estimated number of faces ρ contained in the image. During regional growth, Seedn′ (noisy seed) replaces the Ti,j (adjacent subgraph) meeting the growth criterion, thereby protecting the privacy of Ti,j. Note that the privacy protection of Ti,j does not consume any privacy budget. Meanwhile, the ε2 is allocated through dichotomization. This part of the privacy budget is consumed by the Seedn outside the estimated number of faces ρ, the Ti,j failing to meet the growth criterion, and the T^i,j belonging to the sensitive area but not involved in regional growth. The PPSA algorithm can be realized in the steps in Algorithm 3.Algorithm 3: PPSA.
Input: Original imageX, privacy budget ε, estimated number of faces ρ, preset parameters β, estimate the number of faces in the image ω, subgraph similarity expectation ThOutput: ImageX′ satisfying differential privacy.(1)
Read the original imageX, convert the image into gray matrix and store it in matrix Xm×n(2)
MTCNN is used to extract the face regionS=S1,S2,…Sω and face feature point KS=K1,K2,…K5×ω in face image X(3)
Set the gray matrixX according to the preset parameters β It is divided into subgraph sets TI,J with the same structure(4)
IfKS belongs to a subgraph Ti,j, set Ti,j as Seedn(5)
Find all backupSeedN=Seed1,Seed2,…Seed5×ω(6)
Adjust the face image areaS as S′ according to the sub image size(7)
ε=ε1+ε2(8)
ε1left=ε1(9)
ε2left=ε2/2(10)
SetSeed˜=0,Seed˜ used to record whether this seed has been used during regional growth(11)
Create a linkedlist to record the status of the current seed region merging process(12)
whileSeed˜≤5×ω)(13)
ifε1left>0(14)
iflist_head==NULL(15)
Pick unmarkedSeedn adds noise to it as Seedn′, Seedn′=Seedn+lapΔQ×5×ρ/ε1(16)
ε1left=ε1left−ε1/5ρ(17)
Attempt to merge adjacentTi,j start regional growth(18)
ifTi,j∉S′‖Euclidean_DistanceTi,j,Seedn≥MinEuclidean_DistanceSeedn,Seedm/2(19)
brake(20)
else(21)
ifFSMMSeedn′,Ti,j≤Th(22)
Ti,j=Seedn′(23)
else(24)
Ti,j′=Ti,j+lapΔQ/ε2left(25)
ε2left=ε2left/2(26)
Seedn is set to marked, Seed˜=Seed˜+1(27)
Release all data in the list, and the consolidation of this subregion is completed(28)
elseifε1left=0(29)
Execute line 15–line 28, line 16ΔQ×5×ρ/ε1 adjusted to ΔQ/ε2left, line 17 is adjusted to ε2left=ε2left/2(30)
T^i,j′=T^i,j+lapΔQ/ε2leftThe image preprocessing, privacy budget allocation, and regional growth are described in rows 1–6, rows 7–9, and rows 10–29, respectively. In row 30,T^i,j is the addition of Laplace noise to an unprotected subgraph in S′. In row 18, the PPSA limits the range of regional growth: if a Ti,j meeting the growth criterion but lying beyond the coverage of the current Si′ or the growth range of the current Seedn, it will not be able to complete this area merging; the growth range of Seedn will be limited to half of the Euclidean distance to the nearest adjacent Seedm. If ω≤ρ, only rows 10–27 will be executed for regional growth. If ω>ρ, all the rows from 10–29 will be executed for regional growth.Theorem 2.
The PPSA consumes a privacy budget smaller thanε and satisfies ε-differential privacy.Proof.
In the PPSA, the total privacy budgetε is divided into two parts. Among them, ε1 is used to add noise to seeds. If ω<ρ, there exists ε1left>0 after the end of the algorithm; if ω≥ρ, there exists ε1left=0 after the end of the algorithm. Meanwhile, ε2 is allocated by dichotomy:(10)ε2×121+122+123+⋯12N<ε2N⟶+∞.
Thus, there exists the following:(11)ε−ε1left+ε2left>0.
Therefore, the PPSA consumes a privacy budget smaller thanε. According to Property 1, the PPSA satisfies ε-differential privacy.
Q.E.D.Theorem 3.
The error of the PPSA is no greater than that of the LAP, i.e.,(12)ErrorDPLP<ErrorLAP.Proof.
In the PPSA, the privacy protected image contains four kinds of subgraphs, namely, noise-free subgraphPi,j, seed Seedn, subgraph merged in regional growth Ti,j, and subgraph belonging to a sensitive area but not involved in regional growth T^i,j.(1)
ForPi,j:(13)Pi,j_PPSA′=0Pi,j_LAP'=Pi,j+lapΔQ×I×Jε.Subtracting the two formulas above, we have the following:ErrorPPSAPi,j<ErrorLAPPi,j.(2)
ForT^i,j:(14)T^i,j_LAP′=T^i,j+lapΔQ×I×JεT^i,j_PPSA′=T^i,j+lapΔQ×2Mε2.If2M≤I×J, there exists ErrorPPSAPi,j<ErrorLAPPi,j.If2M>I×J, there exists ErrorPPSAPi,j≈ErrorLAPPi,j.The validity ofErrorPPSAPi,j≈ErrorLAPPi,j depends on the uniqueness of the gray image. In the gray image matrix, the gray value falls between 0 and 255. Under the Laplace mechanism, the smaller the ε, the greater the noise. For any other type of data, the noise effect will be huge, if ε is sufficiently small. For gray value, however, the fluctuation induced by noise is limited. Take T^i,j_PPSA′=T^i,j+lapΔQ×2M/ε2 for example. When lapΔQ×2M/ε2⟶+∞ 时, T^i,j_PPSA′ value in the gray matrix will become 255. When lapΔQ×2M/ε2⟶−∞, that value will change to 0. Thus, ErrorPPSAPi,j≈ErrorLAPPi,j holds, as long as I×J is sufficiently great.(3)
ForSeedn:There are two possible scenarios:ω≤ρ and ω>ρ.Whenω≤ρ, Seedn_DPLP′ only consumes ε1. Then, there exists the following:(15)Seedn_PPSA′=Seedn+lapΔQ×5×ρε1Seedn_LAP′=Seedn+lapΔQ×I×Jε.If5×ρ≪I×J, the numerical gap between ε1 and ε is so small as to be negligible. Subtracting the above two formulas, we have the following: ErrorPPSASeedn<ErrorLAPSeedn.Whenω>ρ, the first ρ face areas consume ε1 until no budget in this part is left. In this case, there exists ErrorPPSASeedn<ErrorLAPSeedn. For the lack of space, the proof is omitted.In the followingω−ρ face images, Seedn_DPLP′ needs to consume ε2. According to the proof of T^i,j, there exists ErrorPPSASeedn≈ErrorLAPSeedn during the consumption of ε2. Thus, it can be deduced that, in any scenario, there exists the following:(16)ErrorPPSASeednε1+ErrorPPSASeednε2<ErrorLAPSeednε.(4)
ForTi,j:There are also two possible scenarios:ω≤ρ and ω>ρ. When ω≤ρ, the proof is as follows:(17)ErrorPPSATi,j=Ti,j_PPSA′=Seedn+lapΔQ×5×ρε1−Ti,jErrorLAPTi,j=Ti,j_LAP′=lapΔQ×I×Jε.Subtracting the above two formulas, we have the following:(18)ErrorPPSATi,j−ErrorLAPTi,j=Seedn+lapΔΔQ×5×ρε1−Ti,j−lapΔQ×I×Jε=Seedn+lapΔQ×5×ρε1−Ti,j+lapΔQ×I×Jε=Seedn_PPSA′−Ti,j_LAP′.For a gray image, when the privacy budget is sufficiently small under the Laplace mechanism,ErrorTi,j_LAP′ is equivalent to ErrorSeedn_LAP′. Thus, we have the following:(19)Seedn_PPSA′−Ti,j_LAP′=Seedn_PPSA′−Seedn_LAP′=Seedn+lapΔQ×5×ρε1−Seedn−lapΔQ×I×Jε=lapΔQ×5×ρε1−lapΔQ×I×Jε<0.Thus,ErrorPPSATi,j<ErrorLAPTi,j.Similarly, whenω>ρ, there exists ErrorPPSATi,j≈ErrorLAPTi,j. For the lack of space, the proof is omitted.Q.E.D.


## 4. Experiments and Results Analysis

### 4.1. Experiments

To demonstrate its feasibility, the PPSA was tested on a 750 ∗ 1020 image containing 8 faces from the WIDER FACE dataset. The original image was divided into 7,650 subgraphs of the size 10 ∗ 10. During the implementation of the PPSA, the original image (Figure6) was first transformed into a gray image (Figure 7). Then, the MTCNN algorithm was called to recognize the face area S and landmark KS (Figure 8). To meet the requirements of the PPSA on regional growth, the face image was divided into multiple equal size TI,J (Figure 9), using the preset parameters. The subgraph area (Figure 10) overlapping the landmark location was taken as the SeedN (Figure 11) of regional growth. However, the S detected by the MTCNN is not compatible with the size fTI,J (Figure 12). To solve the problem, S was resized to S′ (Figure 13), according to the size of TI,J.Figure 6
Original image (RGB).Figure 7
Gray image.Figure 8
S and KS.Figure 9
Rendering ofT(i, j).Figure 10
Rendering ofKS and T(i, j).Figure 11
Rendering of SeedN.Figure 12
Rendering of SeedN.Figure 13
Rendering of SeedN and S′.Figure14 provides the results of the LPA. Figure 15 presents the PPSA results under the same privacy budget. Observations show that the PPSA operates within the face area while the LAP acts across the image. Besides, within the scope of the regional growth algorithm, the PPSA retained the recognizability of landmarks. Although not excluding the possibility of privacy leaks, this strikes a balance between the privacy and availability of the privacy protected multiface image. In addition, the relationship between the number of faces ρ estimated by the PPSA and the true number of faces ω determines how much privacy budget can be obtained by a seed. The allocation of the privacy budget will directly affect the error of the protected image. Figures 16 and 17 present the local results at different relationships between ρ and ω.Figure 14
LAP results.Figure 15
PPSA results.Figure 16
Local results atω≤ρ.Figure 17
Local results atω>ρ.


### 4.2. Results Analysis

To verify the feasibility of our algorithm, multiple face images were collected from WIDER FACE dataset, i-bug face dataset, and AFW dataset and subjected to experiments using TensorFlow + AlexNet CNN. The experiments were carried out in the environment of Intel® Core i9-9900K CPU @ 3.60 GHz, 32G memory, GTX 21080TI GPU, and Windows 10. The privacy budget was set to 1, 2, 3, 4, and 5, in turn. The face recognition performance was measured by precision, recall, andF1-score. The experimental results are displayed in Figures 18–26.Figure 18
WIDER and precision.Figure 19
WIDER and recall.Figure 20
IDER andF1-score.Figure 21
I-bug and precision.Figure 22
I-bug and recall.Figure 23
I-bug andF1-score.Figure 24
AFW and precision.Figure 25
AFW and recall.Figure 26
AFW andF1-score.The experimental results show that the PPSA achieved different results on images from different datasets. The difference arises from the varied image sizes of different datasets: 760 ∗ 1000 in WIDER, 500 ∗ 800 in i-bug, and 1000 ∗ 1500 in AFW. In addition, the different background complexities between the face datasets also influence the operation effects of LAP, sliding window publication (SWP) algorithm, and region growing publication (RGP). By contrast, the operation of the PPSA is not affected by image size and background complexity.From the experimental results, it can be learned that the operation results of the PPSA mainly depend on the number of estimated facesρ and S′. In the images from i-bug, there are relatively few faces in each image, but the face area takes up a large portion of the entire image. In the images from WIDER, there are many faces in each image, and the face area takes up a large portion of the entire image. In the images from the AFW, there are relatively few faces in each image, but the face area takes up a small portion of the entire image, for the images tend to be large. Therefore, it is expected that the PPSA’s privacy protection effect is positively correlated with ω, and negatively with S′. The prediction is consistent with the experimental results.


## 4.1. Experiments

To demonstrate its feasibility, the PPSA was tested on a 750 ∗ 1020 image containing 8 faces from the WIDER FACE dataset. The original image was divided into 7,650 subgraphs of the size 10 ∗ 10. During the implementation of the PPSA, the original image (Figure6) was first transformed into a gray image (Figure 7). Then, the MTCNN algorithm was called to recognize the face area S and landmark KS (Figure 8). To meet the requirements of the PPSA on regional growth, the face image was divided into multiple equal size TI,J (Figure 9), using the preset parameters. The subgraph area (Figure 10) overlapping the landmark location was taken as the SeedN (Figure 11) of regional growth. However, the S detected by the MTCNN is not compatible with the size fTI,J (Figure 12). To solve the problem, S was resized to S′ (Figure 13), according to the size of TI,J.Figure 6
Original image (RGB).Figure 7
Gray image.Figure 8
S and KS.Figure 9
Rendering ofT(i, j).Figure 10
Rendering ofKS and T(i, j).Figure 11
Rendering of SeedN.Figure 12
Rendering of SeedN.Figure 13
Rendering of SeedN and S′.Figure14 provides the results of the LPA. Figure 15 presents the PPSA results under the same privacy budget. Observations show that the PPSA operates within the face area while the LAP acts across the image. Besides, within the scope of the regional growth algorithm, the PPSA retained the recognizability of landmarks. Although not excluding the possibility of privacy leaks, this strikes a balance between the privacy and availability of the privacy protected multiface image. In addition, the relationship between the number of faces ρ estimated by the PPSA and the true number of faces ω determines how much privacy budget can be obtained by a seed. The allocation of the privacy budget will directly affect the error of the protected image. Figures 16 and 17 present the local results at different relationships between ρ and ω.Figure 14
LAP results.Figure 15
PPSA results.Figure 16
Local results atω≤ρ.Figure 17
Local results atω>ρ.


## 4.2. Results Analysis

To verify the feasibility of our algorithm, multiple face images were collected from WIDER FACE dataset, i-bug face dataset, and AFW dataset and subjected to experiments using TensorFlow + AlexNet CNN. The experiments were carried out in the environment of Intel® Core i9-9900K CPU @ 3.60 GHz, 32G memory, GTX 21080TI GPU, and Windows 10. The privacy budget was set to 1, 2, 3, 4, and 5, in turn. The face recognition performance was measured by precision, recall, andF1-score. The experimental results are displayed in Figures 18–26.Figure 18
WIDER and precision.Figure 19
WIDER and recall.Figure 20
IDER andF1-score.Figure 21
I-bug and precision.Figure 22
I-bug and recall.Figure 23
I-bug andF1-score.Figure 24
AFW and precision.Figure 25
AFW and recall.Figure 26
AFW andF1-score.The experimental results show that the PPSA achieved different results on images from different datasets. The difference arises from the varied image sizes of different datasets: 760 ∗ 1000 in WIDER, 500 ∗ 800 in i-bug, and 1000 ∗ 1500 in AFW. In addition, the different background complexities between the face datasets also influence the operation effects of LAP, sliding window publication (SWP) algorithm, and region growing publication (RGP). By contrast, the operation of the PPSA is not affected by image size and background complexity.From the experimental results, it can be learned that the operation results of the PPSA mainly depend on the number of estimated facesρ and S′. In the images from i-bug, there are relatively few faces in each image, but the face area takes up a large portion of the entire image. In the images from WIDER, there are many faces in each image, and the face area takes up a large portion of the entire image. In the images from the AFW, there are relatively few faces in each image, but the face area takes up a small portion of the entire image, for the images tend to be large. Therefore, it is expected that the PPSA’s privacy protection effect is positively correlated with ω, and negatively with S′. The prediction is consistent with the experimental results.


## 5. Conclusions

To protect the privacy of multiface images, this paper combines face detection, regional growth, with the Laplace mechanism of differential privacy to add noises to local sensitive areas in face images and realizes the privacy protection for the local sensitive areas in multiface images under an interactive framework. Compared with the LAP, SWP, and RGP, the proposed PPSA can effectively suppress the noise impact on the protected image and improve the availability of the privacy protected image. Moreover, the PPSA applies to images of various sizes, and its error does not increase with the image size.Although the PPSA can effectively protect the face information in multiface images, the attackers may choose to attack the hair, clothes, and body of persons in the images. Besides, privacy leak may arise from the relationship between persons, and the correlation between each person and the background. The future work will further improve image privacy protection, trying to solve these potential risks.



---

*Source: 5919522-2022-03-15.xml*