Latent Representation Reorganization for Face Privacy Protection

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 09:46:29

Abstract: The issue of face privacy protection has aroused wide social concern along with the increasing applications of face images. The latest methods focus on achieving a good privacy-utility tradeoff so that the protected results can still be used to support the downstream computer vision tasks. However, they may suffer from limited flexibility in manipulating this tradeoff because the practical requirements may vary under different scenarios. In this paper, we present a two-stage latent representation reorganization (LReOrg) framework for face image privacy protection relying on our conditional bidirectional network which is optimized by using a distinct keyword-based swap training strategy with a multi-task loss. The privacy sensitive information are anonymized in the first stage and the destroyed useful information are recovered in the second stage according to user requirements. LReOrg is advantageous in: (a) enabling users to recurrently process fine-grained attributes; (b) providing flexible control over privacy-utility tradeoff by manipulating which attributes to anonymize or preserve using cross-modal keywords; and (c) eliminating the need of data annotations for network training. The experimental results on benchmark datasets have reported the superior ability of our approach for providing flexible protection on facial information.

Main paper: 
 introduction: 
 The issue of face privacy protection has aroused wide social concerns along with the increasing application of face images that carry lots of personal information (e.g. identity or religion) [31, 48, 56] . For example, the AI classifiers and DeepFake tools may easily read personal information and generate illegal clone avatars, which may bring about troubles (e.g. economic fraud) to individuals or organizations if the data are misused. This has led to the set up of more strict laws and regulations (e.g. GDPR, CCPA, PDPA and PIPA [1, 35, 38, 57, 58] ) on data management. The immediate consequence is that people need to comply complicated legal or ethical constrains to avoid making troubles when accessing, using or disseminating the face data, which may block many important scientific researches or intelligent applications. One feasible way is to develop effective anonymization techniques to protect the sensitive attributes and preserve the desired non-sensitive ones because people may also expect that the protected data can still be useful (i.e. utility preservation) [11, 41, 45] , where we collectively call all facial information as attributes (e.g. identity, gender and age). Such kind of techniques usually focuses on cheating both human and machine, and can be used to support various computer vision tasks (e.g. street-view map, autonomous drive and medical diagnostics [3, 30, 32, 60] ) to clear up the restrains on privacy, ethics, laws and regulations.Existing works show that privacy protection and utility preservation is a tradeoff problem, i.e. privacy-utility (PU) tradeoff, because both items are usually correlated [34, 38, 45, 60] . A higher performance of the former usually corresponds to a lower performance of the latter and vice versa. Recently, the generative methods [15, 23, 27, 46] receive increasing attention for producing realistic face images and achieving better PU tradeoff and various excellent methods have been proposed [5, 7, 16, 21, 22, 30, 38, 47, 54, 58] by replacing the original face image with a synthesized anonymous one, where some privacy protection strategies, like k-anonymity and differential privacy [12, 14, 40, 45, 49, 50, 54] , were also studied to provide formal privacy guarantees. Although significant progresses have achieved, the up-to-date works still suffer from limited flexibility in manipulating the PU tradeoff under practical conditions and requirements by focusing on protecting the identity because some attributes may also become sensitive in some cases, especially when they are correlated to some special entities, events or activities, like religion, ethnic, laws and so on. Thus, it is reasonable and necessary to develop a more flexible mechnism for privacy protection.To address the above problem, this paper presents a latent representation reorganization (LReOrg) framework for face privacy protection based on our conditional disentanglement-fusion network (CDFNet). LReOrg has several advantages for flexible anonymization: (1) it can enable users to recurrently process fine-grained attributes; (2) it can provide flexible control over PU tradeoff by manipulating which attributes to anonymize or preserve using cross-modal keywords; (3) it does not require any data annotations for network training. Existing works usually treat anonymization as a binary problem that hides the original identity and struggles to preserve the other attributes. Differently, we generalize this as a fine-grained problem by unifying privacy protection and utility preservation in a recurrent process by letting users to determine how to perform anonymization (see Figure 1 ). To our best knowledge, this is the first time that the recurrent framework has been successfully used for deep face privacy protection. Our main contributions can be summarized as follows:• A more resonable framework LReOrg is proposed for achiving a more flexible face anonymization by taking cross-modal keywords as fine-grained conditions. • A CDFNet is designed to support forward feature disentanglement and backward feature fusion, which can recurrently support sensitive attributes anonymization and nonsensitive attributes recovery. • We introduce a keyword-based swap training strategy supervised by using the CLIP model [44] and a multi-task loss. • We rely on extensive experiments to quantitatively and qualitatively show the state-of-the-art performance of LReOrg by studying the privacy protection and utility preservation performances with respect to different attributes.Note that CDFNet is built by following the architecture of Invertible Neural Network (INN) due to its excellent performance in image generation tasks [2, 25] . The work most related to ours is HiNet [25] which focuses on image steganography based on INN. CDFNet can be seen as a generalization of it, but they are quite different. First, CDFNet focuses on privacy protection, while HiNet focuses on image steganography. Second, CDFNet presents a new conditional version of INN based on cross-modal keywords. Third, the building blocks,their input and output are different. Besides, the network optimization method is also different. 
related works: 
 In this section, we discuss the most related works in contrastive language-image pretraining (CLIP) and face privacy protection.Contrastive Language-Image Pretraining. Cross-modal vision and language representation has received lots of attention in various tasks these years, such as image caption and visual question answering. The success of Transformer [52] and BERT [9] has inspired many interesting works [43, 44, 55] . The recent CLIP model [44] has received wide attention by learning a multi-modal embedding space, which can be used to measure the sematic similarity between text and image. CLIP was trained on a 400 million sized dataset collected from the Internet, which has demonstrated powerful performances on various tasks and datasets. Due to the powerful ability of CLIP, we employ it as the cross-modal attribute discriminator to train our network to make it understand the relationships between text conditions and the anonymized face image.Face Privacy Protection. Anonymization is regarded as an effective way to protect the privacy of face images, which is usually realized by de-identifying or hiding the original face identity while preserving the data usability. The commonly used simple anonymization methods, like blurring, pixelation and blacking out [24, 36] , can destroy the data utility which receives increasing attention to enable the reusability of the anonymized data [20, 41, 45] . In recent years, the generative methods [15, 27, 28, 43, 46] exhibit promising performances on face image synthesis and anonymization by playing adversarial games [5, 21, 22, 30, 37-39, 47, 54, 59] . In [21] , DeepPrivacy (DP1) relys on inpainting to generate anonymous face by blocking out the facial region. In [38] , CIAGAN relys on masked image, landmarks and one-hot vector to perform conditional inpainting for face anonymization. In [54] , IdentityDP relys on disentanglement and differential privacy [12, 49] for anoymous face synthesis after adding Laplace noise to identity feature. Although these methods can well protect the face identity, they suffer from some drawbacks on the naturalness of the anonymous face. In [22] , DeepPrivacy2 (DP2) relys on continuous surface embedding and StyleGAN to further improve DP1. In [47] , Clip2Protect relys on text-guided CLIP [44] and the StyleGAN latent space to generate anonymous face from the viewpoint of makeup, but it is time-consuming to finetune a new StyleGAN generator for each inference. In [30] , LDFA presents a similar method as with DP1 by performing face inpainting based on latent diffusion model [46] . Although the image quality has greatly improved, these up-to-date works usually work in their predefined manner by mainly processing the identity, but lack of a mechanism to flexibly manipulate which attributes to anonymize or preserve in a more intuitive manner (i.e. poor flexibility). Differently, in this paper, we present a new privacy protection mechanism to perform fine-grained anonymization with cross-modal keyworks based on our bidirectional CDFNet, which can work recurrently according to practical requirements. 
method: 
 In this section, we introduce the proposed LReOrg framework. We first have a brief overview in Section 3.1. Then, we introduce the proposed CDFNet network in Section 3.2. Finally, we present our traning strategy in Section 3.4.In Figure 2 , we plot the flowchart of LReOrg for face anonymization conditioned on the cross-modal keyword-based attribute representation in set 𝐾 = 𝐾 1 ∪ 𝐾 2 , where 𝐾 1 = {𝑘 𝑖 , 1 ≤ 𝑖 < 𝑀 } is the sensitive keyword set, 𝐾 2 = {𝑘 𝑖 , 𝑀 ≤ 𝑖 < 𝑁 } is the non-sensitive keyword set and each keyword corresponds to a different face attribute, such as 'identity', 'expression', 'gender' and 'age". The flowchart conissits of four steps. First, we generate a latent representation 𝑠 0 𝑥 for a given face image 𝑥 by embedding it into some well defined latent space using encoder 𝐸. Second, we rely on the anonymous reorganization (AR) module to anonymize the sensitive attributes encoded in 𝑠 0 𝑥 according to the keywords in 𝐾 1 (e.g. {'identity'}) by using our CDFNet and anonymizer 𝐴. Then, we rely on the recovery reorganization (RR) module to recover the nonsensitive attributes for the output 𝑠 𝑀 𝑥 of AR according to 𝐾 2 (e.g. 𝑥 in the backward process of CDFNet. Note that AR can not only separately anonymize single attribute but also jointly anonymize multiple attributes by recurrently processing them following the above steps, where 𝑠 𝑖=0 𝑥 is the initial input and 𝑘 0 ='identity' corresponds to the first attribute to be anonymized. The final output is the reorganized anonymous latent representation 𝑠 𝑀 𝑥 . Recovery Reorganization. This module focuses on non-sensitive attribute recovery in the latent space. Given keyword 𝑘 𝑖 ∈ 𝐾 2 (𝑀 ≤ 𝑖 < 𝑁 ), we first rely on the forward process of CDFNet to obtain the residual feature 𝑟 𝑖 𝑥 of 𝑠 𝑖 𝑥 and the key feature 𝑓 𝑖 𝑥 of 𝑠 0 𝑥 . Second, 𝑟 𝑖 𝑥 and 𝑓 𝑖 𝑥 are fused to a latent representation 𝑠 𝑖+1 𝑥 in the backward process of CDFNet. Similar to AR, RR can also jointly recover multiple attributes by recurrently processing them following the above steps, where 𝑠 𝑖=𝑀 𝑥 is the initial input. The final output is the reorganized anonymous latent representation 𝑠 𝑁 𝑥 .In this part, we introduce a new Conditional Disentanglement-Fusion Network (CDFNet) which is invertible and can receive bidirectional input. For easy understanding, we separate the network into bidirectional processes according to the data flow: the forward process and the backward process. The two processes share a same network structure as well as their building blocks and parameters, but differ only in the fundamental operations of arithmetic in their building blocks due to the opposite data flow. As shown in Figure 3 , the forward process focuses on feature disentanglement by extracting a key feature 𝑓 𝑖 𝑥 and a residual feature 𝑟 𝑖 𝑥 from an input latent representation 𝑠 𝑥 conditioned on the cross-modal keyword feature 𝑘 𝑓 𝑖 extracted by using the text encoder of CLIP. The backward process focuses on feature recovery by fusing a key feature 𝑓 𝑖 𝑥 and a residual feature 𝑟 𝑖 𝑥 as a new latent representation ŝ𝑥 conditioned on 𝑘 𝑓 𝑖 . To distinguish the two processes, we name the building block of the forward process as conditional disentanglement block (CDB), and the building block of the backward process as conditional fusion block (CFB). The corresponding linear and dense blocks in CDB and CFB share the same parameters, so the reversibility is entirely determined by changing operators.Conditional Disentanglement Block can be understood as the building block of the forward process, which consists of a forward condition block (CB) and a disentangle block (DB). The DB block is borrowed from HiNet [25] for disentanglement. The forward condition block is developed in this paper to process the crossmodal conditions by following the INN rules [2, 10] . As shown in Figure 3 , for the 𝑗-th CDB block in the forward process, the inputs are ℎ 1 𝑗 -1 and ℎ 2 𝑗 -1 , and the outputs ĥ1 𝑗 and ĥ2 𝑗 are formulated as:EQUATIONwhere 𝜙 1 , 𝜙 2 and 𝜙 3 denote different linear blocks. Each one is realized with three fully connected (FC) layers: 𝜙 (𝑎, 𝑏) = 𝐹𝐶 (𝑐𝑜𝑛𝑐𝑎𝑡 ( 𝐹𝐶 (𝑎), 𝐹𝐶 (𝑏))). The DB block is realized in a similar way as:EQUATIONwhere 𝜓 1 , 𝜓 2 and 𝜓 3 denote different densenet blocks [53] .Conditional Fusion Block consists of a backward conditional block and a fusion block, which can be obtained by reversing the data flow of CDB, where the backward conditional block (BCB) corresponds to the forward condition block (FCB) and the fusion block (FB) corresponds to the DB block of CDB. Given a pair of input 𝑔 1 𝑗 -1 and 𝑔 2 𝑗 -1 , the output of FB is formulated as:EQUATIONSimilarly, we formulate the output of BCB as:EQUATIONAfter the inputs go through the sequence of CDB and CFB, the outputs of the last CFB block contains a information loss Δ𝑟 = 𝑔 1 𝑛 and a latent representation ŝ𝑖 𝑥 = 𝑔 2 𝑛 which can be fed into generator G to generate a manipulated face image.We anonymize both the identity and the selected attribute information. Since they have different properties, we employ different strategies to protect them.Group-based Identity Anonymizer (GIA). We impose a strict privacy protection strategy to ensure a better identity protection by using the well defined differential privacy theory [12, 49] . In the pre-processing step, the features in the latent space 𝑆 are clustered into 𝑚 groupsG = {G 1 , G 2 , • • • , G 𝑚 }, where we use Ḡ𝑗 to denote the average feature of the 𝑗-th group. Given an identity feature 𝑓 𝑖 𝑥 to be anonymized, we first utilize the classical exponential mechanism of differential privacy to sample a differentially private group G 𝑢 according to the distance between 𝑓 𝑖 𝑥 the Ḡ𝑗 under the constrained condition 𝑎 ≤ 𝑗 ≤ 𝑏 by considering the privacy and utility tradeoff. Then, in G 𝑢 , we adopt the simple random sampling to choose one identity f 𝑖 𝑥 to replace the original one. Because this process totally happens in the latent space, f 𝑖 𝑥 can regarded as a virtual identity. Since differential privacy is resistant to any form of post-processing [12, 13, 49] , the selection of f 𝑖 𝑥 still follows differential privacy. k-farthest Attribute Anonymizer (KAA). We adopt a simple method to anonymize each attribute. In the pre-processing step, we calculate the average feature as the representative of each sub-category of each attribute. Given an attribute feature 𝑓 𝑖 𝑥 , we anonymize it by using the farthest average feature from the subcategories following the k-anonymity rule [40, 50] .We train our CDFNet in a well defined latent space 𝑆 with the help of several pre-trained models, incuding generator G, latent space encoder MLP, image-text encoder CLIP 𝑐 (•) and identity encoder 𝜌 (•). As shown in Figure 4 , we propose a keyword-based swap training strategy to optimize our network. First, we randomly sample a pair of latent representations 𝑠 1 and 𝑠 2 from 𝑆 in each training step. Then, we alternatively train CDFNet according to the cross-modal keywords 𝑘 𝑖 sampled from 𝐾.Identity-oriented Swap (IOS) training. IOS focuses on improving the identity disentanglement ability by performing identity swap training conditioned on keword 𝑘 0 ='identity'. First, we rely on the forward process of CDFNet to disentangle the key feature 𝑓 𝑘 1 and the residual feature 𝑓 𝑟 1 from 𝑠 1 , which is the same for 𝑓 𝑘 2 and 𝑓 𝑟 2 from 𝑠 2 . Second, we rely on the backward process of CDFNet to swap the key features of 𝑠 1 and 𝑠 2 , resulting in two latent representations 𝑠 ′ 1 and 𝑠 ′ 2 , where Δ𝑟 1 and Δ𝑟 2 are the information loss. By feeding 𝑠 1 , 𝑠 2 , 𝑠 ′ 1 and 𝑠 ′ 2 to generator G separately, we obtain four face images 𝐼 1 , 𝐼 2 , 𝐼 ′ 1 and 𝐼 ′ 2 . Our network is optimized by using:𝐿 𝐼𝑂𝑆 = 𝜆 1 𝐿 𝐼 + 𝜆 2 𝐿 𝑝 + 𝜆 3 𝐿 𝑓 + 𝜆 4 𝐿 𝑟 .(EQUATIONwhere𝐿 𝑝 = E[𝜂 (𝐼 1 , 𝐼 ′ 1 ) + 𝜂 (𝐼 2 , 𝐼 ′ 2 )] is the VGGFace based perceptual loss [6, 26, 42] and where 𝑑 (𝑥, 𝑦) is the cosine feature distance between 𝑥 and 𝑦. The latent representation loss 𝐿 𝑓 is defined asEQUATIONEQUATIONAttribute-oriented swap (AOS) training. AOS focuses on improving the attribute disentanglement ability by performing attribute swap training conditioned on keword 𝑘 𝑖 , 𝑖 > 0. The training process is the same as that of IOS, but the loss function is differentEQUATIONwhere𝐿 𝐼 ′ = E[𝑑 (𝐼 1 , 𝐼 ′ 1 ) + 𝑑 (𝐼 2 , 𝐼 ′ 2 )] is the identity preservation loss. To associate the correspondence between facial attribute with the keyword 𝑘 𝑖 , we formulate the attribute swapping loss 𝐿 𝑠 as an adversarial gameEQUATIONby taking the cross-modal CLIP as discriminator, whereEQUATIONAs shown in Figure 4 , 𝑡 𝑖 𝑗 is a sentence by filling 𝑐 𝑗 ∈ 𝐶 𝑖 to 𝑇 𝑖 (e.g. 𝑡 𝑖 𝑗 = '𝑎 𝑝ℎ𝑜𝑡𝑜 𝑜 𝑓 𝑎 𝑚𝑎𝑙𝑒. ′ ), 𝐶 𝑖 is the fine-grained attribute set of the 𝑘 𝑖 (e.g. 𝐶 𝑖 = '𝑚𝑎𝑙𝑒 ′ , '𝑓 𝑒𝑚𝑎𝑙𝑒 ′ ) and 𝑇 𝑖 is the 𝑖-th sentence template (e.g. 𝑇 𝑖 = '𝑎 𝑝ℎ𝑜𝑡𝑜 𝑜 𝑓 𝑎 {}. ′ ). 
experiments: 
 In this section, we perform quantitative and qualitative experiments to show the effectiveness of the proposed approach. More details and results are presented in the supplementary material.Settings. We use the pre-trained StyleGAN2 [27, 28] and GAN Inversion [51] to build the latent space because it would favor attribute disentanglement, where the latent codes 𝑠 1 and 𝑠 2 in Figure 4 are generated by the MLP layer of StyleGAN. StyleGAN is used as G and GAN inversion is used as the encoder E in Figure 2 . Note that the face images we used in the training process were sampled or synthesized from the latent space of StyleGAN, which would favor anonymization. Table 1 presents some representative attribute Datasets. The CelebA-HQ [33] and LFW [19] datasets are employed for evaluation. CelebA-HQ contains 30,000 face images from 6,216 identities, where 5,000 images are employed as the test set and the remaining are used for training. LFW consists of 13,233 face images from 5,749 individuals, where 5,000 images are used for test. We train our model on CelebA-HQ and evaluate it on all.Baseline Methods. We compare our approach with the following representative and up-to-date methods, including: the classical generative methods CIAGAN and DeepPrivacy (DP1) [21] ; the latest blurry method DarBlur [24] and DeepPrivacy2 (DP2) [22] ; the latent represent methods LDFA [30] and Clip2Protect [47] ; and the differentially private identity disentanglement method Identi-tyDP [54] . Since IdentityDP also employed differential privacy for anonymization in the feature space, we adjust our approach using the same manner as a baseline (denoted as IdentityDP) to show the effectiveness of our group-based identity anonymizer.Evaluation Criteria. We evaluate our approach for privacy protection and utility preservation. For privacy protection, we evaluate the protection success rate (PSR, the higher the better) which is calculated as the percentage of protected faces missclassified by the face recognition tools, where the pre-trained ArcFace [8] and AdaFace [29] models are used. Face alignment [4] is used to detect face and calculate the detection rate (the higher the better). We use Fr échet Inception Distance (FID) [18] to evaluate the image quality (the lower the better). We evaluate the attribute preservation rate (APR) by using pre-trained classifiers, the higher the better.In this part, we show the performance of our approach by comparating with existing methods from different viewpoints.Protection and Preservation. We hope that the anonymized face images can still be detected with a high protection successful rate, which means that a good anonymization method should have high face detection rate and low PSR rate. We first compare our results with the representative basic methods DeepPrivacy [21] and CIAGAN [38] in Table 2 . The detection rates of all the methods reach 100%. The PSR rates of our results are higher than DeepPrivacy and CIAGAN by using both Arcface and FaceNet. The average APR and FID scores of our result also outperform DeepPrivacy and CIAGAN, which stay close to the baseline results of the original data.Comparison with blurry methods. In Table 3 , we compare our method with the tradtional Blurring method and the latest generative DartBlur method [24] . It is obvious that both Blurring and DartBlur have very high PSR rates for Arcface and Adaface, but their attribute preservation rates and FID scores are not ideal. In contrast, our resutls show competitive PSR performances with much better APR and FID scores.Comparison with disentanglement method. In Table 4 , we compare our method with the representative feature disentanglement method IdentityDP which protect face identity by adding Laplace noise following differential privacy. The results show that simply adding Laplace noise may not work effectively on identity protection and may also affect the data utility on APR and FID.Comparison with the latest methods. In Table 5 , we compare our method with the closely related state-of-the-art (SOTA) methods. DP2 [22] works with StyleGAN2 generator, which is the improved version of DP1 [21] . LDFA [30] can be also be seen as the improvmement of DP1 by using latent diffusion model. CLIP2Protect [47] is built based on CLIP and StyleGAN2. Compared with DP1 in Table 2 , DP2 has significant performance improvement on both privacy protection and utility preservation. The excellent FID performance of LDFA can be contributed to the denoise ability of the diffusion models. Both DP2 and LDFA suffer from some performance drop on face detection. The excellent attribute preservation performance of CLIP2Protect can be contributed to the finetuned generator for each input image, but the computational costs for inference is significant high and its identity protection ability is limited. In contrast, our method show the best identity protection performances with limited performance decrease on APR and FID.Image Quality and Diversity. In Figure 5 , we present some representative visual results. DartBlur may easily damage the key contents of face images, leading to significant degraded image quality. LDFA and CIAGAN may suffer from some rectangle effect. Iden-tityDP may suffer some distortions. The results of CLIP2Protect have a high probability to look similar as the input face except regardless of makeup, which cannot visually protect the face. DP1, DP2 and Ours show comparable good visual image quality and they show significant differences with the original input data. Since our method has no additonal operations on backgound in the latent space, it cannot well preserve the original background, but it can be recovered by employing anothter recovery step following [38] . In Figure 6 , we show that LROrg can also generate diverse anonymization results by default, which can be contributed to the random mechanism used in our group-based identity anonymizer. One can observe that the generated faces are anonymizd and look different from each other. Transfer Capability. We show the transfer capability of our method on the LFW dataset by using the pre-trained model on CelebA-HQ. We report the results in Table 6 . It is obvious that Our method show consistent results as with that in previous experiments, which again reflects the superior performance of our method. The latent information loss of the StyleGAN latent space would decrease the performance of our FID score, which is a limitation of our method. Since all the evaluation can achieve almost 100% face detection rate, we no longer report them next.Personalized Protection. We regard identity as one attribute so that users can flexibly choose which attribute to preserve or not according to the practical applications. In Figure 7 , we present several examples of our personalized fine-grained anonymization process by removing or preserving some attribute according to the keyword conditions. It is obvious that our method can flexibly anonymize and recover the given attributes according to the keyword conditions, resulting in realistic and desired face images. In Table 7 , we report the quantitative evaluation results of protecting the identity and one user-defined attribute, such as ID+Gender. One can observe that, compared with our default settings, the joint protection strategy can further improve the PSR performance, but the data utility may suffer from different extent of drops. Without recovering the attributes, (b) suffer from lower APR and FID scores. Reversibility. Since LROrg is built based on cINN [2] , the protection is theoretically reversible. In Figure 9 , we demonstrate the anonymization-recovery process for several representative attributes. The results show that our approach can well recover the anonymized attributes. In Table . 8, we report the recovery rate of different facial attributes after anonymization. The identity recovery ability suffer from significant drops. The reason may lie in the information loss during the disentangle-fusion process of our CDFNet as well as the information loss between StyleGAN and GAN Inversion, which is a limitation of our method to be addressed in the follow up works.In this part, we conduct ablation studies to show the ability of our framework by varying the configurations.Anonymization Strategy. For identity anonymization, we compare our differential privacy based method with that of the kanonymity strategy [50] by using the farthest group center for identity anonymization to ensure a good protection. The results in Table 9 show that our method not only has significant advantage over the k-anonymity strategy on APR but also outperforms k-anonymity on PSR. Figure 10 visually illustrates that the k-anonymity may suffer from more utility drops than ours, like age. Impact of Non-Identity Anonymization. As shown in Figure 11 , we also wonder the impact of non-identity attribute anonymization by processing one attribute each time without recovery. Table 10 reports the result. One can observe that anonymizing gender can produce much higher PSR scores than processing age, expression and makeup, which indicates that gender share more correlations with the face identity, which would makes it harder to preserve the gender attribute in privacy protection. And changing the makeup attribute may have the least impacts on identity protection.Latent Feature Distributions. We studied the features disentangled by our CDFNet by clustering them into different groups. According to the plots shown in the Figure 12 , we can observe that our disentangled features show good clustering performances for different kinds of attributes. This reveals that our model has good representation ability. 
conclusion: 
 These years, the issue of face privacy protection has received increasingly attentions. In this paper, we present a keyword conditioned LROrg framework for fine-grained face privacy protection. On top of extensive experiments, we have verified the state-of-theart performances of LROrg on achieving a better privacy-utility tradeoff, where the protection ability can achive further improvement by flexibly manipulating which attribute to anonymize or preserve according to practical requirements. We also find that gender is closely correlated with face identity which may inspire follow up works in aonymization. In comparison with previous methods, our solution is more flexible and effective by working in a distinct recurrent manner.Although LROrg does not rely on data annotations, it suffers from the problem of incomplete latent space which is built by using StyleGAN and GAN Inversion. This would lead to the problem of information loss and further affect image synthesis. Besides, our CDFNet model may also suffer from the same problem, which would limit its performance. We will explore to address the problems in the future work. 
 
 Appendix:  


Decision: This paper presents a generative framework that protects the identity and sensitive attributes of face images. The proposed method employs a two-stage recurrent grafting to de-identify and recover chosen attributes. Experimental results show the method offers a desirable level of privacy protection while retaining task utility.

Initially, this paper receives mixed reviews with 2 BR and 2 BA. The authors have well addressed the reviewer concerns. After the rebuttal, all the reviewers give BA score. After carefully checking the  reviewer comments, this work has an interesting method of solving problems, and the experiment is adequate. 

I would like to accept this paper. The authors are encouraged to carefully prepare the camera ready version according to the reviewer recommendations.

Reviews: review: paper_summary: paper_summary: The paper proposes a face anonymizing technique based on latent vector manipulation of the style transfer model. The method enables inputting the face attributes and dividing them as sensitive and non-sensitive. the sensitive attributes anonymization is carried out. Many experiments are carried out to show the face quality generation and the possibility
strength_weakness: strength_weakness: - many experiments were done to evaluate face generation and privacy protection. This effort is appreciated.
- The idea of dividing face attributes into sensitive and non-sensitive provides flexibility in face anonymization.
limitations: limitations: - The method proposed is performing face attribute editing. This is a well-explored area with numerous existing works [1, 2], and the technical novelty of the proposed method appears limited. 
- I don't understand the need for using CLIP. Why is it necessary to use CLIP to encode keywords instead of directly using attribute indices as values and embedding them into vectors?
- The criteria for distinguishing between sensitive and non-sensitive attributes are not clearly defined. Providing a detailed explanation of this selection process would enhance the paper's clarity.
- The application of group differential privacy is not sufficiently explained. A more thorough description of its implementation is necessary to understand its role and effectiveness in the proposed method.


- Other comments: 
1) The method presentation is poor. Fig. 2, 3 and 4 are hard to understand, and the captions do not provide necessary information.
2) The writing of the paper is not the best. It can be improved.


[1] Qiu, Haonan, et al. "Semanticadv: Generating adversarial examples via attribute-conditioned image editing." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020.
[2] Ding, Zheng, et al. "Diffusionrig: Learning personalized priors for facial appearance editing." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
paper_summary: paper_summary: This paper presents a generative framework that protects the identity and sensitive attributes of face images. The proposed method employs a two-stage recurrent grafting to de-identify and recover chosen attributes. Experimental results show the method offers a desirable level of privacy protection while retaining task utility.
strength_weakness: strength_weakness: - Interesting research topic: Face protection is an important topic and is worth further exploration. 
- The proposed method is well-grounded and achieves its protection goal. 
- The experiments are extensive.
limitations: limitations: It is a coincidence that I reviewed the same paper at ACM MM last year (it was titled "ReGraft: A Recurrent Grafting Framework for Personalized Face Privacy Protection"). I liked this paper overall albeit some minor performance issues and my last year's final rating was borderline acceptance. It regretfully wasn't finally accepted. I am delighted to see this paper is further polished, and the authors have incorporated some of my previous suggestions (e.g., comparison with diffusion-based methods, further analysis on facial attribute protection, and additional coverage of related works). I am not going to raise further concerns as they are mostly already addressed through the authors' revision. I keep my last year's rating.
paper_summary: paper_summary: This paper proposes a two-stage latent representation reorganization (LReOrg) framework to protect face privacy. The sensitive information is anonymized in the first stage, and attribute information is recovered according to usability requirements. LReOrg provides flexible control by manipulating which attributes to anonymize or preserve. Specifically, this paper constructs a novel Conditional Disentanglement Fusion Network (CDFNet) to separate sensitive information for protection based on keywords. Different anonymization strategies are employed for identity and attributes: Group-based Identity Anonymizer (GIA) and k-farthest Attribute Anonymizer (KAA).
strength_weakness: strength_weakness: Fine-grained attributes control is crucial in the field of privacy protection. This paper is innovative, both in its ideas and its technologies. In many details, this paper exhibits unique insights. Adequate experimental have validated the effectiveness of the method.
limitations: limitations: 1. The Group-based Identity Anonymizer (GIA) employs a differential privacy strategy based on the exponential mechanism. However, this strategy does not satisfy the post-processing property of differential privacy during the recovery reorganization stage. The premise of post-processing is that it does not involve original data. However, in the recovery stage, the use of $s_x^0$ extracted from the original data results in excessive consumption of privacy budget.
2. CLIP provide an interpretation of the image, but their interpretability for facial attributes is limited due to the lack of specialized training. Using standard CLIP as a discriminator in Eq.10 is not a good choice.
3. Is the training process of CDFNet randomly selecting $k_i$ (identity and attributes) for training in each epoch? If so, would this lead to the distance of CLIP being influenced by identity? Specifically, at a certain point, when the network already possesses identity-preserving capabilities, $I_2'$ preserves the identity of $I_1$, and $I_1'$ preserves the identity of $I_2$. CLIP erroneously believes it has preserved the attribute ('gender') in attribute-oriented swap (AOS) training due to the correlation between identity and attributes.
4. It is suggested to clarify that $D(·)$ in Eq. 10 denotes the CLIP discriminator, representing the similarity between images and text.
5. Is Figure 7 a step-by-step process, where the fourth column changes gender based on the third column, and the fifth column changes expression based on the fourth column? If so, it is advisable to add explanations in the paper.
6. The identity recovery ability suffers from significant drops. The reason might be that there are too many identity categories while attributes are only a few.
paper_summary: paper_summary: This paper presents a latent reorganization framework for flexible face privacy protection, which can control which attributes to anonymize or preserve. A CDFNet is designed to  recurrently support sensitive attributes anonymization and non-sensitive attributes recovery.   A keyword-based swap training strategy is introduced by using the CLIP model and a multi-task loss.  The experimental results on benchmark datasets
 reported the superior ability of the method to provide flexible protection.
strength_weakness: strength_weakness: The research problem is meaningful and important.

Technological innovation is sufficient, which contains  a new network and a new training strategy.

The experiment is rich, which incorporates video presentations and feature visualisations
limitations: limitations: 1. **De-identification does not work perfectly in this paper.** De-identification is the most demanded function in real-world applications, but the method in this paper does not allow for better performance in attribute preservation. As shown in Fig. 1 and Fig. 5, **other attributes of the face image are changed** when the identity is modified, such as background, hairstyle, and so on. However, existing advanced de-identification methods [1-2] have been able to preserve these attributes and only modify the identity region.  **Although this method supports the attribute recovery process, it is laborious for the user.** The user need to observe the protection results and explicitly indicate the retention of multiple attributes.  More unfortunately, the face images has more than 40 properties that are hard to list.

      [1] 2021ICCV-Personalized and Invertible Face De-Identification by Disentangled Identity Information Manipulation

      [2] 2020ECCV-Password-conditioned Anonymization and Deanonymization with Face Identity Transformers


2. **Why the keywords were used was not clearly explained.** What are the advantages of using text keywords over binary labels.  On my understanding, the keywords just go to guide the network to decouple the corresponding features, but simple numeric labels can go to do that as well. For example, using 10000 for identity change and 0001 for gender change. Furthermore, the paper does not compare with existing methods that use the label de-identification [3].

      [3] 2021TCSVT-The UU-Net_ Reversible Face De-Identification for Visual Surveillance Video Footage

3. **The effectiveness of flexible protection is doubtful.** 

    **(1) Failure to achieve identity-preserving attribute protection.**  Identity-preserving attribute protection is also a mainstream class of face privacy protection methods. According to the description of this paper, the proposed method should be applicable to such methods as well, i.e., only anonymize the attributes and recover the identity.  However, this paper does not validate it, and also according to the description in Table 8, the method in this paper does not seem to be able to recover identity.

     [4]2020TIP-PrivacyNet: semi-adversarial networks for multi-attribute face privacy

     [5]2023TIFS-RAPP: Reversible privacy preservation for various face attributes;

   **(2) Restoring other attributes may affect the already anonymized attribute.** Correlations between attributes are not considered in this paper. In fact, when the gender attribute is restored, there is a high probability that it will affect previously anonymised attributes, which include 'Makeup', 'Color', 'Curly'. This is because these attributes are extremely correlated with gender.

4. **Unreasonable selection of baseline methods and inadequate development of flexible protection experiments.**  （1）The goal of CLIP2Protect is to generate adversarial faces (with insignificant changes), whereas this paper directly modifies identities, which is not comparable. （2）Darblur uses a blurring method with the aim of generating blurred results, which is still not comparable to the method in this paper.  （3）Some of the advanced methods of face privacy protection [6,7]  can go for comparison.

     The experimental section describes flexible (personalized) protection using only a small amount of content, which is unreasonable. The de-identification performance in comparison with other methods should be scaled down and the validation of flexible protection should be increased. 

     [6] 2021TCSVT-The UU-Net_ Reversible Face De-Identification for Visual Surveillance Video Footage

     [7]2021ICCVPersonalized and invertible face de-identification by disentangled identity information manipulation
5.  **Protected faces in the real world can be displayed.** Overall, the work in this paper is more of a new method for face manipulation. It is also crucial for face privacy protection that the method can be effective in the real world. Therefore, adding real-world face image protected results can enhance the persuasiveness of this paper.