Hunting Blemishes: Language-guided High-fidelity Face Retouching Transformer with Limited Paired Data

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 16:35:21

Abstract: The prevalence of multimedia applications has led to increased concerns and demand for auto face retouching. Face retouching aims to enhance portrait quality by removing blemishes. However, the existing auto-retouching methods rely heavily on a large amount of paired training samples, and perform less satisfactorily when handling complex and unusual blemishes. To address this issue, we propose a Language-guided Blemish Removal Transformer for automatically retouching face images, while at the same time reducing the dependency of the model on paired training data. Our model is referred to as LangBRT, which leverages vision-language pre-training for precise facial blemish removal. Specifically, we design a text-prompted blemish detection module that indicates the regions to be edited. The priors not only enable the transformer network to handle specific blemishes in certain areas, but also reduce the reliance on retouching training data. Further, we adopt a target-aware cross attention mechanism, such that the blemish regions are edited accurately while at the same time maintaining the normal skin regions unchanged. Finally, we adopt a regularization approach to encourage the semantic consistency between the synthesized image and the text description of the desired retouching outcome. Extensive experiments are performed to demonstrate the superior performance of LangBRT over competing auto-retouching methods in terms of dependency on training data, blemish detection accuracy and synthesis quality.

Main paper: 
 introduction: 
 The rapid development of social media leads to the fast-growing demand for automatic face retouching in various scenarios, including portrait photos, film and television productions, and so on. The primary objective of face retouching is to achieve natural-looking and realistic results, which maintains crucial characteristics while at the same time eliminating blemishes such as dark circles, acne scars and wrinkles [31, 46] . However, this is still a challenging task due to variations in lighting conditions, skin tones, and the complex nature of blemishes themselves.Different from generic face enhancement tasks, there are typically a small percentage of image pixels that need to be edited in face retouching, and the existing methods perform less satisfactorily due to lack of effective distinction between blemishes and normal skin. On the other hand, face retouching methods are built upon the observation that normal skin exhibits local smoothness. Many attempts have been made to design effective smoothing filters [1] to remove blemishes by leveraging the contextual information surrounding them. To handle diverse blemishes, deep convolutional neural networks are applied to learn the mapping from blemishes to normal skin [20, 30, 31, 44] . Further, generative models are trained to synthesize clean face images, conditioned on the ones with blemishes [14, 46] . These methods are trained on specific paired training data, thus limiting their generalization performance across difference domains where the appearance of blemishes and skin features, such as blemishes on different skin can vary significantly. Considering that the large-scale vision-language pre-training, such as CLIP [28] , has strong capability of zero/few-shot object recognition, We perform an effective attempt for language-guided face retouching, and the resulting model can well generalize to diverse types of blemishes.More specifically, we propose a Language-guided Blemish Removal Transformer (LangBRT) to facilitate face retouching. The key idea behind LangBRT is to perform textual prompt-conditional blemish detection and thus spatially regularize the cross-attention computation in transformer blocks to remove blemishes in a feature space, consequently generating a realistically retouched image corresponding to the prompt, as shown in Figure 1 . To achieve this, we adopt the Contrastive Language-Image Pre-training model (CLIP) [28] to associate natural language with image content, and incorporate a Text-prompted Blemish Detection module (TBD), since a textual description can be used to effectively express rich visual concepts. TBD learns to perform pixel-wise recognition from the encoder features of an input face image. The prior knowledge encapsulated in CLIP enables TBD to distinguish blemishes from normal skin. On the other hand, We find that the prior is also useful for reducing the reliance of our model on paired training data. By injecting the resulting blemish feature maps as side information into the transformer, we can perform target-aware cross-attention computation, which aims to edit the blemish-like regions. We further impose the semantic consistency regularization on the synthesized images, given the textual description of the desired retouching outcome. Extensive experiments on both public and in-the-wild data are performed to verify the effectiveness of the design elements as well as the superior performance over state-of-the-art face retouching methods. In summary, the main contributions of this work are as follows:• Different from the existing face retouching methods adopting generic image-to-image translation frameworks, the proposed LangBRT has a language-guided transformer architecture with target-aware cross-attention computation. • Blemish detection is conditioned on the textual descriptions, which enable a wider range of blemish types to be effectively handled. Another benefit is to effectively reduce the dependency of LangBRT on paired training data. • By injecting the blemish features into transformer blocks, the main feature transformations are limited in the blemishlike regions, which leads to precise retouching results. 
related work 2.1 image-to-image translation: 
 Image-to-image translation can be viewed as a special case of conditional image generation, and there have been many attempts to facilitate this task. Given paired training data, a typical strategy is to train a convolutional neural network by minimizing a variety of regularization functions between the synthesized images and the ground truth [6, 16, 18, 49, 51] . In [6] , a pixel-wise consistency loss to the ground truth was used for training an image translation network. To ensure semantic correctness, Johnson et al. [16] employed a pre-trained VGG-19 network [34] to measure the perceptual consistency, which leads to a better alignment with human perception in terms of image semantics. Similarly, Zhang et al. [49] proposed the Learnt Perceptual Image Patch Similarity (LPIPS) measure, which had been widely used in various image-toimage translation tasks. However, in many real-world scenarios, it is expensive and infeasible to collect a large amount of paired training data. Unsupervised image translation methods, like Cy-cleGAN [51] and DiscoGAN [18] , achieved impressive generation performance by minimizing the reconstruction loss of two-way mapping. By combining GAN [13] and VAE [19] , UNIT [23] learnt to disentangle style and content in feature spaces, such that the images from different domains can be transferred by exchanging the style features.To control the synthesis content, an additional attribute classifier was incorporated to guide the generation process by measuring the semantics encapsulated in the synthesized images [12] . StarGAN [7, 8] learnt a generation network to realize efficient cross-domain transformation, conditioned on domain label. Another effective way is to inject constraint information into the generation network, such as edges [15] , sketches [2] and label maps [43] . Since the latent space of a well-trained StyleGAN [17] has a semantically meaningful organization, image-to-image translation can be performed by projecting a source image back into the latent space and learning a task-specific latent transformation. InterfaceGAN [32] was proposed to discover global latent directions associated with a number of pre-set attributes. To handle unlabeled data, Shen et al. [33] performed factorization on the weights of the generation network and found a set of latent directions associated with well-defined attributes.Motivated by the great success of the transformer architectures in natural language processing [39] , researchers have explored diverse applications of transformer in the computer vision area [10, 24, 38, 45] . The important characteristics of transformer lie in its attention mechanisms, which enable effective modeling of inherent relationships within sequences. In particular, ViT [10] extended the transformer's success to visual tasks. To further enhance the ViT's capabilities in handling high resolution images, Wu et al. [45] proposed a convolutional vision transformer, which incorporated convolutional layers into transformer blocks. On the other hand, SwinTransformer [24] adopted the shifted windowing scheme, which limited attention calculation to non-overlapping windows while allowing cross-window connection. Transformer architectures were also successfully applied to object detection, such as DETR and variants [4, 52] . Inspired by DETR, Wang et al. [41] proposed an end-to-end segmentation transformer to directly predict masks with class labels. Contrastive Language-Image Pre-Training (CLIP) [28] was designed to understand and associate images and textual descriptions, and had gained significant attention due to its versatility and effectiveness. The CLIP's capability of performing cross-modal understanding together with transformers had been widely used in various applications, such as image generation [29] , image classification [5] , retrieval [25] , semantic segmentation [21] , video caption [36] , video action recognition [42] and object localization [9] . To apply CLIP on downstream tasks, Gao et al. proposed CLIP-Adapter [11] to conduct fine-tuning with feature adapters on either visual or language branch. Furthermore, Sun et al. [35] proposed Alpha-CLIP to enhance CLIP with an auxiliary alpha channel to suggest attentive regions, which enables Alpha-CLIP to focus more on the regions of interest. The objective of face retouching is to enhance the appearance of input images while preserving the key facial characteristics. The traditional methods, like nonlinear digital filtering [1] , applied a uniform operation to address different types of flaws. In [3] , Batool et al. detected facial wrinkles and imperfections using Gabor filters. In addition, Velusamy et al. [40] proposed a wavelet band manipulation method to restore the underlying skin texture. However, these methods lacked adaptive retouching capabilities. To address this issue, Lipowezky et al. [22] performed freckle detection and retouching separately. Based on the concept of facial attractiveness, the face retouching process could be guided by an aesthetic enhancement model [37] . Recently, AutoRetouch [31] was an effective attempt to perform end-to-end face retouching. In addition, Zamir et al. [48] proposed a multi-stage approach to progressively restore spatial details and high-level contextualized information. These methods primarily focus on global retouching while neglecting the importance of the local region. Instead, ABPN [20] performed fast local retouching on high-resolution photos through an adaptive blend pyramid network. To guide precise blemish removal while preserving the semantic information of an input image, Hong et al. incorporated a pre-trained face parsing model in HQRetouch [14] . In contrast, BPFRe [46] was a multi-stage approach for face retouching, which divided the retouching process into blemish detection, retouching and refinement phases, and adopted different strategies to utilize unpaired training data to regularize each stage.The main differences between our proposed LangBRT and the above existing face retouching methods are summarized as follows: (1) LangBRT facilitates face retouching by utilizing textual descriptions of blemishes, and it is the first attempt to leverage vision-language pre-training for the task. LangBRT is able to address diverse blemish types, and allows user-defined retouching. This has not been considered by the above methods. (2) Different from the existing methods [14, 20, 46] which directly suppressed features of blemish-like regions, LangBRT limited main feature transformations in blemish-like regions via target-aware cross attention, which ensures precise blemish removal.  
methodology 3.1 overview: 
 It is promising to integrate the description of blemishes together with desired retouching outcome into our image editing process. As shown in Figure 2 , the proposed framework mainly consists of five components, including an encoder 𝐸 extracting the features from a source image, a latent transformer 𝑇 performing feature transformation, a decoder 𝐷 generating a clean face image, a discriminator 𝑆 distinguishing manually retouched images from the synthesized ones, and a Text-prompted Blemish Detection module (TBD). Given the blemish descriptions involving dark circle, acne, wrinkle and so on in specific area, TBD leverages a pre-trained vision-language model to obtain the textual prompts together with an image encoder to extract features from a source image, and produces the corresponding maps to indicate the blemishes associated with the descriptions, respectively. Further, we inject the blemish information into the transformer blocks in 𝑇 via Target-specific Cross-Attention mechanism (TCA) to limit the main feature transformations in the blemish-like regions, and the transformer blocks are guided to progressively restore clean skin in the regions.In LangBRT, prompt is defined as the combination of blemish type and its corresponding location. To streamline model training, we delineate two categories of prompts: (1) The first category is predefined prompt, such as 'dark circles under the eyes', 'forehead wrinkles', 'acne on the right cheek'. Pre-defined prompts encapsulate common blemishes and cater to the general needs of face retouching. To enhance the localization accuracy of these blemishes across diverse inputs, we incorporate dedicated mapping layers for each pre-defined prompt. (2) The second category is user-defined prompt, which is tailored to accommodate users' personalized preferences for precision retouching. Given the variability in refinement requisites among users, these prompts provide users with added control over the retouching process. Let [𝒑𝒕 1 , 𝒑𝒕 2 , . . . , 𝒑𝒕 𝑃 ] denote a set of pre-defined prompts, and [𝒖𝒕 1 , 𝒖𝒕 2 , . . . , 𝒖𝒕 𝑈 ] denote a set of user-defined prompts. We adopt the CLIP text encoder 𝐸 𝑇 𝑒𝑥𝑡 of the pre-trained Alpha-CLIP [35] to generate corresponding embeddings [𝑭 𝒑𝒕 1 , 𝑭 𝒑𝒕 2 , . . . , 𝑭 𝒑𝒕 𝑃 ] and [𝑭 𝒖𝒕 1 , 𝑭 𝒖𝒕 2 , . . . , 𝑭 𝒖𝒕 𝑈 ]. Let the mapping layer 𝜓 𝒑𝒕 𝑖 to learn the precise text features corresponding to the pre-defined prompt. Let 𝒙 denote a source image, which is passed through the image encoder 𝐸 to extract the feature 𝑭 (0) 𝒙 . The manually retouched image denoted as 𝒚 serves as the ground truth. To detect the blemishes associated with the specified prompts, 𝐸 is encouraged to capture the blemish information from the image, and the detection map 𝒎 is derived by measuring the relevancy between the image features and each prompt as follows:EQUATIONwhere ⊙ represents the dot product operation, E denotes the operation to expand the dimensionality of the text embedding to match that of the image feature, and 𝑭 𝒙 ⊙ E (𝑭 𝒖𝒕 𝑗 ) refers to the blemish map associated with the 𝑗-th user-defined prompt 𝒖𝒕 𝑗 , 𝒙 ⊙𝜓 𝒑𝒕 𝑖 (E (𝑭 𝒑𝒕 𝑖 )) refers to the blemish map associated with the 𝑖-th pre-defined prompt 𝒑𝒕 𝑖 , U and P represents the number of userdefined prompt and pre-defined prompt, respectively. We consider that the CLIP embedding space encapsulates rich knowledge on blemishes, and the prompt is an effective representation to retrieve the useful priors for our detection task. This design also helps to reduce the dependency on large amounts of manually retouched data.We realize blemish removal by progressively replacing the blemishes with the synthesized content. Toward this end, we adopt a target-specific cross attention mechanism in each transformer block. Different from generic self-attention computation over all pixel positions, our mechanism limits the main feature transformations in blemish-like regions, while at the same time ensuring that the features in normal skin regions remain unchanged.The blemish detection maps provide the spatial information on the regions to be edited. In each transformer block, the maps play two different roles in constructing the query and key-value vectors for attention computation. Specifically, the image features contain crucial visual details and patterns, and serve as the input of query mapping. The maps are used as the weight to suppress the features in the blemish-like regions. On the other hand, we directly learn key and value vectors from the maps, such that the blemish-like regions will be filled with the content synthesized from scratch. Since the features of blemishes are continuously discarded in the forward process, the learnt key and value vectors are associated with the features of normal skin to ensure clean face synthesis. Formally, we define the cross attention computation in the 𝑖-th transformer block as follows: where 𝑭EQUATIONEQUATIONEQUATION(𝑖 )𝒙 denotes the image features in the 𝑖-th transformer block for 𝑖 = 1, 2, . . . , 𝑁 , {𝑸 (𝑖 ) , 𝑲 (𝑖 ) , 𝑽 (𝑖 ) } represent the query, key and value features,{𝜑 (𝑖 ) q , 𝜑 (𝑖 ) k , 𝜑 (𝑖 )v } are the corresponding linear mapping functions, 𝜙 is the sigmoid activation function, and 𝛼 is a learnable scaling parameter to control the magnitude of the multiplication result. To reduce the computational complexity, we adopt the sliding window strategy of SwinTransformer [24] . The final output of the transformer is fed into the decoder to synthesize a retouched image denoted as ŷ.The training goal of the proposed LangBRT consists of two aspects: precise blemish detection and high-fidelity retouched image synthesis. Due to the lack of blemish annotations, we compute the difference map between each pair of raw image and manually retouched one, which is used as the ground truth of blemish detection. Conditioned on the textual prompts, our detection model produces a set of detection maps, and their combination is required to be consistent with the ground truth. The detection loss function can be defined as follows:EQUATIONwhere 𝜏 is an activation function to normalize the difference map. Minimizing L 𝑑𝑒𝑡𝑐 encourages the encoder to capture the information on blemishes. Furthermore, for the blemishes in the particular domain, we roughly segment the face based on existing face parsing model [50] , as an auxiliary method for the model to refine the region-specific retouching. We obtained coarse segmentation of facial regions such as the forehead, left cheek, and periocular area. Subsequently, we applied component-aware consistency loss exclusively to these regions' blemishes, which is defined as follows:EQUATIONwhere 𝑚 𝑝𝑡 𝑖 refers to the mask corresponding to pre-defined prompt 𝑝𝑡 𝑖 , P is the number of pre-defined prompt.To ensure high-fidelity image synthesis, we adopt an adversarial training approach to optimize the constituent networks. The synthesized image ŷ is expected to be identified as a manually retouched one, and the discriminator aims to identify them as accurately as possible. We define the adversarial training loss functions as follows:EQUATIONEQUATIONwhere 𝑆 (•) denotes the probability of an input image being retouched manually.Considering that deceiving the discriminator cannot guarantee the retouching quality, we further measure the pixel-wise and perceptual consistency between the synthesized result ŷ and the manually retouching image 𝒚, and the corresponding loss function is defined as follows:EQUATIONwhere || * || 𝑝 represents ℓ 𝑝 norm, Φ 𝑙 denotes the features associated with the 𝑙-th layer of a pre-trained VGG-19 [34] network, and 𝛽 is the weighting factor to balance the two types of consistency measurements.In addition to leveraging the textual prompts for blemish detection, we can also use the textual descriptions of desired retouch outcomes to regularize the generation process by measuring the semantic similarity between the synthesized results, specific region and the descriptions in the Alpha-CLIP [35] embedding space. For simplicity, we still use the blemish descriptions and train the model by maximizing the prior-based dissimilarity to the synthesized results as follows:EQUATIONwhere 𝜚 𝛼𝐶𝐿𝐼 𝑃 is pre-trained Alpha-CLIP.As shown in Figure 3 , the prior-based dissimilarity loss function is useful for guiding the generation process.By integrating the above training loss functions, the optimization formulation of LangBRT can be expressed as follows: min 𝐸,𝑇 ,𝐷EQUATIONwhere 𝛾 and 𝜂 are the weighting factors to achieve a trade-off among the regularization terms. Note that the constituent networks are jointly optimized from scratch. The training procedure is summarized in Appendix. 
experiments: 
 In this section, extensive experiments are performed to assess the retouching performance of the proposed LangBRT on both public and in-the-wild data. We first introduce the training and test data, implementation details, and evaluation protocol. Next, we compare LangBRT with state-of-the-art face image editing methods. Finally, we perform ablation study to verify the effectiveness of the main components. The main experiments are performed on the FFHQR dataset [31] , which is the first large-scale public dataset created through professional retouching techniques. It contains 70,000 pairs of "Before" and "After" retouched images with the resolution of 1024*1024, involving various facial characteristics such as age and race. The dataset is partitioned into a training, validation, and test set, containing 56,000, 7,000, and 7,000 images respectively. We follow the setting [31] , in which both training and test images are resized to 512*512. Furthermore, the proposed method is also evaluated on the FaceRetouch-wild (FR-wild) dataset, which contains 700 in-the-wild face images with a large diversity of poses, races, and blemish types.4.1.2 Implementation Details. In LangBRT, the latent transformer contains 7 blocks, and the configurations are the same as Swin-Transformer [24] . The architecture information of the transformer together with the other constituent networks are provided in Appendix. We implement LangBRT using PyTorch with a NVIDIA GeForce RTX 3090. There are a total of 25,000 training iterations with a batch size of 1. The number of user-defined prompt and pre-defined prompt, U and P in Eq.( 1) is set to 17 and 3, respectively. The weighting factors: 𝛽 in Eq.( 9) and {𝛾, 𝜂} in Eq.( 11) are set to 10, 1 and 1, respectively. Our model is optimized through Adam, and the learning rate is initialized as 0.0002 and modified according to a cosine decay schedule.We compare the proposed LangBRT with a number of representative face image editing methods, including Pix2PixHD [43] , GPEN [47] , SwinTransformer [24] , MPRNet [48] , RestoreFormer, [44] , AutoRetouch [31] , ABPN [20] , and BPFRe [46] . Note that SwinTransformer serves as the base model of our LangBRT. Pix2PixHD is a typical image-to-image translation method. MPRNet and RestoreFormer (GPEN) are designed for (face) image restoration. AutoRetouch, ABPN and BPFRe focus on face retouching. We follow the settings of BPFRe to perform quantitative evaluation in terms of Peak Signalto-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Learnt Perceptual Image Patch Similarity (LPIPS). We implement all the competing methods using the open source codes, and they are trained on the same data as our LangBRT for a fair comparison. In addition to using the full training data, we also randomly sample 1%, 5%, 10%, and 20% of the training data to evaluate the performance stability of the competing method in the situations of limited training data. The results are summarized in Table 1 . We can observe that LangBRT consistently outperforms the competing methods in terms of all the metrics. When using only 1% of the training data, the advantage of LangBRT becomes more significant. In particular, LangBRT achieves the PSNR score of 44.51 dB, which is higher than that of BPFRe by about 1.32 dB.The SSIM and LPIPS scores of LangBRT are better than the second best methods: BPFRe, by about 0.41 and 0.16 percentage points. In addition, we additionally adopted the structure of the Temporary Patch GAN model [27] , trained a classifier for source and ground truth images of FFHQR dataset, and visualized the features of the intermediate layer using PCA [26] , as shown in Figure 4 . Notably, it can be seen that the refined results of LangBRT are similar to the ground truth, but there are significant differences from the source images, which also proves the effectiveness of LangBRT.We consider that our model benefits from the vision-language pretraining, and thus has a lower dependence on the training data than the competing methods. This is also confirmed by the representative retouching results shown in Figure 5 .According to the above quantitative comparison result, BPFRe performs better than the other competing methods. To highlight the LangBRT's capability of handling diverse blemishes, we further compare with the state-of-the-art face retouching method BPFRe in Figure 6 . One can find that LangBRT is able to remove dark circles and acne, reduce reflections, smooth skin, while preserving the original tone. In contrast, BPFRe performs less satisfactorily, and the blemishes are only partially removed. This result suggests that Lang-BRT has better generalization performance in real-world scenarios.We further perform user study to assess the retouching performance of the methods in human perception. We randomly sample 30 face images from the in-the-wild data, and ask 50 volunteers to rank the synthesized results of LangBRT and the competing methods.All the models are trained on 1% of the training data in FFHQR. We provide a visual representation of the comparative preferences expressed by the participants in Figure 7 . The result suggests that LangBRT receives the most votes as the best retouching method, and this is consistent with the results obtained from the previous experiments, further validating the superiority of LangBRT in our task.To investigate the contribution of the main components in LangBRT, we build a number of variants using 1% of the training data and perform ablative experiments in this subsection. ( 1) " LangBRT w/o Prior": the vision-language pre-training is not used for blemish detection and semantic regularization. 2 shows that the removal of the vision-language pre-training results in a significant increase in LPIPS by over 9 times. We consider that the pre-trained model plays an important role in providing useful priors of blemishes in the limited data case. Without accurate blemish detection or targetspecific cross attention, " LangBRT w/o TBD" or " LangBRT w/o TCA" cannot limit the main feature transformations in blemishlike regions, such that the information from non-blemish regions cannot be effectively utilized for filling the blemish-like regions, and the retouching performance thus becomes less satisfactory as shown in Figure 8 . In addition, we confirm that L 𝑙𝑜𝑐𝑎𝑙 is useful for boosting the performance by 0.72dB in terms of PSNR (Table 2 ). We plot the PSNR curve of the model with and without L 𝑙𝑜𝑐𝑎𝑙 during the training propose, and find that the loss function consistently leads to higher PSNR values(Figure 9 ). Furthermore, we visualize the feature changes of representative images before and after processed by Transformer𝑇 , the results shown in Figure 10 suggest that L 𝑙𝑜𝑐𝑎𝑙 enhances the transformer's precision in refining defect areas.LangBRT has the capability of detecting and removing the blemishes associated with the user-defined textual descriptions. To illustrate the effectiveness of the text-prompted blemish detection module, we visualize the detection maps involving different prompts in Figure 11 . It can be observed that the module produces different detection maps when using the prompts: "dark circle", "forehead wrinkles" and "pimple on the right cheek". Furthermore, LangBRT is able to remove the blemishing accordingly. In Figure 12 , we compare with BPFRe and human retouchers, and find that the detection maps of our model are more consistent with the manual results. 
conclusion: 
 This paper presents a text-driven latent transformer for precise facial blemish detection and removal. To handle diverse blemish types, we incorporate the vision-language pre-training and leverage the prior knowledge encapsulated in the embedding space. By providing the textual prompts of blemishes, we design an effective detection module to measure the association between the encoder features and textual prompts, and produce the maps to highlight the spatial information of specific blemishes. This design not only improves the generalization performance on a wide range of blemishes, but also reduces the dependence of the model on the paired training data. To precisely remove blemishes while preserving non-blemish content, we further inject the blemish map into each transformer block to perform target-aware attention computation. In the forward process, the features of blemish-like regions are replaced with the synthesis content progressively. Extensive experiments demonstrate the superiority of the proposed method over the state-of-the-arts, especially in the case where a limited amount of paired training data is available.  
 
 Appendix:  


Decision: The strengths of this paper includes Innovative use of language guidance for facial retouching, as well as clear, comprehensive argumentation and convincing experimental results. The limitation of this paper includes insufficient clarity on dataset selection, lack of comparison with RetouchFormer, limited real-world validation as well as potential scalability issues. In the rebuttal, the authors have addressed most of the concerns, hence all the reviewer gave positive recommendations. Therefore, the AC recommends to accept this paper.

Reviews: review: paper_summary: paper_summary: The paper presents a novel approach to facial blemish removal using a Language-guided Blemish Removal Transformer (LangBRT). The proposed method effectively integrates vision-language pre-training and text-prompted blemish detection to achieve high-fidelity face retouching with limited paired data. The experimental results demonstrate the superior performance of LangBRT compared to state-of-the-art methods. Overall, the paper is well-structured and the methodology is sound. However, several areas require improvement and clarification to enhance the readability and impact of the work.
strength_weakness: strength_weakness: 1) The novelty of the LangBRT approach is well-argued, particularly the method of combining text prompts with facial retouching.

2) The paper demonstrates high scores on the FFHQR dataset, which is a notable achievement.
limitations: limitations: 1) The paper lacks a quantitative comparison with RetouchFormer: Semi-supervised High-Quality Face Retouching Transformer with Prior-Based Selective Self-Attention (AAAI 2024), which is the SOTA on FFHQR. Including this comparison would strengthen the evaluation of the proposed method.

2) The explanation of the Text-prompted Blemish Detection (TBD) module is somewhat abstract. Including a more detailed algorithmic description or pseudo-code would aid in understanding.

3) Clarify the process of how the dataset was randomly selected. A detailed explanation would provide more transparency regarding the experimental setup.

4) The paper should provide more information on the number of predefined prompts. Explain how predefined prompts are aligned with user input prompts during training, and how the model distinguishes between different types of blemishes in the same region, such as "forehead wrinkles" and "acne on forehead".

5) The experimental setup, including dataset details and evaluation metrics, is adequately described, but additional clarity on the selection criteria for the in-the-wild data would be beneficial.

6) The ablation study is comprehensive, but consider providing more visual examples of the intermediate outputs from the ablation variants to illustrate the contributions of each component more effectively.
paper_summary: paper_summary: This paper tackles the challenge of automatically retouching face images while reducing the dependency on paired training data. Motivated by the limitations of existing methods in handling complex and unusual blemishes, the authors propose the Language-guided Blemish Removal Transformer (LangBRT). LangBRT leverages vision-language pre-training and incorporates a text-prompted blemish detection module to identify and edit specific blemishes, while maintaining normal skin regions. The framework integrates a target-aware cross attention mechanism within transformer blocks to ensure precise blemish removal. Extensive experiments demonstrate that LangBRT outperforms state-of-the-art auto-retouching methods, showing superior performance in blemish detection accuracy and synthesis quality, even with limited training data.
strength_weakness: strength_weakness: 1. Innovative Approach with Language Guidance: The LangBRT framework introduces a unique method that leverages vision-language pre-training for facial blemish removal. This approach reduces the dependency on paired training data and allows the model to handle specific blemishes based on textual prompts, which is a significant improvement over traditional methods that require extensive paired datasets.

2. Target-Aware Cross Attention Mechanism: The incorporation of a target-aware cross attention mechanism ensures precise editing of blemish-like regions while maintaining the integrity of normal skin areas. This mechanism enhances the accuracy and quality of the retouched images, which is critical for applications requiring high fidelity and realism.

3. Regularization for Semantic Consistency: By adopting a regularization approach that promotes semantic consistency between the synthesized image and the text description, LangBRT ensures that the final output aligns closely with the desired retouching outcome. This feature not only improves user satisfaction but also sets a new standard for the interpretability of automated retouching results.
limitations: limitations: 1. Complex Implementation and Training: The LangBRT framework's reliance on advanced components like vision-language models, cross attention mechanisms, and regularization techniques makes it complex to implement and train. This complexity can be a barrier for widespread adoption, especially for users with limited computational resources or expertise.

2. The modification of ViT in this downstream task lack of the overall innovation of the related community.

3. Also, the main figure of the proposed method seems a little bit fuzzy and can be state clearer with sperate ones.

3. Limited Real-World Validation: While the paper demonstrates impressive results in controlled experiments, there is a need for more extensive real-world validation. The model's performance in diverse and uncontrolled environments needs to be thoroughly tested to confirm its robustness and adaptability.

4. Potential Scalability Issues: The framework's reliance on textual prompts and extensive feature extraction processes may pose scalability challenges when applied to large-scale datasets or in real-time applications. Ensuring efficient computation and maintaining performance at scale remains a significant challenge.
paper_summary: paper_summary: This paper proposes a model named LangBRT, which provides an efficient and accurate solution for facial blemish removal by combining a text-prompted blemish detection module and a target-aware cross attention mechanism. It also ensures high-quality output of the synthesized image through regularization techniques.
strength_weakness: strength_weakness: The content of this paper is clear, readable, the argument of the paper is comprehensive, the experiment is sufficient, and the credibility is high. The math part of the paper is logically clear.
The Text-prompted Blemish Detection is an innovative method introduced in the paper. For blemishes in specific areas such as dark circles, acne, and wrinkles, the TBD can utilize a pre-trained visual language model to obtain text prompts. It then uses an image encoder to extract features from the source image and generates corresponding mappings that indicate the blemishes related to the description.
Additionally, the paper introduces a Target-Specific Cross-Attention mechanism (TCA). The TCA is capable of injecting defect information into the transformer blocks in T, which limits the main feature transformation of the defect-like areas and guides the transformer blocks to progressively restore the clean skin in that area.
The experimental results of the paper also demonstrate that the proposed method can produce more satisfactory results in removing different types of blemishes.
limitations: limitations: 1. The model proposed in the paper may have a relatively complex structure, which could cause higher computational costs during both model training and inference.
2. Figure 2 shows the description of the Discriminator but does not show the position and relationship between it and other modules in the overall framework. And there is no reference to Discriminators in the annotations.
3. The format of formula 11 looks a little strange.

Furthermore, I find that the mapping layer in the TBD of figure 2 is an ascending dimension process according to the diagram, but the size of its input and output are the same in figure 2.
paper_summary: paper_summary: The paper introduces LangBRT, a novel face retouching transformer that achieves high-fidelity blemish removal with limited paired training data. It leverages vision-language pretraining for precise facial blemish detection and employs a text-prompted detection module to guide the editing process. LangBRT uses a target-aware cross-attention mechanism to edit blemish-like regions accurately while maintaining normal skin areas unchanged. Extensive experiments demonstrate its superior performance over existing methods, especially under conditions with scarce training data. The model is capable of handling diverse blemishes and provides user-defined retouching options.
strength_weakness: strength_weakness: 1. The method proposed in this paper is very effective and has achieved satisfactory results on mainstream face retouching datasets.
2. The writing of the paper is smooth, the drawing is exquisite, and it can meet the requirements of publication.
3. The idea of using alpha clip to assist retouching is very natural and effective, which is a successful attempt.
4. The experimental results are solid with detailed analysis.
limitations: limitations: 1. Authors should try to conduct experiments on different ethnic groups and try different degrees of flawed data
2. The proposed method contains too many elements. Personally, I prefer the distillation of simple and effective methods
3. Please compare recent methods dedicated to retouching, such as：
[G. Hong et.al, "HQRetouch: Learning Professional Face Retouching Via Masked Feature Fusion and Semantic-Aware Modulation,"  2023 IEEE International Conference on Image Processing (ICIP)]