Audio-Driven Identity Manipulation for Face Inpainting

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-04 07:03:31

Abstract: Recent advances in multimodal artificial intelligence have greatly improved the integration of vision-language-audio cues to enrich the content creation process. Inspired by these developments, in this paper, we first integrate audio into the face inpainting task to facilitate identity manipulation. Our main insight is that a person's voice carries distinct identity markers, such as age and gender, which provide an essential supplement for identity-aware face inpainting. By extracting identity information from audio as guidance, our method can naturally support tasks of identity preservation and identity swapping in face inpainting. Specifically, we introduce a dual-stream network architecture comprising a face branch and an audio branch. The face branch is tasked with extracting deterministic information from the visible parts of the input masked face, while the audio branch is designed to capture heuristic identity priors from the speaker's voice.
The identity codes from two streams are integrated using a multi-layer perceptron (MLP) to create a virtual unified identity embedding that represennts comprehensive identity features. In addition, to explicitly exploit the information from audio, we introduce an audio-face generator to generate an `fake' audio face directly from audio and fuse the multi-scale intermediate features from the audio-face generator into face inpainting network through an audio-visual feature fusion (AVFF) module. Extensive experiments demonstrate the positive impact of extracting identity information from audio on face inpainting task, especially in identity preservation.

Main paper: 
 introduction: 
 Audio and image are the two most common modes that people use to perceive the world [10] . They are closely related, especially when it comes to recognizing other people or sensing their emotional states [35] . People are often able to build a mental model of what a person looks like based on a voice, because vocal and visual signals are highly correlated [19, 42] . A large number of studies have proved that a person's biophysical parameters, such as gender, age, Figure 1 : Illustration of our pipeline. We extract a highdimensional implicit identity embedding from audio and generate an audio face to fully exploit the identity information in audio. This information contains identity clues, such as age and gender, which can serve as guidance for the face inpainting network.health status, etc, can be inferred from the voice [32, 37] . What's more, by modeling the relationship between audio and image, researchers have drawn faces directly from speech clips [35, 47] . Although the images were not clear enough, they proved to be closely related to a person's identity.There is a relationship between a person's voice and his facial structure [47] . On the one hand, the underlying skeletal and articulator structure of the face directly affects the shape, size and other acoustic characteristics of the vocal tract, resulting in different voices that people make [33, 41] . On the other hand, objective factors such as gender, age, environment, etc, are all important components of voice characteristics [21] . In fact, it turns out that each of these factors can be inferred independently from voices or faces [22, 23] . Thus, by associating key facial attributes with important features of voices, the audio information may assist in generating a more restored face when removing face occlusion or repairing faces.Face inpainting is an important work, which is widely used in many fields, such as blemish restoration [1] , occlusion removal [5, 24] , face makeup [9] , blink repair [8, 49] , etc. In recent years, deeplearning based face inpainting methods have achieved good results [11, 30, 44] . Thanks to the widespread use of GAN [14] and Transformer [45] , they can generate realistic looking faces without maintaining the face identity. Different from natural images, faces are more structured and have their own unique identity characteristics. Face inpainting tasks can be oriented to more practical needs and complement other face tasks, such as face recognition [13] , face detection [17] , facial attribute classification [7] , etc, which have high requirements for the preservation of face identity.In order to preserve or complete the face identity, some methods try to introduce external prior knowledge. [18, 52] constraint identity consistency by calculating semantic feature losses on the output and ground truth, which is more like feature regularization than completion. Some other methods [8, 25, 40] reference images are available when making inferences. They extract key identity attributes, such as eyes, from reference images to compensate for the loss of identity. [28] design a cross-modal disentanling network to extract identity information both from text and reference images for eyeglasses removal. While these methods demonstrate the advantages of reference images and text for identity preservation, these references are not readily available in practical applications such as surveillance analysis. With the development of multimedia, it is easy to obtain a face that is speaking from videos, which may be covered by a mask and need to be repaired. Therefore, we propose an audio-driven face inpainting approach, which infers face identity from audio to achieve high-fidelity face generation.In general, our method is a dual-stream network including a face branch and an audio branch, as shown in Figure 1 . In the face branch, we encode the masked faces into a high-dimensional face embedding representing deterministic identity prior derived from the unmasked regions. In the audio branch, we use a pre-trained audio embedding network to extract the audio identity embedding, which is a heuristic prior that implicitly expresses identity information about the face. Then, we fuse the two codes and obtain a complete identity code through an MLP. We also introduce an identity embedding loss for constraining the complete identity features with the face identity labels. Although our method can integrate the identity information of the audio in this way, it is difficult for a single face decoder to decode this implicit representation. Therefore, we introduce an additional audio face decoder to reconstruct faces from audio identity embedding, through which we pass the intermediate multi-scale feature maps to the face decoder as lowlevel semantic complements. We fuse features from two decoder with an audio-visual feature fusion (AVFF) module and generate a final face. In the end, we apply an identity consistency loss to constraint the final face and the audio face. We show some identity preservation and swapping results in Figure 2 .In general, this paper has the following main contributions:• For the first time, we introduce audio into face inpainting for face identity preservation and leverage implicit representation and explicit features for identity reasoning. • We design an audio-visual feature fusion (AVFF) module to fuse multi-scale features from the face and audio decoder.He learns an attention map containing global and local information for better feature fusion. • We introduce an identity embedding loss and an identity consistency loss. Identity embedding loss is used to generate completed identity features, and identity consistency loss is used to constrain the feature consistency between the final face and audio face. • We pre-process the previous audio-face dataset to obtain a high-quality audio-face paired dataset and demonstrate that our method performs better in generating high-fidelity faces than state-of-the-art methods. 
related work: 
 Face Inpainting. Image inpainting aims to reconstruct the missing areas of the input images. Most of the existing image inpainting methods [11, 44] reasonably infer the missing pixel through the information around the hole. Compared with natural images, face images have stronger topological structure and local coherence. Therefore, it is of great significance to effectively predict the structure of faces by using the information around the hole. CSA [29] proposes a coherent semantic attention layer that better retains the missing structural information of the images. By recurrent feature reasoning, RFR [26] continuously fuses reasonable pixels around the hole to produce clear results. MISF [27] focuses on the smoothness between adjacent pixels of the images, which can realize high-fidelity image restoration. ICT [44] introduces transformer into the image inpainting task for the first time, which is used to reconstruct the structural priors of the images. VQFR [15] realizes high-quality blind face restoration based on the vector quantization dictionary and parallel decoders. Although these methods try to reconstruct the face structures, they do not consider the identity knowledge of the faces. A person's voice has a strong correlation with his face structure. Thus, the voice may help restore the topology of faces, and the rich face identity information contained in voices may be conducive to maintaining the original attributes of faces. Identity from audio Inferring speaker identity from audio is a long-standing task. Early work [20, 38] designed some hand-crafted features to map audio into a compact low-dimensional identity space for speaker identification. In recent years, some methods [4, 34] extend the representation to a much higher dimension to adequately extract speaker-discriminating features by deep learning network. More explicitly, some methods [12, 16, 50] predict specific identity attributes, such as age, gender, etc., directly from audio. These methods demonstrate that audio can provide a rich identity information supplement for face inpainting. Face reconstruction from audio Reconstructing faces from audio has received much attention in recent years. A few methods directly learn an audio-face mapping from large data without any face prior. Y Wen [48] et al. use GAN to train a face generator, using one discriminator to determine real or fake faces and another to discriminate identities. Similarly, Speech2Face [35] trained a highly capable decoder on a million spectra-face data pairs to generate audio-visual identity-consistent faces. Nevertheless, these two methods can only generate relatively low-quality faces. On the other hand, talking face [3, 53, 54] has become a hot topic of recent research. It aims at generating mouth-synchronized faces from audio, so it focuses more on the content of the audio than on the identity. In contrast, our method aims to extract the speakers' identity and ignores the audio's content. 
approach: 
 Given a person's masked face and an audio recording, our method aims to infer identity characters from audio and generate an identitypreserving face. The overview of our method is shown in Figure 3 . We masked a face 𝑥 ∈ R 𝐻 ×𝑊 ×3 with a large rectangle mask m ∈ {0, 1} 𝐻 ×𝑊 ×1 indicating the pixels need to be inpainted (with value 1) or not (with value 0). The audio recording is processed into a log Mel spectrogram with a fixed size. The masked face and the log mel-spectrogram are denoted as 𝑥 𝑚 and 𝑎 ∈ R ℎ×𝑤 .Our method includes a face branch and an audio branch. In the face branch, we send 𝑥 𝑚 into a face identity encoder to extract an identity embedding 𝐼 𝑓 from the remaining area of the input face. Similarly, in the audio branch, an audio identity embedding 𝐼 𝑎 is generated from audio using a pre-trained speaker recognition network. 𝐼 𝑓 and 𝐼 𝑎 are first concatenated to generate a completed face identity embedding 𝐼 𝑐 , then fed into different decoders reconstructing a inpainted face 𝑥 𝑓 and an audio face 𝑥 𝑎 . In addition to implicit identity embedding fusion, multi-scale features from two decoders are reasonably fused in an audio-visual feature fusion (AVFF) module. We describe the fusion process in Section 3.1. We also introduce an identity embedding loss and an identity consistency loss (described in Section 3.2 ) to constrain the completed identity embedding and the consistency between the final face 𝑥 𝑓 and audio face 𝑥 𝑎 .Identity Embedding Fusion. The face identity embedding 𝐼 𝑓 from the face branch can be considered a deterministic prior learned from the visible areas of the face. In contrast, the audio identity (𝐶 = 512, 𝐶 ′ = 64) are 1D vectors, we directly concatenate them along channel dimension and pass them through a linear layer to generate an intermediate feature 𝑓 ∈ R 𝐶 ×1×1 sending into the face decoder. In addition, we take an MLP as a modulator mapping 𝑓 into a high-dimensional space (set to 2048 by default ) represented as the completed face identity 𝐼 𝑐 . Audio Face Decoder. Although the intermediate features 𝑓 include identity information from both face and audio, it is difficult to reconstruct an audio-visual consistent face from a single face decoder 𝐷 𝑓 . An immediate consequence is no difference in the generated faces when the audio changes. To address this problem, we introduce an audio face decoder 𝐷 𝑎 generating face directly from audio identity embedding 𝐼 𝑎 . The middle features in this decoder explicitly describe audio identity at the pixel level.However, reconstructing faces from audio is not easy work because the network needs to learn the mapping relationship between audio and faces from a large amount of data. Although early work [48] has been attempted, the faces they generate are at a very low resolution (64 × 64), making them unsuitable for generating high-quality faces. Instead of applying their method directly, we retrain it in a collected high-quality audio-face dataset. We also add several upsampling layers to fit a high resolution (256 × 256). In the end, our audio face decoder takes audio identity embedding 𝐼 𝑎 as input to generate an audio face 𝑥 𝑎 ∈ R 𝐻 ×𝑊 ×3 . Audio-Visual Feature Fusion. The feature from audio face decoder 𝐷 𝑎 explicitly describe the identity information from audio. We extract mutli-scale feature vectors 𝑓 𝑖 𝑣 , 𝑓 (AVFF) module to integrate two face features, shown in Figure 4 .For the i th level, given two feature vectors 𝑓 𝑖 𝑣 and 𝑓 𝑖 𝑎 , AVFF extract local and global information through two branches. The fusion process can be described as:𝑚 = Sigmoid(𝜉 𝑙 (𝑐𝑎𝑡 (𝑓 𝑖 𝑣 , 𝑓 𝑖 𝑎 )) + 𝜉 𝑔 (𝐺 (𝑓 𝑖 𝑣 + 𝑓 𝑖 𝑎 ))), 𝑓 𝑖 𝑟 = 𝑚 * 𝑓 𝑖 𝑣 + (1 -𝑚) * 𝑓 𝑖 𝑎 .(EQUATIONwhere 𝑐𝑎𝑡Given a masked face 𝑥 𝑚 and a audio recording 𝑎, our method can generate an final face 𝑥 𝑓 and an audio face 𝑥 𝑎 , which are identityconsistent. The ground truth face and mask is denoted as 𝑥 and m. We train our method with the following losses. Identity Embedding Loss. In the process of identity embedding fusion, we integrate the face identity embedding 𝐼 𝑓 from the face encoder and the audio identity embedding from the audio encoder and generate an identity embedding 𝐼 𝑐 through an MLP. 𝐼 𝑐 includes a definitive description of the unmasked face and inference of the identity attributes in audio so that it can be considered a complete identity description. In order to constrain this embedding consistent with the identity of ground truth face 𝑥, we extract its identity using a face identification network 𝜓 pre-trained on VGGFace2 [2] and calculate an L1 distance. We explain the identity embedding loss as follows:EQUATIONIdentity Consistency Loss. In addition to supervising that the predicted identity embedding and the real embedding are consistent, we find it necessary to supervise the identity consistency of the final face and the audio face, described in the following:EQUATIONIn this way, we explicitly constrain that the final face refers to the identity properties of the audio face during the generation. Reconstruction Loss. To maintain the similarity between the generated face and the real face, we calculate a L1 distance between the ground truth face and the final face, audio face respectively. EQUATION) To be noticed, L 𝑎 can be considered as a regularization for audio face, and does not destroy the ability of pre-trained audio decoder to reason about the identity of the audio. GAN Loss. Follow [29] , we also introduce a GAN loss in to make the final image look more realistic. It is defined as:L 𝑔𝑎𝑛 = E[𝑙𝑜𝑔(1 -𝐷 𝑤 (𝑥 𝑓 ))] + E[𝑙𝑜𝑔𝐷 𝑤 (𝑥)](5) where D is the discriminator parameterized by 𝑤. We optimize our method with the global loss function:L = 𝜆 𝑖𝑑𝑒 L 𝑖𝑑𝑒 + 𝜆 𝑖𝑑𝑐 L 𝑖𝑑𝑐 + 𝜆 𝑐 L 𝑐 + 𝜆 𝑎 L 𝑎 + 𝜆 𝑔𝑎𝑛 L 𝑔𝑎𝑛 (6)We set the loss weights as 𝜆 𝑖𝑒 = 0.001, 𝜆 𝑎𝑣 = 1, 𝜆 𝑐 = 1, 𝜆 𝑎 = 0.01, 𝜆 𝑔𝑎𝑛 = 0.002. 
experiments 4.1 experimental settings: 
 Data Preparation. Faces in Previous audio-face datasets like VoxCeleb [34] are low resolution and blurred, which are unsuitable for generating high-quality faces. The face quality in the Celeb-ID [8] is much higher (with a resolution of 300x300), but it lacks audio recordings. Fortunately, both two datasets share some same identity labels. Therefore, to obtain high-quality face-audio pairs, we match faces in the Celeb-ID dataset and the audio recordings in VoxCeleb with the same identity called VoxCeleb-ID. In addition, we collected two high-quality talking video datasets: FaceForen-sics++ [39] , and HDTF [51] . They are collected from YouTube with a resolution of 720p or 1080p, most of which are clear front-face talking videos. We collect 761 and 358 videos from the URL provided by two datasets, then recognize and crop faces from video frames.Methods FaceForensics++ HDTF VoxCeleb-ID ↑ PSNR ↑ SSIM ↓ LPIPS ↓ L1 ↓ Landmark ↑ PSNR ↑ SSIM ↓ LPIPS ↓ L1 ↓ Landmark ↑ PSNR ↑ SSIM ↓ LPIPS ↓ L1 ↓ Landmark CSA [For face images, we align them along the eye area and resize all faces to 256 x 256. For the audio recordings, we crop all audio recordings into 6 seconds segments. If the audio length is not long enough, we repeat the audio to make it at least 6 seconds. The audio sampling rate is 16KHz, and the channel number is one. Following [48] , we remove silence regions of each segment with a voice activity detector and extract a log mel-spectrogram using a Hann window of 25mm, 10ms hop, and 1024 FFT frequency bands. Finally, we get a 64x1000 dimensional vector for each audio segment.Since the number of audio recordings is much more than faces in VoxeCeleb-ID, we randomly sample at most 20 audio segments for each identity. In FaceForensics and HDTF, we randomly sample 10 faces as they are similar from different video frames. Moreover, since our method does not focus on audio content, we ignore the lips change in FaceForensics++ and HDTF and manually select the standard faces which are frontal and lips change-free. For each standard face, We calculate a VGG-Face feature from a ResNet-50 pre-trained in the VGGFace2 [2] dataset as the face identity label that is used to calculate identity embedding loss. After all that, we got three pre-processed high-quality audio face-paired datasets, shown in Table 1 . Model Pretraining. We follow the idea from [48] for reconstructing faces from audio. However, since their method is trained in a low-resolution (64 × 64) audio-face dataset, they can not generate high-quality faces. Therefore, we only borrow part of its parameters and add several upsampling layers to fit our high-resolutionAcc@1 Acc@5 Acc@8 Acc@10 CSA [29] 17 4 : Face retrieval performance. We measure retrieval performance by accuracy at K (Acc@K, in %), which indicates the chance of retrieving the same person's faces within the top-K results while using the reconstruction faces of different methods.dataset (256 × 256). We train the audio face decoder with faces and audio segments in FaceForensics++ and HDTF. During training, we randomly select one face and one audio segment from the sample identity. The maximum training iteration is 100k.Training. Since our method does not focus on the audio content and lips change, we train and evaluate our method with standard faces in three datasets. Given a face, we randomly select an audio segment and a face identity embedding of the same person. During training, we chose a rectangle mask with the size of 128 × 128 and set the learning rate as 2e-4 and batch size as 4. We chose ADAM as the optimizer with 𝛽 1 = 0.5, 𝛽 2 = 0.999 and train all the networks on an NVIDIA RTX 3090. More details of our method can be found in the supplementary material.We evaluate our method and recent face inpainting methods with the standard faces on three datasets. PSNR, SSIM, LPIPS, L1 distance are used in our experiments to measure the pixel-level similarity between our result and ground truth face, and we also calculate a Landmark distance conducted by dlib to measure the structural similarity of faces. In order to evaluate the face fidelity, we employ HOG, VGGFace [36] , SphereFace [31] , CosFace [46] and Ar-cFace [6] to measure the identity distance. The smaller the identity distance, the higher the face fidelity.Comparison with state-of-the-arts. We compare our method with five recent face inpainting methods, CSA [29] , RFR [26] , ICT [44] , MISF [27] , and VQFR [15] , in which ICT focus on pluralistic image completion, and VQFR focuses on blind face restoration. All methods are retrained in the same dataset for a fair comparison. We report the pixel and structural similarity comparison results in Table 2 . Our method performs better than other methods in most metrics for all three datasets demonstrating its strength in generating high-quality faces. For identity preservation, we show quantitative results in Table 3 . In FaceForensics++ and HDTF, our method outperforms all methods in all metrics and is slightly worse than MISF [27] in VoxCeleb-ID, which indicates the advantages of our approach in generating high-quality faces and face identity preservation.We also show some qualitative results in Figure 5 . Previous methods may generate face contents with mistaken identities when faces are missing in large areas. For example, in the first row of Figure 5 , when the input face is mostly masked and the remaining area does not provide an identity reference, previous methods incorrectly generates a male face. In contrast, in our method, audio face learns an identity prior from sound, reconstructs a female face, and correctly guides our face decoder to generate identity-consistent female face. In addition, we show some results under three different sizes of irregular masks in Figure 6 to verify the universality of our method. Face Retrieval. To verify that our method can generate results which are more closely related to the original facial features, we measure retrieval performance by accuracy at K (Acc@K, in %), which indicates the chance of retrieving the same person's faces within the top-K results while using the reconstruction faces of different methods as input. We used the test set of VoxCeleb-ID for this experiment, with 190 face images of different identities for retrieval and others for gallery. We query the face images by comparing the Euclidean distance of ArcFace face features between the reconstruction faces of different methods and the faces in the gallery. Table 4 shows the face retrieval performance. The experimental results show that our method can preserve the identity of faces more adequately.Although our method focuses on audio-driven identity preservation, it can also perform identity swapping with different reference audios. Since our face decoder is influenced by the audio embedding and intermediate features in the audio decoder, we can get a face with different identities if we change the input audio. We show the visualization results in Figure 7 . For the same input face, if we input a young woman's voice, our method will generate a face with female features. On the other hand, if we change to the voice of a young man, the final result will be more like a male face. We also show some pluralistic results of ICT [44] , which generates different faces without control. In contrast, our method can explicitly change the identity through audio and perform an audio-guided controllable face inpainting. We conduct careful ablation study on VoxCeleb-ID dataset. Our baseline builds on a UNet. To validate the effectiveness of each component in our method, we add them in baseline one by one to observe how the result changes.Quantitative and qualitative results are shown in Table 5 and Figure 8 . When we only employ one audio embedding network to exploit the high-dimensional identity, the identity distance decreases while PSNR decreases. The reason is that a single decoder is challenging to reconstruct high-quality faces from the identity code, leading to face distortion and blur, shown in the third column in Figure 8 . Since the features from the face decoder and audio face decoder are not pixel-aligned, directly concatenating two features will hurt the performance in all aspects. Although our AVFF module causes a decrease in PSNR and SSIM, it further reduces the identity distance compared to audio embedding, proving its effectiveness in identity preservation. We attribute the decline in PSNR and SSIM to the fact that the intermediate features of the audio decoder contain not only identity information but also other noise. We solve this problem by introducing identity consistency loss. The identity embedding loss constrains the space of the completed identity embedding with ground truth identity, which improves image quality and fidelity.To demonstrate the role of audio for identity preservation, we extract the identity embeddings from the images generated by baseline model and our method, and visualize thems by t-SNE [43] , shown in Figure 9 . Our method achieves better clustering than baseline indicating that audio is useful to provide discriminative identity information. 
discussion: 
 Potential Social Impact. Our method can achieve controlled face inpainting through changing audio. On the one hand, the encoding of facial identity may cause privacy leakage. On the other hand, face swapping produces fake face images, which may deceive the face recognition systems. Thus, this technique should be used with caution. Limitation. Our method realizes the generation of high fidelity faces by interacting with audio features. The results are affected by the quality of audio faces. Although our method can produce real-looking audio faces, the facial details need to be enhanced. Due to the limitation of identity number in our datasets, our model can only express several simple attributes, such as age and gender. What audio can bring to face inpainting needs more exploration. 
conclusion: 
 This paper first verifies the critical role of audio in face inpainting.We propose an audio-driven high-fidelity face inpainting method. It captures implicit and explicit identity representations from audio and learns deterministic priors from input faces via a dual-stream network. We design an audio-visual feature fusion module that can effectively integrate multi-scale deep features of cross-modal data. We also introduce two identity losses for preserving face identity. Experiments show that voice can help generate face structure and identity prior, and our method can generate high-fidelity faces with audio guidance. 
 
 Appendix:  


Decision: this paper investigates the face generation problem with multi-modal conditioning. specifically, it argues that the audio modality can provide additional information like gender and age other than the visual surroundings. a dual stream network is proposed for this task and is verified on face inpainting benchmarks. 

initially, the paper received ratings of BA, WA, BR, BA. reviewers had mixed feelings about it. introducing the audio modality is well recognized by the reviewers, and similarly appreciated are the visual results, the potential application, and overall writing. there are concerns over the biasing the system through the usage of a voice recognition system, voice-to-face being a one-to-many mapping and not sufficiently modelled in this work, lack of discussion and comparison with related work, the face cropping out and the experimental settings, and some clarity issues. during the rebuttal, the authors provided feedback for these concerns. while there remains some un-resolved issues regarding the mask in the training and the ablations, the reviewers overall post postive feedbacks. 

since the AC was unable to finish the meta review in time, the PC stepped in and carefully went over all the reviews and rebuttal. after consideration, the PC recommends Accept. with that said, the authors are strongly encouraged to modify the manuscript based on the rebuttal and further clarify issues regarding the the mask in the training and the ablations.

Reviews: review: paper_summary: paper_summary: Introduce the concept of audio face to improve the face inpainting task. The proposed the AVFF module improves the identity reservation of the overall generation model.
strength_weakness: strength_weakness: 1. good visual effect  of the generated image
2. maintain identity consistency better than the previous methods
limitations: limitations: The audio contains various types of information, such as rhythm, tone, voice, and speaking content. A more fine-grained disentanglement design is needed to improve the identity extraction ability. Relying solely on a speaker recognition model might introduce bias based on the training dataset used for the speaker recognition task.
paper_summary: paper_summary: The paper presents an innovative approach to face inpainting that integrates audio cues to facilitate identity manipulation. The core idea is that a person's voice contains distinct identity markers, such as age and gender, which can enhance face inpainting tasks. The authors propose a dual-stream network architecture that includes a face branch for deterministic identity extraction and an audio branch for heuristic identity priors. The system generates a unified identity embedding through a multi-layer perceptron (MLP) and introduces an audio-face generator to fuse multiscale features into the face inpainting network via an audio-visual feature fusion (AVFF) module. Experiments demonstrate the method's effectiveness, particularly in identity preservation.
strength_weakness: strength_weakness: Novelty: This paper unprecedentedly introduces the audio modality into the face inpainting task, representing a significant departure from traditional approaches that rely solely on visual information. By leveraging the unique identity cues embedded in a person's voice, such as age, gender, and emotional state, the authors propose a dual-stream network architecture that integrates audio analysis with facial feature reconstruction. This cross-modal strategy not only enhances the fidelity of face inpainting but also introduces a new dimension for identity manipulation, allowing for both identity preservation and swapping.
Application: The paper illustrates its strength in identity preservation, which aligns well with the identity swapping task, suggesting broad applications for the proposed method. The ability to manipulate facial identity while maintaining high fidelity in image reconstruction opens up numerous possibilities across various fields.
limitations: limitations: This article discusses the task of facial reconstruction using audio information [47] and proposes the novel application of audio information for facial inpainting tasks. What are the unique challenges and difficulties in utilizing audio information, especially for the task of facial inpainting? The authors mention the potential future applications of this work for face recognition and facial attribute classification in the introduction (lines 106-109). What considerations are behind this suggestion? Overall, this paper is a pioneering work in the task of facial repair using audio information, but it lacks a more thorough comparison with other facial tasks that also utilize audio information. In particular, the paper does not sufficiently discuss the potential value of using audio information for applications such as face recognition and facial attribute classification.
paper_summary: paper_summary: This paper proposes a method to infer visual identity information from a speaker’s voice and utilizing the extracted identity information for face inpainting and face swapping. More specifically, an audio encoder is utilized to extract the identity information. These identity information is then fused into the cropped face by a face decoder to serve as an identity guidance to reconstruct the face.
strength_weakness: strength_weakness: ①The task is interesting and the connection between human voice and visual appearance has the potential for a wide range of application.

②The paper is well organized and easy to follow.
limitations: limitations: ### Weakness and Question

① My primary concern lies in the correspondence between the voice identity and face identity. While this paper posits that this correspondence is a deterministic one-to-one relationship, I contend that it should be a more ambiguous mapping (a many-to-many relationship). For instance, both of the following scenarios are plausible: individuals with similar appearances may exhibit entirely different vocal characteristics, and conversely, people with similar voices may possess completely distinct facial features. Therefore, employing probabilistic generative models, such as Variational Autoencoders (VAEs) or diffusion models, may better capture the ambiguous mapping. Deterministic models like GANs may lead to a neutral face (the reconstrcted audio face $x_a$ in this paper).

② When preparing masks for facial regions, the center crop masks are relatively coarse and may result in models learning not only facial identity information but also some redundant background information. Utilizing finer-grained masks, such as the results obtained by facial segmentation or anonymization, would be more effective.

③ The quantitative ablation experiments in both the main text (table 5) and supplementary materials (table 1) indicate that many of the proposed modules do not make significant contributions, which raises doubts about the necessity of the modules proposed in this paper.

④ Some further experiments may make the proposed method more convincing. For instance, it would be informative to explore the outcomes of guiding the same masked face:
+ with/without audio 
+ audios with different identity but the same content 
+ audios with the same identity but different contents

Similarly, investigating the effects of using the same audio to guide different masked faces would provide valuable insights.
paper_summary: paper_summary: This paper introduces audio signal into the face inpainting task and recover the target face whose identity corresponds to the property of the audio signal. Also, the audio-visual feature fusion (AVFF) module is proposed to fuse multi-scale features from the face and audio decoder. The identity embedding loss and the identity consistency loss are designed to train the module.
strength_weakness: strength_weakness: This paper is the first to introduce audio guidance into the face inpainting task and can serve as an inspiration for future related works. Also, the writing is good and the paper is quite easy to follow.

The dataset preparation is thorough and experimental results are comprehensive enough to support the proposed method.
limitations: limitations: The motivation of the network design in Audio-Visual Feature Fusion Module is a little unclear. It seems meaningless to add the identity feature map and what is the design principle? What do the add branch and concat branch stand for?

How to choose the mask size and location in the face image? Does it impact the performance?

From the qualitative results, the center area of the generated faces is somehow faint compared to other methods. Why getting leading scores on PSNR and SSIM metrics? Is it related to the input visible areas of the face? Adding metric like CPBD could be more convincing.

The experimental setting is unfair. Additional information like audio signal is provided to the propose method during the generation process, while other compared methods get information no more than prompt word like male or female. How to explain the experimental setting?

Some statements are ambiguous. a) What does the word manipulation in refer to? The proposed model seems to recover the face identity under the guidance of audio clues. The terms inpainting and manipulation seem a little contradictory. b) I_i is not marked in Figure 3.