S2TD-Face: Reconstruct a Detailed 3D Face with Controllable Texture from a Single Sketch

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 03:48:27

Abstract: 3D textured face reconstruction from sketches applicable in many scenarios such as animation, 3D avatars, artistic design, missing people search, etc., is a highly promising but underdeveloped research topic. On the one hand, the stylistic diversity of sketches leads to existing sketch-to-3D-face methods only being able to handle pose-limited and realistically shaded sketches. On the other hand, texture plays a vital role in representing facial appearance, yet sketches lack this information, necessitating additional texture control in the reconstruction process. This paper proposes a novel method for reconstructing controllable textured and detailed 3D faces from sketches, named S2TD-Face. S2TD-Face introduces a two-stage geometry reconstruction framework that directly reconstructs detailed geometry from the input sketch. To keep geometry consistent with the delicate strokes of the sketch, we propose a novel sketch-to-geometry loss that ensures the reconstruction accurately fits the input features like dimples and wrinkles. Our training strategies do not rely on hard-to-obtain 3D face scanning data or labor-intensive hand-drawn sketches. Furthermore, S2TD-Face introduces a texture control module utilizing text prompts to select the most suitable textures from a library and seamlessly integrate them into the geometry, resulting in a 3D detailed face with controllable texture. S2TD-Face surpasses existing state-of-the-art methods in extensive quantitative and qualitative experiments. The code will be publicly available.

Main paper: 
 related work: 
 3D Face Reconstruction. Reconstructing 3D faces from 2D images has achieved widespread success. Methods such as [17, 19, 26, 37, 59, 66, 67] can generate realistic 3D faces from facial images captured in various poses, environments, and expressions. These methods typically utilize 2D landmarks, segmentation, texture information, etc. to guide the deformation of 3DMMs [4, 5] , and further leverage differentiable rendering techniques [14, 21, 34, 41, 51, 55] for fine-grained reconstruction [19, 37] . These validated 2D-to-3D supervision approaches are applicable for training sketch-to-3D-face framework. Some methods [2, 18, 24, 35, 36] focus on texture reconstruction. They typically decompose textures into components such as diffuse, specular, ambient occlusion, normal, and translucency, applicable in re-rendering in new environments or creating 3D avatars. However, the textures provided by these methods lack diversity, missing complex styles such as cartoon styles or makeup as shown in Fig. 1 , and still exhibit disparities in high-frequency details compared to textures directly derived from image UV mapping.(a) (b) (c) (d) (e) (f) (h) (g)Translate Sketches to Other Modalities. Some methods [3, 22, 25, 43, 62, 64] reconstruct 3D shapes from sketches of common objects such as cups, chairs, cars, airplanes, etc. They typically supervise the 3D geometry based on 2D silhouettes using differential renderers [62] , involving point set matching and optimization (such as chamfer distance) [22, 25] , or innovation in 3D representation forms (such as Signed Distance Fields [46] ) [43, 64] . These methods are usually limited to specific types of objects, and a domain gap exists when reconstructing faces. Few methods reconstruct 3D faces from facial sketches, they either specialize in sketches with frontal poses [28] or heavily depend on professional sketches with precise shading as input [61] , which may not align with practical requirements. Some methods [11-13, 23, 40, 52] translate facial sketches into 2D face images, often utilizing frameworks such as Generative Adversarial Networks (GANs) [15] or Neural Radiance Fields (NeRFs) [45] to synthesize face images. Combining these sketchto-face-image methods with image-to-3D-face methods [19, 59, 66] seems like a straightforward solution. However, the local stroke information of the original input sketch (such as dimples, wrinkles, etc.) is easily lost in this process, leading to reconstruction results that are not consistent with the input sketches. 
methodology 3.1 preliminaries: 
 Data Processing. For a given RGB face image 𝑰 ∈ R 𝐻 ×𝑊 ×3 , based on the common practices in [13, 52, 56, 57] , we generate 5 types of sketches 𝑺 𝑡 𝑖 ∈ R 𝐻 ×𝑊 ×3 :EQUATIONwhere we integrate existing various sketch operations [7, 56, 57] into a single function Φ sketch (•). We make Φ sketch (•) differentiable and will also apply it to the sketch-to-geometry loss. 𝑡 𝑖 represents different sketch types, 𝑖 ∈ [1, • • • , 5], as illustrated in Fig. 2 (a)-(e). Following [59] , we utilize landmark detectors [59] to obtain 2D landmarks 𝒍𝒎𝒌 ∈ R 2×240 and employ DML-CSR [63] to generate Face Model. Based on [8, 27, 48] , we define the coarse vertices and albedo of a 3D face using the following formula:EQUATIONwhere 𝑽 and 𝑻 are the mean geometry and the mean albedo, respectively. 𝑉 3𝑑 ∈ R 3×35709 is the coarse face vertices and𝑇 𝑎𝑙𝑏 ∈ R 3×35709 is the albedo. 𝜷 𝑖𝑑 ∈ R 80 , 𝜷 𝑒𝑥𝑝 ∈ R 64 and 𝜷 𝑎𝑙𝑏 ∈ R 80 are the identity geometry parameter, the expression geometry parameter and the albedo parameter, respectively. 𝑩 𝑖𝑑 , 𝑩 𝑒𝑥𝑝 and 𝑩 𝑎𝑙𝑏 are the face identity bases, the expression bases and the albedo bases, respectively. We utilize angles 𝜷 𝑎 ∈ R 3 (pitch, yaw, and roll) to obtain the rotation matrix 𝑹 (𝜷 𝑎 ) ∈ R 3×3 , for the rotation of 𝑉 3𝑑 . We employ 𝜷 𝑡 ∈ R 3 to control the translation of 𝑉 3𝑑 . Note that 𝑇 𝑎𝑙𝑏 is not the final facial texture and it will not appear during the inference process of the framework. 𝑇 𝑎𝑙𝑏 solely assists in supervising the geometry during the training process.Face Attributes in UV Space. UV mapping is a reversible 3D modeling process commonly used to project the attributes of 3D objects into the 2D image plane. We can transfer facial geometry information, facial texture information, and other attributes to UV space.These techniques are employed in many 3D face reconstruction methods [10, 18, 20] , often combined with differentiable renderers [34, 51] , facial texture completion [2, 10, 18] , and illumination estimation [19] . In the following section, we denote the facial attribute 𝑿 in UV space as 𝑿 𝑢𝑣 .Camera. Following [17, 37, 59] , we utilize a camera with a fixed perspective projection for the re-projection of 𝑉 3𝑑 into the 2D image plane, yielding 𝑉 2𝑑 ∈ R 2×35709 .Illumination Model. Based on [17, 19] , we employ Spherical Harmonics (SH) [50] to predict the shading information:EQUATIONwhere ⊙ denotes the Hadamard product, 𝑵 is the surface normal of 𝑉 3𝑑 , 𝚿 : R 3 → R is the SH basis function and𝜷 𝑘 𝑠ℎ ∈ R 3 is the corresponding SH parameter, 𝑘 ∈ [1, • • • , 9]. 𝑨 represents the albedo information, which could be set as 𝑇 𝑎𝑙𝑏 to calculate the shaded texture. Following [19, 37] , we also set 𝑨 to a fixed gray value 𝑨 𝑔𝑟𝑎𝑦 to display the geometry shading.Detail Reconstruction. The coarse geometry 𝑉 3𝑑 based on 3DMMs can not capture the high-frequency details of a 3D face. To address this, we perform detail reconstruction based on [19, 37] , which is achieved by computing a displacement map:EQUATIONwhere the coarse geometry and detail geometry in UV space, respectively. 𝑵 𝑢𝑣 ∈ R 256×256×3 represents the surface normal corresponding to 𝑉 3𝑑 𝑢𝑣 . 𝜷 𝐷 ∈ R + is used to control the magnitude of the displacement map 𝑫 𝑢𝑣 . We denote the surface normal of the detail geometry 𝑉 ′ 3𝑑 as 𝑵 ′ .𝑫 𝑢𝑣 ∈ R 256×256 representsRendering. Based on [19, 34, 51] , we construct a differentiable renderer Φ render (•) using the fixed camera, which could yield the following results:EQUATIONwhere the input of the differentiable renderer Φ render (•) includes coarse geometry and shading information, achieving detailed rendering effects through the refinement of the normal map in the shading information. The rendering results 𝑰 𝑎 , 𝑰 𝑏 , 𝑰 𝑐 , and 𝑰 𝑑 represent the coarse texture, detail texture, coarse geometry shading, and detail geometry shading, respectively, which are used in sketchto-geometry loss, as shown in Fig. 4 .We aim to reconstruct detailed geometry consistent with the delicate features of the input sketch. The sketch-to-3D-face process of S2TD-Face is divided into two stages: coarse geometry reconstruction and detailed geometry reconstruction, as shown in Fig. 3 Coarse Geometry Reconstruction. During the training process, for each data sample, we randomly select a sketch 𝑺 𝑡 𝑖 of type 𝑡 𝑖 as input. We employ ResNet-50 [29] as the backbone Φ coarse to predict parameters 𝜷 𝑎 , 𝜷 𝑡 , 𝜷 𝑖𝑑 , 𝜷 𝑒𝑥𝑝 , 𝜷 𝑠ℎ , and 𝜷 𝑎𝑙𝑏 . These parameters are processed by the face model [8, 48] to generate coarse geometry 𝑉 3𝑑 and the PCA albedo 𝑇 𝑎𝑙𝑏 , as described in Eqn. 2. 𝑇 𝑎𝑙𝑏 is used for photometric loss L pho , perception loss L per , and sketch-togeometry loss L sketch . Note that during the inference, there are no restrictions on the sketch types, and neither 𝑇 𝑎𝑙𝑏 nor 𝜷 𝑎𝑙𝑏 are involved. Additionally, 𝜷 𝑠ℎ for controlling light can vary as shown in Fig. 1 . We utilize 𝑉 3𝑑 to map the sketch image 𝑺 𝑡 𝑖 to UV space, resulting in 𝑺 𝑢𝑣 𝑡 𝑖 . 𝑺 𝑢𝑣 𝑡 𝑖 . The UV space representation 𝑉 𝑢𝑣 3𝑑 of 𝑉 3𝑑 will jointly serve as the input for reconstructing the detailed geometry.3𝑑 and 𝑺 𝑢𝑣 𝑡 𝑖 as input, we employ a pix2pix network [30] Φ detail to predict the displacement map 𝑫 𝑢𝑣 for reconstructing the detailed geometry 𝑉 ′ 3𝑑 𝑢𝑣 in Eqn. 4. Throughout this process, 𝜷 𝐷 serves as a learnable parameter controlling the magnitude of 𝑫 𝑢𝑣 , which is fixed during inference.To train Φ coarse and Φ detail in S2TD-Face, we employ the following training methods and supervision loss functions.Various Sketch Types. The facial sketch types are diverse, with some containing realistic shading information, while others only consist of simple lines. To ensure the robustness of Φ coarse and Φ detail across different sketches, randomization is applied to all operations involving sketch type selection during the training process. Specifically, the training input employs a random sketch type 𝑡 𝑖 , as shown in Fig. 3(b) , and in the sketch-to-geometry loss L sketch , the type 𝑡 𝑗 is also randomly selected, as shown in Fig. 4 . These strategies ensures that Φ coarse and Φ detail possess strong adaptability to different sketch types.Sketch-to-geometry Loss. Existing supervision methods fail to accurately capture the local details of sketches (such as dimples, wrinkles, etc.) and reflect them onto the geometry deformation. To address this, we propose a novel sketch-to-geometry loss L sketch to ensure the geometry 𝑉 3𝑑 and 𝑉 ′ 3𝑑 fidelity to the sketch input and supervise Φ coarse and Φ detail robustly across different sketch types, as shown in Fig. 4 where type 𝑡 𝑗 is randomly selected. Since Φ coarse , Φ detail , Φ render , and Φ sketch are all differentiable, and we have the sketch ground truth 𝑺 𝑡 𝑗 corresponding to the type 𝑡 𝑗 , we could compare the differences between 𝑺 𝑛 𝑡 𝑗 𝑛 ∈ {𝑎, 𝑏, 𝑐, 𝑑 } and 𝑺 𝑡 𝑗 by using photometric loss and perception loss:EQUATIONwhere L sketch contains two error parts: photometric error and perception error. The former computes L2-norm error, while the latter computes the cosine distance. 𝜆 1 and 𝜆 2 are the corresponding weights. Φ per (•) is a face recognition network from [17] , used to extract features from the input, and Photometric Loss and Perception Loss for 𝑰 𝑏 . To enhance the robustness of the training process, we supervise the detail texture rendering result 𝑰 𝑏 in Eqn. 5 similar to [19, 37] . Note that this process operates at the rendering image level 𝑰 𝑏 , while L sketch operates at the rendering sketch level. The photometric loss L pho and perception loss L per used are defined as follows:EQUATIONEQUATIONwhere 𝑴𝑰 𝑏 = 𝑴 𝐶 ⊙ 𝑴 𝑟𝑒𝑛𝑑𝑒𝑟 ⊙ 𝑰 𝑏 and 𝑴𝑰 = 𝑴 𝐶 ⊙ 𝑴 𝑟𝑒𝑛𝑑𝑒𝑟 ⊙ 𝑰 . Similar to the operations in L sketch , Φ per (•) is a face recognition network [17] and < •, • > is the vector inner product. We emphasizeOperation Flow Gradient Flow again that in L sketch , L pho , and L per , the texture of 𝑰 𝑎 or 𝑰 𝑏 is derived from 𝑇 𝑎𝑙𝑏 , aims to supervise the geometry. 𝑇 𝑎𝑙𝑏 is not the final texture and will not appear in the inference process.Landmark Loss. We employ landmark loss to compare the predicted 2D landmarks 𝒍𝒎𝒌 ′ from 𝑉 2𝑑 with the ground truth 𝒍𝒎𝒌 obtained by [59] , adopting the dynamic landmark marching [65] to address the non-correspondence between 2D and 3D cheek contour caused by pose variations. The landmark loss L lmk is defined as:EQUATION)Part Re-projection Distance Loss. Since L lmk can only operate on sparse vertices in 𝑉 2𝑑 , we further utilize Part Re-projection Distance Loss (PRDL) [59] L prdl to supervise 𝑉 2𝑑 . L prdl leverages the precise 2D part silhouettes provided by segmentation 𝑪 to constrain the predicted geometry of facial features:EQUATION)where 𝑷 represents the set of facial components, 𝑷 = {left_eye, right_eye, left_eyebrow, right_eyebrow, up_lip, down_lip, nose, skin}. 𝚪 𝑝 (𝑉 2𝑑 ) and 𝚪 𝑝 (𝑪) respectively denote the shape descriptors of PRDL defined for prediction and target in [59] . 𝜆 𝑝 represents the weight of each part 𝑝. During training, specific parts 𝑝 of samples may be occluded or invisible, we set 𝜆 𝑝 = 0 for parts 𝑝 in these samples and 𝜆 𝑝 = 1 for the rest parts.Overall Losses. In summary, we minimize the total loss L to optimize the geometry reconstruction frameworks Φ coarse and Φ detail :EQUATION)where L reg is the regularization loss for parameters 𝜷. Training Details. We firstly train Φ coarse , then freeze Φ coarse to train Φ detail and finally train Φ coarse and Φ detail together. Therefore, during the first training stage when using L sketch , 𝑺 𝑏 𝑡 𝑗 and 𝑺 𝑑 𝑡 𝑗 are not used, and 𝑰 𝑏 in L pho and L per is replaced by 𝑰 𝑎 .We aim to design a texture control module for S2TD-Face that can select appropriate samples from a given texture library based on input text prompts, obtain textures in UV space, and seamlessly map them onto the geometry 𝑉 ′ 3𝑑 . As illustrated in the Fig. 3(c ), when the input prompt 𝑇 𝑒𝑥𝑡 is 'Cartoon Beard Man', we use Φ image and Φ text from CLIP [49] to encode the face images 𝑰 𝑖 𝐿𝑖𝑏 from the known texture library 𝐿𝑖𝑏 and the input text 𝑇 𝑒𝑥𝑡:EQUATIONwhere 𝐹 𝐼 𝑖 and 𝐹 𝑇 are the image encoding features and the text encoding features, respectively. |𝐿𝑖𝑏 | is the image number in the given texture library 𝐿𝑖𝑏. In the text-to-image matching process, each 𝐹 𝐼 𝑖 is compared to 𝐹 𝑇 to compute similarity, and S2TD-Face can either select the image with maximum similarity to the input text or randomly choose one from the top-k similar images. We denote the final matching result as 𝑰 𝑡𝑒𝑥 .The role of Φ uv-albedo in the texture control module is to transform the texture of the face image into UV spcae that are compatible with 𝑉 ′ 3𝑑 . We input the text-image matching result 𝑰 𝑡𝑒𝑥 to Φ uv-albedo to get the desired texture information in UV space. Specifically, Φ uv-albedo is based on the state-of-the-art monocular 3D face reconstruction method [59] . Φ uv-albedo firstly estimates the 3DMMs [8, 48] shape 𝑉 𝑡𝑒𝑥 and the PCA albedo 𝑨 𝑝𝑐𝑎 from 𝑰 𝑡𝑒𝑥 , and then utilizes the shape information 𝑉 𝑡𝑒𝑥 to map the input image 𝑰 𝑡𝑒𝑥 into UV space, obtaining 𝑨 𝑖𝑚𝑔 , as shown in the Fig. 3(c ). Due to the pose influence of 𝑉 𝑡𝑒𝑥 , some facial areas of 𝑰 𝑡𝑒𝑥 are invisible and the UV-texture information 𝑨 𝑖𝑚𝑔 may not cover the entire facial surface. Therefore, Φ uv-albedo calculates the invisible areas according to 𝑉 𝑡𝑒𝑥 and complete UV-texture using 𝑨 𝑝𝑐𝑎 , finally resulting in the fusion texture 𝑨 𝑓 𝑢𝑠𝑖𝑜𝑛 :EQUATIONwhere 𝑴 𝑖𝑚𝑔 is a mask computed by the differentiable renderer Φ render with the help of 𝑉 𝑡𝑒𝑥 , which represents the visible regions of the reconstructed shape 𝑉 𝑡𝑒𝑥 that consistent with the input texture mapping 𝑨 𝑖𝑚𝑔 . 𝑴 𝑝𝑐𝑎 = 1 -𝑴 𝑖𝑚𝑔 indicates the regions that require further complement by the predicted 3DMMs PCA albedo 𝑨 𝑝𝑐𝑎 . To reduce visual differences at the fusion boundaries, we apply median filtering [7] to 𝑴 𝑖𝑚𝑔 . The texture control module finally applies 𝑨 𝑓 𝑢𝑠𝑖𝑜𝑛 onto the geometry 𝑉 ′ 3𝑑 through UV mapping, resulting in a detailed and textured 3D face, as shown in the Fig. 3(d) .Table 1 : Quantitative comparison on Sketch-REALY benchmark. We transform the test samples from REALY [9] into two types of sketches: 'Shading' (realistic shaded sketches) and 'Line' (sparse line sketches), as shown in Fig. 5 , and perform quantitative comparison respectively. Lower values indicate better results. The best and runner-up are highlighted in bold and underlined, respectively. We also investigate the effect of removing the sketch-to-geometry loss L sketch (denoted as 'Ours (w/o L sketch )') for ablation study. [17, 19, 20, 26, 37, 54] : firstly translating 2D sketches to face images [52] and subsequently reconstructing 3D faces or directly using sketches as input. For fairness, methods marked with † represent the best results obtained from these two ways. The face scanning for geometry evaluation in Sketch-REALY. 
experiments 4.1 experimental settings: 
 Implementation Details. We implement S2TD-Face based on PyTorch [47] . The input sketches are resized into 224×224. 𝑨 𝑔𝑟𝑎𝑦 = (127, 127, 127). We use Adam [33] as the optimizer with an initial learning rate of 1𝑒 -4. 𝑩 𝑖𝑑 and 𝑩 𝑎𝑙𝑏 are from BFM2009 [48] and 𝑩 𝑒𝑥𝑝 is from FaceWarehouse [8] .Data. We utilize face images from publicly available datasets, including CelebA [42] , 300W [53] , RAF [38, 39] , and DAD-3DHeads [44] , which are commonly used in 3D face reconstruction tasks. We employ [66] for face pose augmentation and [59] for face expression augmentation. As a result, we obtain about 600𝐾 face images for training. Φ sketch is based on [7, 56, 57] and each face image is processed by Φ sketch to obtain 5 different styles of sketches as input to the framework (resulting in 5 × 600𝐾 sketches). The images in the texture library 𝐿𝑖𝑏 of the texture control module are sourced from FFHQ [32] and online collections, totaling about 1000 images.Sketch-REALY. The REALY benchmark [9] comprises 100 precise 3D face scanning data (as shown in Fig. 5 (d )) from LYHM [16] , which are from different identities and include accurate landmarks, region masks, and topology-consistent meshes. During the evaluation, REALY initially aligns the prediction and ground truth using landmarks. It subsequently divides the reconstructed results into 4 parts (nose, mouth, forehead, and cheek) using region masks. Finally, it utilizes the ICP algorithm [1] for precise registration between prediction and ground truth and computes the corresponding average Normalized Mean Square Error (NMSE) for different face regions. The REALY test samples are divided into 2 parts, consisting of 100 frontal-view images and 400 side-view images. REALY has served as the benchmark for geometric evaluation by most stateof-the-art methods [10, 18, 59] . We propose a new Sketch-REALY benchmark based on REALY [9] to tailor it for sketch-to-3D-face reconstruction tasks, highlighting the performance of geometry facial sketches and achieves the best results consistent with the input sketch details. Note that these methods [17, 28, 37] all lack the ability to control the texture of the reconstructed 3D faces, while our S2TD-Face uniquely offers this capability.Impact of L sketch on Geometry. We investigate the effect of sketch-to-geometry loss L sketch for supervising geometry deformation. As shown in Tab. 1, based on our proposed geometry reconstruction framework and Sketch-REALY benchmark, we present results for both when the framework is applied independently (denoted as 'Ours (w/o L sketch )') and when combined with L sketch (denoted as 'Ours'). The former indicates our geometry reconstruction framework performs superior to existing state-of-the-art methods across most cases. The latter further shows that incorporating L sketch contributes to improved geometry deformation. The combination of L sketch with our geometry reconstruction framework further refines the accuracy of the reconstructed geometry.Impact of L sketch and Φ detail on Visual Quality. We use SSIM and PSNR to investigate the impact of L sketch and Φ detail on visual quality. Utilizing rendering techniques [34, 51] and sketch extraction methods [7, 56, 57] , we can acquire coarse or detailed geometry shading sketches (𝑺 𝑐 𝑡 𝑗 or 𝑺 𝑑 𝑡 𝑗 ) for the predicted results. These sketches are subsequently compared with the ground truth 𝑺 𝑡 𝑗 to compute the SSIM and PSNR scores. The test images are sourced from [9] . Quantitative results are shown in Tab.2. When neither L sketch nor Φ detail is involved, relying solely on Φ coarse for reconstruction leads to poorer visual quality. Combining Φ coarse with either L sketch or Φ detail individually results in improved visual quality, while employing L sketch and Φ detail together yields the best results. Note that comparing the second and third rows of Tab. 2, the combination of L sketch with Φ coarse even outperforms the combination of Φ detail with Φ coarse , further indicating the effectiveness of our proposed sketch-to-geometry loss L sketch in faithfully capturing the geometric information of the input sketch.We summarize two limitations of our methods. Firstly, although Fig. 1 and Fig. 6 show that S2TD-Face can generate controllable textures based on text prompts, these textures are all obtained from existing facial images through UV-mapping and UV-texturecompletion techniques. Low-frequency 3DMMs PCA albedo information is employed through the completion process, sometimes leading to noticeable visual differences at the fusion boundaries. Secondly, while our method can reconstruct high-fidelity geometry from face sketches of various styles, the geometry is derived from 3DMMs, resulting in the overall appearance still resembling real human faces when dealing with the animation styles. 
conclusions: 
 This paper proposes a method tailored to the sketch-to-3D-face task, named S2TD-Face. S2TD-Face is capable of reconstructing highfidelity topology-consistent detailed geometry from face sketches of diverse styles. It enables the controllable textures of 3D faces spanning cartoon, sculptural, and realistic facial styles based on text prompts. The contributions include an effective geometry reconstruction framework, a novel sketch-to-geometry loss for guiding geometry deformation, and a novel texture module for texture control based on text prompts. Extensive experiments show that the outstanding performance of our method surpasses existing stateof-the-art methods. The code will be publicly available. 
: 
 heavily on realistically shaded sketches as input [61] . On the other hand, texture plays a crucial role in accurately portraying facial appearance, highlighting the necessity for texture control within the sketch-to-3D-face process, while existing methods lack this capability. Furthermore, the absence of matching data between sketches and 3D faces makes it hard to train the framework. This paper proposes a method to reconstruct topology-consistent 3D faces with fine-grained geometry that precisely matches the input sketch and allows users to control the texture of reconstruction through text prompts, named S2TD-Face (Sketch to controllable Textured and Detailed Three-Dimensional Face). We introduce S2TD-Face in three parts: geometry reconstruction, training strategies, and texture control module.One straightforward approach to reconstructing 3D faces from sketches might involve first translating 2D sketches to 2D face images [13, 40, 52] , followed by utilizing existing 3D face reconstruction methods [17, 19, 26, 31, 59, 66, 67] to obtain the 3D faces. However, this approach suffers from the following shortcoming. It heavily relies on the cooperation of both the sketch-to-image and image-to-3D-face stages, where inherently sparse but important geometric information like dimples or wrinkles in sketches is often lost during the transformation process, as the two transformation steps are independent, leaving sketches unable to directly constrain the final 3D geometry. In contrast, S2TD-Face uses a direct and efficient geometry reconstruction framework. It firstly predicts the coefficients of 3DMMs [4, 5] from input sketches to reconstruct coarse geometry and then utilizes coarse geometry and sketches in UV space to generate displacement maps for detailed geometry.To ensure the framework reconstructs 3D detailed faces that accurately reflect the delicate features of the input, we introduce a novel sketch-to-geometry loss function to supervise both coarse and detailed geometry. This function combines differentiable rendering techniques [34, 51] to extract sketches of different styles from both geometry stages and compare them with ground truth sketches, guiding geometry deformation, as shown in Fig. 4 . To ensure the robustness of the reconstruction framework across different sketch styles, we generate 5 different types of sketches for each face image by using traditional filtering operators [6] and deep learning methods [56, 57] , as shown in Fig. 2 (a)-(e), with each sample randomly selecting a sketch type as input during training. The framework is trained by 2D signals like landmarks, segmentation, and perception features, as shown in Fig. 2 (f)-(h), without relying on the 3D face scanning data. Based on the widely-used REALY benchmark [9] , we tailor it to better suit sketch-to-3D-face tasks for geometry evaluation by transforming the test samples into different styles of sketches, conducting fair evaluation on state-of-the-art methods [17, 19, 20, 26, 28, 37, 54] . Extensive experiments indicate that our method significantly outperforms existing methods.S2TD-Face controls the texture of reconstructed 3D faces based on a text-image module, offering the following capabilities: it can search for suitable texture from a face library based on the text prompt, transform the selected texture information to UV space, and seamlessly apply the UV-texture to the reconstructed geometry. As shown in the first row of Fig. 1 , when the user provides a text prompt describing the desired texture, S2TD-Face can reconstruct 3D textured faces in styles such as 'Cartoon Boy' or 'Oil Painting'.In summary, the main contributions of S2TD-Face are as follows:• An effective framework for reconstructing 3D detailed highfidelity faces from sketches with a novel sketch-to-geometry loss, which accurately captures the local strokes of the input. • A novel texture control module for controlling the texture of the reconstructions, resulting in textured 3D faces with various styles ranging from cartoons to realistic appearances. • Extensive experiments show that our method achieves excellent performance and outperforms the existing methods. 
 
 Appendix:  


Decision: This paper presents a method named S2TD-Face for reconstructing the detailed 3D face from a single sketch image. 
The Reviewers did not find an agreement on the paper. Two of them tended toward accepting the paper, founding the proposed method with novel contributions. In my views, according to the reviews, the reasons for accepting the paper are prevailing on those for rejecting it. So my suggestion is to have the paper accepted as a poster.

Reviews: review: paper_summary: paper_summary: This paper proposes a method for reconstructing textured 3D faces from sketches and text prompt. The geometry reconstruction framework predicts the coefficients of 3DMMs from input sketches to reconstruct coarse geometry and then utilizes coarse geometry and sketches in UV space to generate displacement maps for detailed geometry. Then, a sketch-to-geometry loss is proposed to keep geometry consistent with the sketch. Finally, a text-image module is used to control the texture of reconstructed 3D faces based on text prompt. Extensive experiments show that it achieves excellent performance and outperforms the existing methods.
strength_weakness: strength_weakness: (1) This paper proposes a novel sketch-to-geometry loss to ensure both coarse and detailed geometry fidelity to the sketch input, and to maintain robustness across different sketch types.
(2) A novel texture control module for controlling the texture of the reconstructions, resulting in textured 3D faces with various styles.
limitations: limitations: (1) The five styles of sketches in figure 2 all appear to be synthetic, and I'm concerned about how well such training strategies will work for simple hand-drawn or stick-drawing style sketches. Additionally, reconstructions based on 3DMM may result in generated faces that lean towards realistic human faces, potentially lacking in representation of abstract styles.
(2) The qualitative comparison experiment selected in Figure 7 is limited, and such comparisons are somewhat inappropriate. Perhaps it would be better to select some more recent works as comparison baselines.
(3) Using CLIP text-image matching for semantic-level alignment might result in some details from the input sketches not being reflected in the textures, causing the synthesized final face to lose some information from the input sketch (e.g. the wrinkles on the forehead).
paper_summary: paper_summary: The paper presents a method named S2TD-Face for reconstructing the detailed 3D face from a single sketch image. Unlike existing approaches that rely on multiple transformation steps (e.g., two stages manner: i. mapping sketch image to 2D picture; ii. reconstruction 3D face geometry from 2D picture.), S2TD-Face directly predicts the coefficients of 3DMMs to reconstruct coarse geometry and then uses the coarse geometry and sketches in UV space to generate displacement maps for detailed face geometry. Besides, the method introduces a novel sketch-to-geometry loss function to supervise both coarse and detailed geometry reconstruction. 

Extensive experiments show that S2TD-Face outperforms existing methods.  The ablation study as shown in Tab.1 also reveals the effectiveness of the proposed sketch loss.
strength_weakness: strength_weakness: 1. Effective geometry reconstruction: S2TD-Face accurately captures the local details (e.g., wrinkles in the last row in Fig.1) of the input sketch. It achieves this by predicting 3DMM coefficients (mentioned as coarse geometry reconstruction) and optimizing a UV space displacement map.
2. Texture control module: S2TD-Face offers a texture control module that allows users to control the texture of the reconstructed 3D faces based on text prompts. This enables the generation of 3D faces with various styles ranging from cartoons to realistic appearances.
3. Superior performance: Extensive experiments show that S2TD-Face achieves excellent performance and outperforms existing methods on the Sketch-REALY benchmark. The qualitative results (e.g.Fig.6) show that the proposed method can capture the high-frequency details in the input sketch image.
limitations: limitations: 1. Texture generation limitations: The textures generated by S2TD-Face are obtained from existing facial images (i.e. FFHQ). First, they find the closest 2D image picture in a predefined library (such as FFHQ) via prompts and image embedding matching in the CLIP embedding space.  Then, they use a texture reconstruction method [59] to obtain the texture map in the PCA space of the parametric 3D face model ( 3DMM). To alleviate the bias between the 2D image and the texture PCA space, they fuse the texture map in PCA space and the image corresponding UV space texture map.  This process may result in noticeable visual differences at the fusion boundaries. Besides, the diversity of the pre-defined image library limits the diversity of the generated texture map. 

2. Realistic appearance limitation: The geometry reconstructed by S2TD-Face is derived from 3DMMs, which means that the overall appearance still resembles real human faces. This may limit the ability to handle animation styles that require non-human facial features.
paper_summary: paper_summary: The authors propose a sketch-to-3D face method called S2TD-Face. S2TD-Face is able to reconstruct detailed geometries from facial sketches, and utilize CLIP to control the texture of the face. Qualitative and quantitative experiments illustrate the effectiveness of this method.
strength_weakness: strength_weakness: 1. This paper is clearly written and easy to understand

2. The visualization quality of this paper is quite high.
limitations: limitations: 1. I believe the primary goals of facial sketch reconstruction should be: 1) to recreate details challenging to capture in RGB images, such as wrinkles; 2) to maintain smoothness in the absence of these facial detail lines; and 3) to ensure the geometry of the reconstruction closely matches the contour lines of the sketches. However, the results presented in the paper fall short in these aspects.

2. Methodologically, the only technical contribution of this work appears to be the L_{sketch} loss function.

3. Is the texture control model truly necessary? Why not simply use SD and ControlNet?

4. Experimentally, the more appropriate baseline should be ControlNet + HRN, a method set that better recovers facial geometry in terms of contour lines compared to the proposed methods.

5. The authors utilized the approach outlined in Reference[59] to give additional supervision to the method, which likely led to their superior face alignment performance on the RELAY Benchmark.
paper_summary: paper_summary: The authors propose to reconstruct 3D detailed faces based on sketches and textual prompts. To enhance the high-fidelity geometry, a two-stage reconstruction is used to generate coarse 3d and detailed 3d respectively. In order to select a suitable texture, the input prompt text is used to select the most similar texture from the texture library based on the matching of features. Finally, textures and detailed geometries are blended to obtain a controlled 3D face. Experiments are conducted on sharing and line sketches in Sketch-REALY to verify the effectiveness.
strength_weakness: strength_weakness: + The idea is intuitive to combine a 3D geometry and prompt-driven texture for the controlled final 3D reconstruction. 
+ The motivation for using sketch-to-geometry loss to directly generate 3D geometry directly from sketch is clear.
+ The paper is well-written, making it easy to follow.
+ The quantitative results for 3D face reconstruction from sketch seem promising. But unsure whether customized prompts are used for each sketch.
limitations: limitations: 1. Each technology component draws on existing methodologies. It is not clear what is new in the proposed method. 
2. The control using prompts is not strong enough to reflect the final 3D face. What if the selected texture is far away from the generated 3D geometry? If the prompt is not accurate (boy->girl), what is the generated 3D face? The authors should introduce the pool of all input prompts and explore the impact of using different prompts in different granularity for controlling the 3D face.
3. How to ensure that landmarks are well detected in large poses or simplified sketches where key components (such as the second example face in Figure 1) are not even visible.
4. Experiments should be performed on at least one sketch benchmark other than the sketch-REALY benchmark. Many hyperparameters need to be adjusted and which ones are sensitive to the final result? Any suggestions if the proposed method is applied to different training data sets?
5. The impact of sketches in different granularity on 3D generation should also be discussed. What is the potential of the proposed method to handle simplified sketches?

More comments:

a) Why don't the authors train coarse geometries and detailed end-to-end training? Is it unstable to train the two together?

b) The authors should report the computational cost of the module (b-d) in Figure 3 especially the texture control module which uses a texture library of 1000 images for matching. What were the main considerations in building this library?

c) Since the 3D geometry construction module is based on 3DMM in the proposed framework? Have the authors tried neural radiation fields as an alternative?

d) What are the differences and connections between the proposed method and these two related works? 

- [1] Gao, L., Liu, F.L., Chen, S.Y., Jiang, K., Li, C., Lai, Y. and Fu, H., 2023. SketchFaceNeRF: Sketch-based facial generation and editing in neural radiance fields. ACM Transactions on Graphics, 42(4).

- [2] Sun, Y., Wu, Q., Zhou, H., Wang, K., Hu, T., Liao, C.C., Miyafuji, S., Liu, Z. and Koike, H., 2023, October. Make Your Brief Stroke Real and Stereoscopic: 3D-Aware Simplified Sketch to Portrait Generation. In Proceedings of the 25th International Conference on Multimodal Interaction (pp. 388-396).