ShowMaker: Creating High-Fidelity 2D Human Video via Fine-Grained Diffusion Modeling

Authors: quanwei  yang
jiazhi  guan
kaisiyuan  wang
lingyun  yu
wenqing  chu
hang  zhou
zhiqiang  feng
haocheng  feng
errui  ding
jingdong  wang
hongtao  xie
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 10:41:19

Abstract: Although significant progress has been made in human video generation, most previous studies focus on either human facial animation or full-body animation, which cannot be directly applied to produce realistic conversational human videos with frequent hand gestures and various facial movements simultaneously.
To address these limitations, we propose a 2D human video generation framework, named ShowMaker, capable of generating high-fidelity half-body conversational videos via fine-grained diffusion modeling.
We leverage dual-stream diffusion models as the backbone of our framework and carefully design two novel components for crucial local regions (i.e., hands and face) that can be easily integrated into our backbone.
Specifically, to handle the challenging hand generation caused by sparse motion guidance, we propose a novel Key Point-based Fine-grained Hand Modeling module by amplifying positional information from raw hand key points and constructing a corresponding key point-based codebook. 
Moreover, to restore richer facial details in generated results, we introduce a Face Recapture module, which extracts facial texture features and global identity features from the aligned human face and integrates them into the diffusion process for face enhancement. 
Extensive quantitative and qualitative experiments demonstrate the superior visual quality and temporal consistency of our method.

Main paper: 
 introduction: 
 The recent advancement of generative models [5, 20, 10, 34, 31] has significantly propelled digital human technology, which is widely applied in business, education, and multimedia entertainment. Empowered by these generative models, numerous works [28, 38, 3, 47, 13, 30, 4, 22, 14, 43, 6, 15, 17, 32, 21, 33, 49, 44] have given their priority to 2D video synthesis to human generation. Though most of them focus on the head region, few studies [15, 17, 32, 21, 33, 49, 35, 40, 23] attempt to generate full-body videos by animating a reference image with a sequence of driving motion signals via a warping paradigm, but their results fall short in terms of both generation quality and temporal consistency. However, the demand for creating high-fidelity 2D avatars under more challenging conversational scenarios (e.g., TV shows and stand-up comedy) is increasing rapidly, which cannot be fulfilled with such a fidelity state.separately model the textural information from the reference image and the motion information from dynamic 2D skeletons [45] or 3D reconstructions [26, 50] . Effective interaction is also achieved between the two streams via widely used cross-attention [37] . Despite their captivating performance in full-body animation, we identify three major challenges that require rethinking. a) Creating a human-like conversational avatar requires a holistic synthesis of the human body. Though concurrent studies [16, 51] leverage 3D body reconstruction, which contains rich depth and shape information in face and hand regions, it suffers from occasional incorrect pose estimation for hands or temporal jittering problems. In addition, tedious pre-processing and time-consuming per-frame optimization [46] are required, which cannot be applied in a user-friendly system. We find using 2D representations more stable and efficient. b) It is quite intricate to generate human hands with sparse representations. Human hand regions occupy only a limited number of pixels in the original video frame. This easily leads to unstable hand synthesis with blurry texture or incorrect shape. Naive designs in [1, 12] cannot achieve detailed texture synthesis and delicate control in such local regions (e.g., faces and hands). c) Facial identity preservation is another problem that has not been well investigated. It becomes particularly challenging for these previous works [1, 12] due to the strong entanglement between identity information and 2D driving signals, which inevitably leads to identity degradation during cross-identity animation. The recent study [16] attempts to involve more controllable signals (i.e., 3D morphing parameters) for face enhancement, but the identity preservation performance is still not satisfying enough.To tackle the above challenges, in this paper, we propose a holistic human video generation framework named ShowMaker, capable of generating an expressive conversational human video with fine-grained modeling conditioned solely on 2D key points. We adopt a similar dual-stream architecture as in [1, 12, 42, 51] as the backbone of our framework and introduce two new designs. We first introduce a novel Key Point-based Fine-grained Hand Modeling module to reliably recover hand regions in detail from sparse driving guidance. Our key insight is to leverage resolution-independent representations (i.e., the coordinates of the hand key points) to provide stable structure guidance. In terms of hand texture, we novelly design a key point-based hand texture codebook equipped with a set of learnable basis vectors, which has a one-to-one correspondence with the hand key point topology. Specifically, we predict a set of weights from the hand key points of each hand and produce key point-based hand textural compensation through the weighted combination, which can be directly injected into the diffusion process via cross-attention. Furthermore, to improve the facial region quality of the target subject, we propose a Face Recapture module to construct comprehensive face representations by performing structure encoding, texture encoding, and identity encoding simultaneously. This multi-level encoding strategy can significantly alleviate the identity leakage issue during cross-identity animation on 2D avatars, which is not available in other comparative methods. Extensive quantitative and qualitative experiments demonstrate that our proposed method can synthesize 2D human video with better visual quality and more accurate body movements, especially hand gestures.The main contributions of this paper are summarized as follows: (1) We propose a novel holistic human video generation framework with fine-grained modeling, named ShowMaker for creating 2D human conversational videos conditioning on 2D key points. (2) We propose a Key Points-based Fine-grained Hand Modeling module, which achieves robust hand synthesis via a key point-based codebook. (3) We propose a Face Recapture module, which can effectively recover richer facial details and recapture the identity of the target subject. 
related work: 
 Talking Head Generation. The goal of talking head synthesis is to generate realistic face videos based on a driving video or audio. Depending on the driving signal, talking head synthesis can be divided into video-driven methods and audio-driven methods. Video-driven methods [11, 48, 3, 30, 4, 43] aim to establish the mapping between the reference face and the driving head motion, such as head/mouth movements, expressions, etc. For instance, DaGAN [11] learns to face depth maps and 3D facial geometry in a self-supervised manner to estimate motion fields. While MetaPortrait [48] estimates the warping flow between the driving and reference face through predefined dense facial key points. Audio-driven methods [38, 47, 13, 22, 14, 6, 41] focus on ensuring the generated mouth movements are synchronized with the given audio. For example, Wav2lip [28] edits the mouth area of the face video based on the input audio signal and employs a discriminator to ensure lip-sync. Similarly, StyleSync [6] generates lip-sync face videos by combining audio information with masked facial spatial information in style space. Although these methods can produce high-quality face videos, they lack comprehensive pose movements and gestures, limiting their application scenarios.The human video synthesis task aims to generate a full-body human video with the reference person appearance and driving pose. According to the pipeline, these methods can be divided into two categories: implicit methods and explicit warp-based methods. For the first category, some early GAN-based algorithms [15, 17, 35] map or manipulate the reference appearance to the driving pose in the latent space, subsequently generating the target video. In recent years, significant progress has been made in human video generation by leveraging powerful diffusion models. Some approaches [39, 18, 42] integrate the reference appearance details with driving information through cross-attention mechanisms in the Denoising U-Net, resulting in enhanced generation quality. The warp-based methods [32, 21, 33, 49, 23, 40] warp the reference image or its features to the driving pose by various estimated flows, such as 2D optical flows, and 3D flow fields. In addition to the aforementioned methods, some efforts [27, 25] employ neural representations of 3D human mesh points in a canonical pose to model human appearance details, thereby achieving multi-view and temporal consistency in human videos. However, these methods primarily focus on appearance consistency and full-body pose accuracy, overlooking the importance of facial details and hand movements for video expressiveness and specific scenario suitability. 
method: 
 The overview of our proposed framework ShowMaker is shown in Fig. 1 , which can achieve highquality video-driven conversational avatar synthesis. In the following, we first define our task in Sec. 3.1 and provide an overview of our framework pipeline in Sec. 3.2. Then detailed explanations of our novelly designed modules are demonstrated in Sec. 3.3 and Sec. 3.4. Finally, the training strategies are introduced in Sec. 3.5.For a conversational video V , we leverage the pre-trained DWPose [45] to extract human body key points of all frames, including face, body, and hands, and then we paint them into 2D heatmaps P as driving pose. Given a reference image I r ∈ R H×W ×3 from a target person, and the driving poses P d = {P 1 , P 2 , . . . , P F } ∈ R F ×H×W ×3 , our task aims at generating target video sequence V = { Î1 , Î2 , . . . , ÎF } ∈ R F ×H×W ×3 with similar facial and body movements as in the driving poses P d . The entire generative process can be formulated as V = G (I r , P d ), where G represents the generative model.Fig. 1 provides an overview of our approach. We follow the recent works [1, 12, 42, 51] to adopt a dual-stream learning paradigm. The upstream branch (Green Area) processes the reference image, extracting the appearance information using a VAE encoder and Reference U-Net. In the downstream branch (Blue Area), the Pose Encoder takes the 2D pose heatmaps as input to extract human pose information, including face and hand. The subsequent Denoising U-Net effectively integrates the extracted appearance and driving information to predict noise intensity. Additionally, we introduce a novel Key Points-based Fine-grained Hand Modeling module to capture robust hand structure and texture features, and the comprehensive face representations are constructed by the Face Recapture module. The Denoising U-Net is derived from the pre-trained SD [31] , replacing self-attention with reference-attention and CLIP [29] cross-attention with face attention, where hand attention integrates hand features from the Key Points-based Fine-grained Hand Modeling module, and face attention incorporates face features from the Face Recapture module. Additionally, temporal attention layers are introduced to promote the temporal consistency of the generated images.Reference U-Net. Numerous studies have demonstrated that the U-Net network outperforms the CLIP image encoder in terms of appearance extraction. Firstly, the image resolution input to CLIP is limited to 224 × 224, which is insufficient for capturing detailed appearance information. Secondly, the features extracted by the CLIP image encoder are primarily oriented towards high-level semantic matching. We thus follow recent works [1, 12, 42, 51] to adopt a duplicate U-Net as our appearance extraction network named Reference U-Net, initializing its weights from the pre-trained SD. Specifically, as shown in Fig. 1 , the reference image is first encoded into the latent space via the VAE Encoder and then passed to the Reference U-Net. The feature map z r from each layer in Reference U-Net is utilized in the reference attention mechanism for appearance detail fusion. Initially, z r is repeated F times along the temporal dimension to form z r ∈ R B×F ×h×w×c to match the size of the denoising feature map. Here, B indicates the batch size.Pose Encoder. To reduce computational complexity, we employ a lightweight network as the pose encoder which consists of 8 convolutional layers initialized with Gaussian weights, with the final layer utilizing zero convolution. The pose information extracted by the pose encoder is then added to the noise latent and fed into the Denoising U-Net.Although the driving pose provides important guidance for hand synthesis, the hand regions only occupy a limited number of pixels. The convolution operation in the pose encoder and Denoising U-Net repeatedly downsamples the spatial size, which progressively weakens this guidance and results in unexpected structural and textural artifacts. Considering hand key points are clearly defined and cannot be constrained by the resolution of the hand region, we make our attempt to enhance this guidance by involving absolute coordinates of hand key points as additional inputs. Specifically, given the m key points of one hand K = [k 1 ; k 2 ; . . . ; k m ], we first use Fourier positional encoding [24] to map their coordinates into a high-dimensional space, which can capture subtle differences in hand gestures. Formally, the frequency function F in positional encoding is defined as:EQUATIONHere k ∈ R 2 is the coordinate value of a single hand key point, which has been normalized to [0, 1]. Notably, we retain the original coordinate value in Eq. 1 and we set L = 20 by default. By using the function F, the coordinates of hand key points K h ∈ R B×F ×2×2m can be mapped to high dimensional space F h ∈ R B×F ×2×2m * (2L+1) , which enables reliable enhancement on the guidance from the raw driving poses P d . Here, the first 2 denotes two hands (both left and right hand).Based on the enhanced hand structure guidance, we additionally design a key point-based hand texture codebook that achieves fine-grained hand texture synthesis. Particularly, as illustrated in Fig. 1 (c), the key point-based hand texture codebook consists of a set of learnable basis vectorsC hand = {γ i } m i=1 , γ i ∈ R d, which is built in the same topology as the m hand key points mentioned above. It is worth noting that though the left hand and right hand are symmetric in geometry, they share the same topology (i.e., the order of key points is the same). Moreover, these basis vectors are constrained to be orthogonal so that each basis vector represents a distinct hand texture pattern and only requires an emphasis on the texture modeling around its corresponding key point. Specifically, any two basis vectors γ i , γ j satisfy the following conditions:EQUATIONSubsequently, a self-attention layer followed by a linear projection layer is employed to predict the weights of hand texture patterns A ∈ R B×F ×2×m from F h . By applying a weighted combination on all the basis vectors, we obtain the key point-based hand texture compensation defined as:EQUATIONFinally, the output feature G h ∈ R B×F ×2×d is sent to the hand attention layers in the Denoising U-Net for cross-attention calculation. Additionally, to explicitly inject hand motion information into the denoising latent feature maps, we apply a hand mask M h to provide emphasis guidance on the hand regions, which can be expressed as:EQUATIONwhere latents denotes the denoising latent feature maps, W Q , W K , W V represent learnable weights for the cross-attention modules. Similar to the challenge with hand generation, producing a satisfactory face in human video synthesis is difficult.Most approaches rely on postprocessing strategies to address this issue, but these methods significantly increase training and inference costs. In this paper, we design a Face Recapture module to extract comprehensive face information and inject it into the face attention layer in the Denoising U-Net to improve the quality of the generated faces. Specifically, as shown in Fig. 2 , we enhance the face area of the generated image from two aspects: texture details and global identity. First, we use a face detection model to crop and align the face from the reference image. Next, the pre-trained VAE Encoder and face recognition model ArcFace are leveraged to extract the facial texture feature F tex and global identity feature F id , respectively. For the facial texture feature, we perform feature enhancement by combining a MLP and a self-attention layer. We then repeat these features F times along the temporal dimension to match the size of driving poses and produce F id ∈ R B×F ×1×d id and F tex ∈ R B×F ×(hw)×d f . In the meantime, we crop out the corresponding facial pose information F d from the driving pose feature maps according to the face mask M f . Similarly, a MLP and a self-attention are proposed for feature enhancement on F d ∈ R B×F ×(hw)×d f . In order to equip F d with textural and identity information, we adopt two separate cross-attention blocks to embed texture and identity information into the F d , respectively. Specifically, by taking F d as query, F tex as key and value, we define the texture fusing process as G tex = Att tex (F d , F tex , F tex ). Similarly, by taking F d as query, F id as key and value, we can obtain the fused identity feature G id through G id = Att id (F d , F id , F id ). The comprehensive face information is obtained byEQUATIONFinally, G f ∈ R B×F ×(hw)×d f is injected into the face attention layer in the Denoising U-Net.Similarly, the face mask is utilized to constrain the latent feature maps after face attention:EQUATIONDuring the diffusion process, the random noise is continuously added to the real image until it reaches a state of Gaussian noise. Conversely, the generation process within the diffusion model is the inverse of the diffusion process, which takes random Gaussian noise as input and generates real images through gradual denoising. In the training process, the diffusion model leverages the Denoising U-Net to predict the added noise at various time steps and the optimization objective can be defined as:EQUATIONwhere c is the conditional features, ϵ θ represents the Denoising U-Net, z t is the denoising latent feature maps at timestep t.In our work, we adopt a two-stage training strategy to separately perform appearance modeling and temporal consistency modeling. The first stage is image-level training. During this stage, we set F = 1, the training goal is to accurately map the appearance details from the reference image to the driving pose. The VAE encoder and CLIP Image encoder are fixed, and the rest of the network except for the temporal attention are all trainable. In terms of the second stage, the objective is to improve the temporal coherence of the generated frames. During this stage, we only perform training on the temporal attention layer with all other network weights fixed and the weights for the temporal attention layers in the Denoising U-Net are initialized by the pre-trained AnimateDiff [7] . The loss function can be reformulated as:EQUATIONwhere z r represents the appearance feature maps extracted by Reference U-Net, G h represents the hand movement feature maps from the Key Point-based Fine-grained Hand Modeling module, and G f is the face feature maps obtained by the Face Recapture Module. Additionally, M h and M f denote the masks for the face and hand regions, respectively.It is worth noting that in order to focus on the face and hand area generation, in the later phase of the first training stage, we adopt the hand mask and face mask to calculate the L 1 loss of the corresponding area as the final loss every 10 iterations. 
experimental settings: 
 Datasets. In order to verify the effectiveness of the proposed method, we select the videos of two actors, Seth and Oliver, in the talkshow [46] dataset for training and testing. In addition, to enrich the diversity of characters and hand movements, we record videos of seven people in the indoor scene.The video length of each person is about 10 minutes and we divide the videos into multiple clips of Comparison Methods. We select several state-of-the-art approaches, including TPS [49] , Disco [39] , AnimateAnyone [12] , MagicAnimate [42] , MagicPose [1] , Make-Your-Anchor [16] as our comparison methods. TPS is a general animation model based on warping operation, which adopts GAN as the backbone, the remains are diffusion-based human video generation models. Among them, Make-Your-Anchor provides the specific person generation model. We finetune these models on the same training dataset with the official codes and pre-trained models.Metrics. We comprehensively measure the quality of generated images from pixel space and feature space using SSIM, PSNR, and FID [8] and adopt FVD [36] to verify the temporal consistency of generated videos. In addition, the motion accuracies of body (L body ), face (L f ace ), and hand (L hand ) are measured by calculating the mean Euclidean distance between the key points of the generated images and real images. TPS is a general object animation approach and struggles with complex human poses and hand postures, resulting in the poorest overall generation quality. Disco, AnimateAnyone, MagicAnimate and MagicPose focus on generating coarse-grained human poses without fine-grained modeling of hands and faces, leading to lower generation quality, particularly in hand gesture accuracy. Note that Make-Your-Anchor [16] only provides a pre-trained model on Seth data and thus we also report the experimental results on the Seth for a fair comparison.It is observed that our method outperforms [16] on all metrics.Qualitative Comparison. For qualitative comparison, we conduct two experimental settings, including self-identity and cross-identity driven. The self-identity driven comparison results are shown on the left of Fig. 3 and 4 , where obvious artifacts are marked with red dotted boxes. Our method is able to generate accurate gestures and high-quality hand details while other approaches fail to convey complex gestures. In addition, face identity is well-preserved in our results, while other methods lead to unnatural face shapes, textures, and identity. Note that cross-identity driven is more challenging and our method still achieves high-fidelity gesture details and satisfactory image quality as indicated in the right of Fig. 3 and 4 . The hand areas of other results are blurry and unrealistic. In summary, our method makes use of hand and face information through the well-designed Key Points-based Fine-grained Hand Modeling module and Face Recapture module, enhancing the video quality which could meet complicated requirements in conversational scenarios.Human Evaluation. We conduct a user preference study to evaluate the performance of the proposed framework. There are 21 samples and 15 human voters in total. For each sample, we present six video results generated with ShowMaker and other SOTA methods to the human voter in a random order. The human voters are required to estimate the video results in three aspects: a) Motion Accuracy: Does the video accurately reproduce the motion in the driving video? b) Appearance Consistency: Does the subject in the video have a consistent appearance with the reference image? c) Temporal Consistency: How is the temporal coherence of this video? The rating score ranges from 1 to 5 and higher scores indicate better preference. As shown in Tab. 3, our method achieves the highest scores compared with its counterparts, which demonstrates that our method is preferred by a significant margin on motion accuracy and appearance consistency.To verify the contributions of different components, we train three variants by removing the Key Points-based Fine-grained Hand Modeling module (HM), the Face Recapture module (FR), and two-stage training (Stage 2), respectively. For quantitative results, we report the FVD and motion accuracy metrics in Tab. 2. It is observed that the HM and FR could enhance gesture accuracy and face quality by a significant margin. The twostage training brings about notable improvements in FVD which indicates better temporal consistency. Additionally, we present qualitative comparisons to verify the effectiveness of the proposed HM and FR module. As shown in Fig. 5 , the generated hands are more satisfactory, and the face shapes and textures are realistic and clear with the proposed modules. 
conclusion and discussion: 
 Conclusion. In this paper, we propose the framework ShowMaker, which achieves high-fidelity 2D human video synthesis with two novel designs to achieve fine-grained diffusion modeling. We first propose a Key Point-based Fine-grained Hand Modeling module for robust and fine-grained hand synthesis which takes advantage of 2D hand key points and a key point-based codebook. To further reconstruct the facial details and identity information, we introduce a Face Recapture module, which effectively equips the structure information with detailed textural information and global identity.Quantitative and qualitative evaluation has indicated the superiority of our framework beyond the existing approaches.Limitations Despite the success of our framework, we also recognize some limitations during the exploration. Firstly, our Key Point-based Fine-grained Hand Modeling module can robustly manage hand synthesis with occasional incorrect hand gestures. However, DWPose [45] suffers from performance degradation when handling videos with severe motion blur leading to considerable perturbation in the driving signal, which inevitably results in unexpected artifacts in our results. Secondly, our framework produces less satisfactory results when handling challenging cases, such as reflection on glasses. These will be part of our future work.Broader Impact. Our method focuses on synthesizing realistic 2D avatars with rich facial expressions and complex hand gestures, which is intended for developing digital humans under more daily scenarios like TV shows. However, it may also be misused for some malicious purposes on social media, which leads to negative impacts on the whole society. Therefore, we will make our efforts to strictly oversee the dissemination of our models as well as the resulting content and also restrict access solely to research-oriented demands. We believe that the proper use of this technique will enhance positive societal development in both machine learning research and daily life.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: We will provide the code in the camera-ready version. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: We describe safeguards for responsible release of data or models by requiring that users adhere to usage guidelines. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We explicitly mention and properly respect the license and terms of use in the paper and the creators or original owners of assets used in the paper are properly credited. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Pose Encoder Structure Fig. 6 shows the detailed network structure of the pose encoder.Tab. 4 gives the number of training and testing clips for the indoor recording dataset and talkshow dataset. The length of each clip is 2-15 seconds.When preparing the training data, we first produce the DWPose results from each frame and set the center of shoulders as the cropping center. Then we crop the original video frame using an adaptive cropping size, where the cropping size is designed as a fixed ratio (we set it to 2.65) of the shoulder width. This operation forces the human body to lie in a roughly consistent position. Finally, all cropped images are resized to 512 × 512.In the inference stage, for the cross-identity driven setting, we further bridge the gap between body shapes by scaling the driving poses to match the reference pose. The scaling ratios of width and height are defined as W r /W d , and H r /H d , where W and H represent the shoulder width and the height of the human body, respectively. The shoulder width is measured as the Euclidean distance between the left and right shoulder key points, while the body height is determined by the Euclidean distance from the nose key point to the pelvis key point.To further verify the effectiveness of the proposed Key point-based Fine-grained Hand Modeling (HM), we show more ablation experiments in Fig. 7 , where Vanilla refers to removing the entire HM module from our proposed ShowMaker, Hand image refers to adopting the cropped hand image to extract texture features through the VAE encoder and then feeding it into the hand attention, and w/o positional encoding refers to not using positional encoding in the HM module. It can be seen that using the hand image as compensation information has no obvious effect on improving the hand texture and structure, As for HM, positional encoding can significantly improve the high-frequency details and structural accuracy of the generated hands.Fig. 8 shows two generated video sequences, each sequence contains 3 adjacent video frames. It can be seen that the generated video frames at different times have consistent appearance and no temporal texture changes occur.Fig. 9 shows two challenging sample generations. In the example on the left, the driving pose is extracted from a female video, while the reference image is a male, and the driving gestures and the reference gesture are very different. In the example on the right, the reference image is not a common frontal image instead of a side image. Our proposed ShowMaker can still generate target frames well in these two cases, which shows that our method has satisfied robustness. • The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: Reviewers mostly agree that the proposed system is well-designed, and provides good results for the less studied problem of half body generation. The authors' rebuttal addressed most of the reviewers' concerns who in many cases increased their score. Overall there seems to be consensus for acceptance.

Reviews: review: paper_summary: paper_summary: This paper represents an early attempt to advance the field of 2D digital human synthesis in real-world scenarios, extending the traditional talking head task to include more complex body movements, such as hand gestures. The authors present a novel framework utilizing dual-stream diffusion models for full-body video synthesis, complemented by two specialized modules designed for fine-grained detail modeling. Specifically, the proposed Key Point-based Fine-grained Hand Modeling effectively addresses the challenge of hand gesture generation through the integration of a key point-based codebook and high-dimensional positional encoding. Additionally, the Face Recapture module enhances facial textures and identity consistency, enabling the generation of more realistic and vivid 2D avatar videos.
strength_weakness: strength_weakness: There are few mistakes and typos in the paper:
1) The Face Recaption module takes “Reference Pose” as input in Fig 1 (d), which is not mentioned in Fig 2 and Sec 3.3. The description of Face Recaption module should be the same throughout the full manuscript.

2) Eq (5) and Eq(6) are nearly the same. The authors can merge them into one equation.
questions: questions: 1) I understand the authors adopted the positional encoding to enhance the information provided by the sparse hand key points, but I’d like to see a further analysis of the positional encoding used in the Hand Modeling module. Since it cooperates with the key point-based codebook, I am wondering if a well-trained codebook is already enough for hand restoration.

2) Have the authors adopted any alignment strategy in cross-driving experiments? In my understanding, cross-driving animation usually suffer from mis-alignment due to the subject differences.
limitations: limitations: The authors adequately addressed the limitations and negative societal impact.
paper_summary: paper_summary: This paper proposes a 2D human video generation framework called ShowMaker, which can generate half-body conversational videos based on 2D keypoints as motion conditions. ShowMaker includes a texture enhancement module for the face and hands, and demonstrates some effectiveness.
strength_weakness: strength_weakness: 1. The experimental settings lack clarity, especially regarding the balance of training duration for each ID in the testing examples. It is unclear how much training data corresponds to each ID in the visualized examples, and more details are needed.
2. The method proposed by the authors has some improvements in details compared to the similar method Make-an-Anchor. However, from Figure 4, it seems that the generation of facial details is unstable, resulting in some color changes (such as in the beard area), which raises concerns about the facial generation capability of the proposed method.
3. The ablation studies only excluded the face modeling and hand modeling modules. However, introducing region images to enhance the generation results is a straightforward approach in human body generation tasks and could serve as a baseline for comparison.
questions: questions: 1. In Table 1, it is unclear what "One ID" represents. It is also unclear whether the training set only contains data for the testing IDs. Additionally, it is unclear whether the compared methods, such as Animate Anyone and TPS, were also trained on the testing IDs. More details are needed to clarify these points.
2.  Are the IDs included in the training set and testing set are completely identical?
limitations: limitations: Please refer to the questions and weaknesses.
paper_summary: paper_summary: This paper proposes a novel conversational human video generation framework named showmaker based on the fine-grained diffusion model. This paper proposes a novel Key Point-based Fine-grained Hand Modeling module and construct a key point-based codebook to handle the challenging hand generation. Meanwhile, this paper extracts facial texture features and global identity features from the aligned face and integrates them into the diffusion process for face enhancement. Sufficient experiments demonstrate the superiority of the proposed framework, and the generated human videos are of high quality.

I have read the comments of other reviewers and the response of the authors, I tend to give a accept score.
strength_weakness: strength_weakness: 1.Although the author compares the proposed method with talking head synthesis in the demo, it would be more beneficial to include this comparison in the main text or as an appendix, as it is crucial for understanding the contribution.
2.The presence of several writing errors in the paper impedes the correct comprehension of its content.
3.The details regarding some network structures and inference processes, such as the pose encoder, are insufficiently described and require further elaboration.
questions: questions: 1. The input for the Face Recapture Module in Figure 1 incorporates a reference pose, whereas the input in Figure 2 lacks such a reference pose.
2. Shouldn't line 140 on page 3 be replicated F times along the temporal dimension?
3. What is the rationale behind selecting a VAE encoder as the texture extractor in the Face Recapture Module?
4. Could you elaborate on the shapes of Gh and Gf? Additionally, why was a discrete codebook chosen to represent hand information? What advantages does this design offer?
limitations: limitations: Authors adequately discuss the limitations and potential negative societal impact
paper_summary: paper_summary: This paper proposed a pipeline for 2D human motion retargeting, based on 2D key points and face encoding via fine-grained diffusion modeling.
strength_weakness: strength_weakness: 1. Missing recent baselines in the experimental comparison. E.g. Diffusion-based method: MagicPose from ICML 2024, GAN-based methods: TPS from CVPR2022 etc.

2. The novelty of the proposed method is limited. It seems to me that the only novel component of this paper compared to previous works is the face encoding and key point-based hand modeling. The ReferenceNet for identity control and Pose Encoder for pose control are all from previous works (ReferenceNet from Animate Anyone, MagicAnimate, MagicPose, Champ, and ControlNet from DisCo, MagicAnimate, Animate Anyone, MagicPose). Temporal Attention the same.

3. I would like to see the performance of the proposed method compared to baselines on the TikTok dataset since this is a public dataset, and all previous works reported metrics on it. I'm not implying the authors are hiding something but I think it's necessary to follow previous settings and report the performance on the new dataset (talkshow).

4. Lack of Long Video Generation Results. The presented videos in the supplementary materials are very short compared to MagicAnimate's videos on TikTok dataset. It's necessary to provide longer visualization.

5. Lack of motion retargeting visualization while the condition of human DW-Pose is significantly different than the Human Pose in the reference image. The visualization in both supp and main paper shows only the cases where condition and reference image share similar human poses. This also raises the concern about face encoding and key point-based hand modeling: will such network design preserves human identity and ensure temporal coherence when the poses have a huge gap between reference and condition?
questions: questions: See weakness. Major questions as following:

1. Examples of Long Video Generation. Ideally, the generated video of the TikTok test set as MagicAnimate shows. (~at least 20 seconds or longer)

2. Motion retargeting visualization while condition and reference have quite different human poses. Please see the previous paper, MagicPose Figure 5,7,17, for reference.

3. Previous work, e.g., MagicPose, has shown that the model can synthesize the back of the human subject. Can this model do the same? Given that the front of the human is the reference image and another pose condition, how can the model learn to distinguish the front/back of the human from such conditions?

4. Can this model provide a satisfying result if the identity reference is a non-frontal image,e.g., the left or right side of the human?

5. Is there any visualization on out-of-domain motion retargeting? E.f. cartoon-style animation?

6. More detailed explanation of the novelty.
limitations: limitations: The authors include a limitation section.

Comments: comment: Thank you very much for your response, we are glad to address your concerns!
We greatly appreciate your response and the valuable insights. In our revised manuscript, we will update the discussion and comparison of MagicPose in the related work and experiments sections, respectively.  In terms of out-of-domain data, we agree with your suggestion and we will make efforts to demonstrate its generalization ability on multiple styles of conversational videos.
Thanks for the author's detailed response to my questions. My concern on visualization and experiments has been well-addressed. Hence, I'm raising my score. 

For out-of-domain data, I still suggest the authors explore training/finetuning the trained model on real-human data on **half-body 2D talking avatar** of cartoon-style videos since it would be an outstanding contribution to demonstrate the generalization ability, and it would be interesting to see the results.

For MagicPose ICML 2024, kindly remind that it's released around Sep 2023, which is close to Animate-Anyone/MagicAnimate, and it's a prior work rather than a concurrent one. But this is a tiny point since ShowMaker has already demonstrated better visualization quality for gestures.
My concerns about the the alignment strategy and the effect of position encoding have been well addressed in the rebuttal. I am willing to increase my score.
Thank you very much for your reply and insightful comments. Please let us know if you have any other questions or concerns, and we will respond to them carefully.
Thank you for the response, which has addressed the points I previously did not understand. I am inclined to maintain my rating.