ShowMaker: Creating High-Fidelity 2D Human Video via Fine-Grained Diffusion Modeling
Authors: Quanwei Yang
Jiazhi Guan
Kaisiyuan Wang
Lingyun Yu
Wenqing Chu
Hang Zhou
ZhiQiang Feng
Haocheng Feng
Errui Ding
Jingdong Wang
Hongtao Xie
Venue: NeurIPS 2024 poster (2024)
Keywords: human video generation; diffusion model
Primary area: diffusion_based_models

Abstract: Although significant progress has been made in human video generation, most previous studies focus on either human facial animation or full-body animation, which cannot be directly applied to produce realistic conversational human videos with frequent hand gestures and various facial movements simultaneously.
To address these limitations, we propose a 2D human video generation framework, named ShowMaker, capable of generating high-fidelity half-body conversational videos via fine-grained diffusion modeling.
We leverage dual-stream diffusion models as the backbone of our framework and carefully design two novel components for crucial local regions (i.e., hands and face) that can be easily integrated into our backbone.
Specifically, to handle the challenging hand generation caused by sparse motion guidance, we propose a novel Key Point-based Fine-grained Hand Modeling module by amplifying positional information from raw hand key points and constructing a corresponding key point-based codebook. 
Moreover, to restore richer facial details in generated results, we introduce a Face Recapture module, which extracts facial texture features and global identity features from the aligned human face and integrates them into the diffusion process for face enhancement. 
Extensive quantitative and qualitative experiments demonstrate the superior visual quality and temporal consistency of our method.

ShowMaker: Creating High-Fidelity 2D Human Video via Fine-Grained Diffusion Modeling
Quanwei Yang Wenqing Chu 3
Jiazhi Guan 2 Kaisiyuan Wang Lingyun Yu
Hang Zhou 3 Zhiqiang Feng Haocheng Feng
Errui Ding 3 Jingdong Wang Hongtao Xie 1+
University of Science and Technology of China Tsinghua University Department of Computer Vision Technology (VIS), Baidu Inc.
Abstract
Although significant progress has been made in human video generation, most previous studies focus on either human facial animation or full-body animation, which cannot be directly applied to produce realistic conversational human videos with frequent hand gestures and various facial movements simultaneously. To address these limitations, we propose a 2D human video generation framework, named ShowMaker, capable of generating high-fidelity half-body conversational videos based on 2D key points via fine-grained diffusion modeling. We leverage dual-stream diffusion models the backbone of our framework and carefully design two novel components for crucial local regions (i.c., hands and face) that can be easily integrated into our backbone. Specifically, to handle the challenging hand generation caused by sparse motion guidance, we propose a novel Key Point-based Fine-grained Hand Modeling module by amplifying positional information from raw hand key points and constructing a corresponding key point- based codebook. Moreover, restore richer facial details in generated results, we introduce Face Recapture module, which extracts facial texture features and global identity features from the aligned human face and integrates them into the diffusion process for face enhancement. Extensive quantitative and qualitative experiments demonstrate the superior visual quality and temporal consistency of our method."
Introduction
The recent advancement of generative models 5 21 36 33 has significantly propelled digital human technology, which is widely applied in business, education, and multimedia entertainment.
Empowered by these generative models, numerous works (30 40 3
52
15
48
16
18
34 23 35| 54 49 have given their priority to 2D video synthesis to human generation. Though 34
most of them focus on the head region, few studies [16] I8 22 generate full-body videos by animating a reference image with sequence of driving motion signals a warping paradigm, but their results fall short in terms of both generation quality and temporal consistency. However, the demand for creating high-fidelity 2D avatars under more challenging conversational scenarios (e.g.. TV shows and stand-up comedy) increasing rapidly, which cannot be fulfilled with such fidelity state.
23
35
54
37
45
25
attempt to
Recently, efforts have been made investigate architectures built upon pre-trained diffusion models for controllable human body animation II 3 47). These methods adopt dual-stream design to
Work done during an internship Baidu Inc.
Correspondence to: htxieduste. edu. cn; wangkaisiiyunn@aaadduuu Project webpage: https://mmmuvoi.iitub...oo iahko
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

separately model the textural information from the reference image and the motion information from
dynamic 2D skeletons [50] or 3D reconstructions [28 55
Effective interaction also achieved between the two streams via widely used cross-attention 39 Despite their captivating performance
in full-body animation, we identify three major challenges that require rethinking. a) Creating a human-like conversational avatar requires a holistic synthesis of the human body. Though concurrent studies [ leverage 3D body reconstruction, which contains rich depth and shape information in face and hand regions, it suffers from occasional incorrect pose estimation for hands or temporal jit- tering problems. In addition, tedious pre-processing and time-consuming per-frame optimization [5T] required, which cannot be applied in user-friendly system. We find using 2D representations more stable and efficient. b) It is quite intricate to generate human hands with sparse representations. Human hand regions occupy only a limited number of pixels in the original video frame. This easily leads to unstable hand synthesis with blurry texture or incorrect shape. Naive designs in [T [3 cannot achieve detailed texture synthesis and delicate control in such local regions (e.g., faces and hands). <) Facial identity preservation is another problem that has not been well investigated. It becomes partieularly challenging for these previous works | [3] due the strong entanglement between identity information and 2D driving signals, which inevitably leads to identity degradation during cross-identity animation. The recent study (17] attempts to involve more controllable signals (i.e., 3D morphing parameters) for face enhancement, but the identity preservation performance is still not satisfying enough.
To tackle the above challenges, in this paper, we propose a holistic human video generation framework named ShowMaker, capable of generating an expressive conversational human video with fine-grained modeling conditioned solely on 2D key points. We adopt a similar dual-stream architecture as in
13 56 the backbone of our framework and introduce two new designs. We first introduce novel Key Point-based Fine- graine Hand Modeling module to reliably recover hand regions in detail from sparse driving guidance. Our key insight is to leverage resolution-independen representations (i.e., the coordinates of the hand key points) to provide stable structure guidance. In terms of hand texture, we novelly design key point-based hand texture codebook equipped with set of learnable basis vectors, which has a one-to-one correspondence with the hand key point topology. Specifically, we predict a set of weights from the hand key points of each hand and produce key point-based hand textural compensation through the weighted combination, which can be directly injected into the diffusion process via poss-attention. Furthermore, improve the facial region quality of the target subject, we propose a Face Recapture module to construct comprehensive face representations by performing structure encoding, texture encoding, and identity encoding simultaneously. This multi-level encoding strategy can significantly alleviate the identity leakage issue during cross-identity animation on 2D avatars, which is not available in other comparative methods. Extensive quantitative and qualitative experiments demonstrate that our proposed method can synthesize 2D human video with better visual quality and more accurate body movements, especially hand gestures.
The main contributions of this paper are summarized as follows: (1) We propose a novel holistic human video generation framework with fine-grained modeling, named ShowMaker for creating 2D human conversational videos conditioning on 2D key points. (2) We propose a Key Points-based Fine-gran grainer Hand Modeling module, which achieves robust hand synthesis via a key point-baser codebook. (3) We propose Face Recapture module, which can effectively recover richer facial details and recapture the identity of the target subject.
2 Related Work
Talking Head Generation. The goal of talking head synthesis is to generate realistic face videos based on driving video or audio. Depending on the driving signal, talking head synthesis can be divided into video-driven methods and audio-driven methods. Video-driven methods [43] 3] 32
[4]
48 42)
aim to establish the mapping between the reference face and the driving head motion, such as
head/mouth movements, expressions, etc. For instance, DaGAN [12] learns to face depth maps and 3D facial geometry self- superviedd manner to estimate motion fields. While MetaPortrait [53] estimates the warping flow between the driving and reference face through predefined dense facial key points. Audio-driven methods 40 52
6. 46 focus on ensuring the generated mouth movements are synchronized with the given audio. For example, Wav2lip 30 edits the mouth area of the face video based on the input audio signal and employs discriminator to ensure lip-syne. Similarly, StyleSync [6) generates lip-syne face videos by combining audio information with masked
24 15

---

facial spatial information in style space. Although these methods can produce high--uality face videos, they lack comprehensive pose movements and gestures, limiting their application scenarios.
Human Body Animation The human video synthesis task aims to generate a full-body human video with the reference person appearance and driving pose. According the pipeline, these methods can be divided into two categories: implicit methods and explicit warp-based methods. For the first category, some early GAN-based algorithms []6 18 37 map or manipulate the reference appearance to the driving pose in the latent space, subsequently generating the target video. In recent years, significant progress has been made human video generation by leveraging powerful diffusion models. Some approaches [44 19 47 integrrae the reference appearance details with driving information through cross- aesss-attntioon mechanisms
the Denoising U-Net, resulting enhanced 35 [54, [25 45 warp the reference image or its 23
generation quality. The busp-baeed methods [4
features to the driving pose by various estimated flows, such as 2D optical flows, and 3D flow fields. In addition to the aforementioned methods, some efforts (29] 27 employ neural representations of 3D human mesh points in canonical pose to model human appearance details, thereby achieving multi-view and temporal consistency in human videos. However, these methods primarily focus on appearance consistency and full-body pose accuracy, overlooking the importance of facial details and hand movements for video expressiveness and specific scenario suitability.
3 Method
The overview of our proposed framework ShowMaker is shown in Fig. [ which can achieve high- quality video-driven conversational avatar synthesis. the following, we first define our task in Sec. 3.1 and provide an overview of our framework pipeline in Sec. 3.2 Then detailed explanations
of our novelly designed modules are demonstrated in Sec. 3.3 and Sec. 3.4 Finally, the training
strategies are introduced in Sec. 3.5
3.1 Task Formulation
For a conversational video V, we leverage the pre-trained DWPose (50) to extract human body key points of all frames, including face, body, and hands, and then we paint them into 2D heatmaps P as driving pose. Given reference image 1. E 12 HxxXXx x3 from a target person, and the driving poses Pa
P2............ PF} xxxxxxxx our task aims at generating target video sequence V = (i,, l2...., ]F} E BRFxxxxXXX with similar facial and body movements as the driving poses Pa. The entire generative process can be formulated as V - = (Ir, Pa), where G represents the generative model.
3.2 Pipeline Overview.
Fig. provides an overview of our approach. We follow the recent works [3 47 56 to adopt dual-stream learning paradigm. The upstream branch (Green Area) processes the reference image, extracting the appearance information using VAE encoder and Reference U-Net. In the downstream branch (Blue Area), the Pose Encoder takes the 2D pose heatmaps as input to extract human pose information, including face and hand. The subsequent Denoising U-Net effectively integrates the extracted appearance and driving information to predict noise intensity. Additionally, we introduce a novel Key Points-based Fine- grningr Hand Modeling module to capture robust hand structure and texture features, and the comprehensive face representations are constructed by the Face Recapture module. The Denoising U-Net is derived from the pre-trained SD 333, replacing self-attention with reference-attention and CLIP [31] cross-attention with face attention, where hand attention integrates hand features from the Key Points-based Fine- sine-grained Hand Modeling module, and face attention incorporates face features from the Face Recapture module. Additionally, temporal attention layers are introduced to promote the temporal consistency of the generated images.
Reference U-Net. Numerous studies have demonstrated that the U-Net network outperforms the CLIP image encoder in terms of appearance extraction. Firstly, the image resolution input to CLIP is limited to 224 224, which insufficient for capturing detailed appearance information. Secondly, the features extracted by the CLIP image encoder are primarily oriented towards high-level semantic matching. We thus follow recent works
47 56) to adopt duplicate U-Net as our appearance extraction network named Reference U-Net, initializing its weights from the pre-trained SD.
3

---

(b) Appearance Encoding
Reference U-Net
Cropped Reference
Driving
(d) Face Encoding
VAE Encoder
Face-Kitention
Hand Attention
Reference Image
Recapture Module
Tomporal
Diffusion
Face Mask My
Pose
Decoder
Denoising U-Net
Driving Poses
Hand Mask Mr
Generated Frames
(c) Key point- based Fine-grained Hand Modeling
Malls
You- Min
Key Point-baased
Positional Encoding
Self-Att
Hand Texture Codebook
Driving Hand Key Points
Figure 1: Overview of our proposed framework ShowMaker: Our framework adopts a dual-stream design including Reference U-Net and Denoising U-Net, where the former takes reference image imput for appearance encoding and the latter takes noise latent and driving poses as input for diffusion processing. We further equip the backbone with Key Point-based Fine- grained Hand Modeling module and Face Recapture module for fine-grained avatar synthesis.
Specifically, as shown in Fig. I the reference image is first encoded into the latent space via the VAE Encoder and then passed to the Reference U-Net. The feature map z, from each layer in Reference U-Net is utilized in the reference attention mechanism for appearance detail fusion. Initially, is repeated times along the temporal dimension to form z, E BxFxxxxx to match the size of the denoising feature map. Here, B indicates the batch size.
Pose Encoder. reduce computational complexity, we employ lightweight network as the pose encoder which consists of 8 convolutional layers initialized with Gaussian weights, with the final layer utilizing zero convolution. The pose information extracted by the pose encoder then added to the noise latent and fed into the Denoising U-Net.
3.3 Key pasnt-based Fine-grained Hand Modeling
Although the driving pose provides important guidance for hand synthesis, the hand regions only occupy limited number of pixels. The convolution operation in the pose encoder and Denoising U-Net repeatedly downsamples the spatial size, which progressively weakens this guidance and results in unexpected structural and textural artifacts. Considering hand key points are clearly defined and cannot be constrained by the resolution of the hand region, we make our attempt to enhance this guidance by involving absolute coordinates of hand key points additional inputs. Specifically, given the m key points of one hand K = [k]; kz;...; ; kmm], we first use Fourier positional encoding [26] to map their coordinates into high-dimensional space, which can capture subtle differences in hand gestures. Formally, the frequency function F positional encoding is defined as:
F(k) [(sin (2**k). cos (2" .... sin (2--1 cos (g--1TTk) kt).
(I)
Here & E R² the coordinate value of single hand key point, which has been normalized to [0, 1]. Notably, retain the original coordinate value in Eq. and set L = 20 by default. By using the function F. the coordinates of hand key points K2, Rx can be mapped to high dimensional space F2, B x2x3m=(2L++1 which enables reliable enhancement on the guidance from the raw driving poses Pa. Here, the first denotes two hands (both left and right hand).

---

Based on the enhanced hand structure guidance, we additionally design a key point-based hand texture codebook that achieves fine-grained hand texture synthesis. Particularly, as illustrated in Fig. (c), the key point-hased hand texture codebook consists of set of learnable basis vectors Chand = }= 1= V Rd which built in the same topology as the m hand key points mentioned above. It is worth noting that though the left hand and right hand are symmetric in geometry, they share the same topology (i.e., the order key points is the same). Moreover, these basis vectors are constrained to be orthogonal that each basis vector represents distinct hand texture pattern and only requires an emphasis on the texture modeling around corresponding key point. Specifically, any two basis vectors Vi "i satisfy the following conditions:
ド 1 i=
0
Nia T5 >=
Subsequently, a self-attention layer followed by a linear projection layer is employed to predict the weights of hand texture patterns A E 188xxx2Xx from F.. By applying weighted combination on all the basis vectors, we obtain the key point-based hand texture compensation defined as:
G
Σ an
(3)
Finally, the output feature G2, E BB is sent to the hand attention layers in the Denoising U-Net for cross-attention calculation. Additionally, to explicitly inject hand motion information into the denoising latent feature maps, we apply hand mask Mr to provide emphasis guidance on the hand regions, which can be expressed as:
latents Atthanallatents Gi, Gi.) Mr + latents,
(WQ latents) (Wx G (Wy- -G)).
(4)
Atthond (Iatensss Gh. G₁)) Softmax
Vd
where latents denotes the denoising latent feature maps, Wo. WK. Wy represent learnable weights for the cross-attention modules.
3.4 Face Recapture Module
Similar the challenge with hand generation, pro- ducing satisfactory face in human video synthe- sis difficult. Most approaches rely on post- processing strategies to ad- dress this issue, but these methods significantly in- crease training and infer- ence costs. In this paper, we design Face Recap- ture module to extract com- prehensive face information and inject into the face at- tention layer in the Denois- ing U-Net to improve the quality of the generated faces.
Pre-trained Arc face
Att
VAE Encoder
Cropped Reference
M.P
Selt-Att
V
Pose
Self-Att
Att
Driving Poses
Face Cropping
Face Recapture Module
Figure 2: Architecture of the proposed Face Recapture Module.
Specifically, shown in Fig. 2 we enhance the face area of the generated image from two aspects: texture details and global identity. First, we use face detection model to crop and align the face from the reference image. Next, the pre-trained VAE Encoder and face recognition model ArcFace are leveraged to extract the facial texture feature Frez and global identity feature Faa, respectively. For the facial texture feature, perform feature enhancement by combining MLP and a self-attention
5

---

layer. We then repeat these features F times along the temporal dimension to match the size of driving poses and produce Fi E RBxxxIxda and Fie E 1=xxFx(w))
In the meantime, crop out the corresponding facial pose information Fa from the driving pose feature maps according to the face mask My. Similarly, a MLP and self-attention are proposed for feature enhancement on F2 E information, we adopt two separate cross-attention blocks embed texture and identity information into the Fa, respectively. Specifically, by taking F as query, Feer as key and value, we define the texture fusing process Gie Att (F d. F Freas F Fe Similarly, by taking F2 as query, Faa as key and value, we can obtain the fused identity feature Gid through G = Att,a Fa, Fid, Faa). The comprehensive face information is obtained by
In order to equip Fa with textural and identity
G - MLP + Gid).
(5)
Finally, Gy (x is injected into the face attention layer in the Denoising U-Net. Similarly, the face mask is utilized to constrain the latent feature maps after face attention:
latents = face (latents, Gy.Gy)= My + latents.
(6)
3.5 Training strategies
During the diffusion process, the random noise is continuously added to the real image until it reaches a state of Gaussian noise. Conversely, the generation process within the diffusion model is the inverse of the diffusion process, which takes random Gaussian noise as input and generates real images through gradual denoising. In the training process, the diffusion model leverages the Denoising U-Net to predict the added noise at various time steps and the optimization objective can be defined as:
L L lE,,,, (Ile eg (ze,e, e, 0)11)
where is the conditional features, ég represents the Denoising U-Net, z, is the denoising latent feature maps at timestep
In our work, we adopt two-stage training strategy to separately perform appearance modeling and temporal consistency modeling. The first stage image-level training. During this stage, we set F = 1, the training goal is to accurately map the appearance details from the reference image to the driving pose. The VAE encoder and CLIP Image encoder are fixed, and the rest of the network except for the temporal attention are all trainable. In terms of the second stage, the objective is to improve the temporal coherence the generated frames. During this stage, we only perform training on the temporal attention layer with all other network weights fixed and the weights for the temporal attention layers the Denoising U-Net are initialized by the pre-trained AnimateDiff g The loss function can be reformulated as:
L = E [|c- co (Z,, Zr. Gn, Ma. Gy, My, t)1))
(8)
where z, represents the appearance feature maps extracted by Reference U-Net, G₁, represents the hand movement feature maps from the Key Point-based Fine-grained Hand Modeling module, and Gy is the face feature maps obtained by the Face Recapture Module. Additionally, M,, and My denote the masks for the face and hand regions, respectively.
worth noting that in order focus on the face and hand area generation, in the later phase of the first training stage, we adopt the hand mask and face mask to calculate the Ln loss of the corresponding area as the final loss every 10 iterations.
Experiments
4.1 Experimental Settings
Datasets. order to verify the effectiveness of the proposed method, we select the videos two actors, Seth and Oliver, in the talkshow [51] dataset for training and testing. In addition, to enrich the
diversity of characters and hand movements, we record videos of seven people in the indoor scene. The video length of each person is about 10 minutes and we divide the videos into multiple clips of

---

Self Driving
Cross Oriving
Reference Image
Reference Image
Ours
Ours
Animate Anyane
Animate Anyone
Magic Animate
Magic Animate
Magic
Magic
TPS
Figure 3: Qualitative results compared with other methods. Our approach achieves high-fidelity gesture details and satisfactory image quality in both self and cross-driving settings.
about 8 seconds for training convenience. In addition to simple rhythmic gestures, these recorded videos also include many complex gestures such as numbers. We randomly divide the training set and test set according to 9:1.
Implementation Details. For data processing, crop the body region with resolution of 512 512 and utilize the pre-trained face detection model [2) to crop and align faces following FFHQ (20] The resolution of the face image is 256 x 256. The hand and face masks are determined based on the largest circumscribed rectangle of the corresponding key points. All experiments were completed on 8 A800s, with learning rate of le-5. For the first training stage, the batch size B is set to 24, and sequence length F is set to 1. The training step is 100k, which takes about six days. For the second stage, B and F are set to 1 and 24, respectively with 30k training step, taking about one day. During inference, we adopt a CFG 10 of 7.5 and perform 30 denoising steps using the DDIM sampler. Additionally, to ensure temporal consistency between different clips of the same video, use the same reference image, with an overlap of 4 frames between adjacent driving clips.
Comparison Methods. We select several state-of-the of-th--r--- art approaches, including TPS [54] Disco 44 Anyoee [13], MagicAnimate 47. MagicPose (.. Make- Your-Anchor [77 as our compari- son methods. TPS is general animation model based on warping operation, which adopts GAN as the backbone, the remains are diffusion-basee provides the specific person generation model. We finetune these models on the same training dataset with the official codes and pre-trained models.
human video generation models. Among them,
Make-
Metrics. We comprehensively measure the quality of generated images from pixel space and feature space using SSIM, PSNR, and FID [9] and adopt FVD 38 to verify the temporal consistency of generated videos. In addition, the motion accuracies of body (Loody), face (L (Lface), and hand (Lhand) are measured by calculating the mean Euclidean distance between the key points of the generated images and real images.
7

---

Table 1: Quantitative results of our approach compared with SOTAs. Our method achieves the best performance on image quality, temporal consistency, and motion precision.
Method
SSIM t
PSNR
FVD 4
Ltody 4
Lyace +
Liaan
TPS
0.65
29.02
94.77
1120.37
5.99
1.26
17.99
Disco
0.69
29.13
80.76
540.76
5.85
1.52
4.33
Anyoee
0.80
29.41
16.87
365.83
2.73
0.62
1.10
MagicAnimate
0.70
28.55
50.24
665.21
4.48
1.33
3.02
MagicPose
0.82
30.03
16.37
370.75
2.32
0.68
1.12
Ours
0.85
32.23
15.43
197.43
2.27
0.19
0.77
Make Your-Anchor (Seth)
0.63
29.18
32.32
428.84
4.55
1.07
1.64
Ours (Seth)
0.85
33.14
9.83
193.25
2.10
0.21
0.72
4.2
Comparison with Other Methods
Quantitative Results. The quantitative results of our methods compared with SOTAs are shown in Tab. Our method achieves the best results on SSIM. PSNR, and FID metries, underscoring its clear advantages on image quality. In addition, our method obtains the best FVD indicating that the temporal consistency of generated videos is superior. Moreover, our approach outperforms others on the motion accuracy metric including (Lbody), face face,,), and hand (Ihand) which demonstrates the precision of generated poses and gestures. Benefiting from the proposed Key boints-based Fine-grained Hand Modeling module and Face Recapture module, our framework is able generate accurate hand gestures and sigh-fidelity facial details. TPS is general object animation approach and struggles with complex human poses and hand postures, resulting in the poorest overall generation quality. Disco, AnimateAnyone, MagicAnimate and MagicPose focus on generating coarse- coarse-grained human poses without fine-graines grained modeling of hands and faces, leading to lower generation quality, particularly in hand gesture accuracy. Note that Make- Your-Anchor [17] only provides a pre- pre-trained model on Seth data and thus also report the experimental results on the Seth for fair comparison. is observed that our method outperforms
17 on all metrics.
Qualitative Comparison. For qualitative comparison, we conduct two experimental settings, includ- ing self-identity and cross-identity driven. The self-identity driven comparison results are shown on the left of Fig. 3 and 4 where obvious artifacts are marked with red dotted boxes. Our method is able to generate accurate gestures and high-quality hand details while other approaches fail to convey complex gestures. In addition, face identity is well-preserved in our results, while other methods lead to unnatural face shapes, textures, and identity. Note that cross- cross-idenityy driven is more challenging and our method still achieves high-fidelity gesture details and satisfactory image quality indicated in the right of Fig. 3 and The hand areas of other results are blurry and unrcalistic. In summary, method makes of hand and face information through the well-designed Key Points-based Fine-grained Hand Modeling module and Face Recapture module, enhancing the video quality which could meet complicated requirements in conversational scenarios.
Human Evaluation. We conduct a user preference study to evaluate performance of the proposed framework. There are 21 samples and 15 human voters in total. For each sample, we present six video results generated with ShowMaker and other SOTA methods to the human voter a random order.
The human voters are required to estimate the video results in three aspects: a) Motion Accuracy:
Does the video accurately reproduce the motion in the driving vidco? b) Appearance Consistency: Does the subject in the video have consistent appearance with the reference image? e) Temporal Consistency: How is the temporal coherence of this video? The rating score ranges from to 5 and higher scores indicate better preference. As shown in Tab. B our method achieves the highest scores compared with its counterparts, which demonstrates that our method is preferred by a significant margin on motion accuracy and appearance consistency.
4.3 Ablation Study
To verify the contributions of different components, we train three variants by removing the Key Points-based Fine-grained Hand Modeling module (HM), the Face Recapture module (FR), and two-stage training (Stage 2), respectively.
8

---

Reforence
Self Driving
Cross Driviig
w/o Hand Medsling
wifo Face Recapture
Make
Our
Figure Qualitative ablation results when removing different components in our framework.
Qualitative comparison between Make- Your- Anchor and our ShowMaker.
Figure 4:
Table 2: Ablation analysis of HM, FR, and two-stage training.
Table 3: User Study.
Variations
FVD
Lymi
Liund
w/o HM
212.25
1.12
Metuols
Aniiaate
Make-Vnur- Anchor
Discu
wlo FR
208, 28
0.23
Motion Accuracy
4.53
3.13
2.87
3,69
4.43
2.62
1.54
w/o Stage 2
373.87
Asperranee Cousisency
4,23
3.47
2000
3.62
4.18
2.16
1.91
Ours
197.43
0.19
0.77
Tempural Consieicaay
4.38
3.00
230
3.11
3.5
1.78
1.20
For quantitative results, we report the FVD and motion accuracy metrics in Tab. It is observed that the HM and FR could enhance gesture accuracy and face quality by a significant margin. The two- stage training brings about notable improvements in FVD which indicates better temporal consistency, Additionally, we present qualitative comparisons to verify the effectiveness of the proposed HM and FR module. As shown Fig. 5 the generated hands are more satisfactory, and the face shapes and textures are realistic and clear with the proposed modules.
5 Conclusion and Discussion
Conclusion. In this paper, we propose the framework ShowMaker, which achieves high-fidelity 2D human video synthesis with two novel designs to achieve fine-grained diffusion modeling. We first propose Key Point-based Fine-grained Hand Modeling module for robust and fine-grained hand synthesis which takes advantage of 2D hand key points and key point-based codebook. To further reconstruct the facial details and identity information, we introduce Face Recapture module, which effectively equips the structure information with detailed textural information and global identity. Quantitative and qualitative evaluation has indicated the superiority of our framework beyond the existing approaches.
Limitations. Despite the success of our framework, we also recognize some limitations during the exploration. Firstly, our Key Point-based Fine-grained Hand Modeling module can robustly manage hand synthesis with occasional incorrect hand gestures. However, DWPose (50) suffers from performance degradation when handling videos with severe motion blur leading to considerable perturbation in the driving signal, which inevitably results unexpected artifacts in our results. Secondly, our framework produces less satisfactory results when handling challenging cases, such as reflection on glasses. These will be part of our future work.
Broader Impact. Our method focuses on synthesizing realistic 2D avatars with rich facial expressions and complex hand gestures, which is intended for developing digital humans under more daily scenarios like TV shows. However, it may also be misused for some malicious purposes on social media, which leads to negative impacts on the whole society. Therefore, we will make our efforts to strictly oversee the dissemination of our models as well as the resulting content and also restrict access solely to rerearch-oriented demands. We believe that the proper use of this technique will enhance positive societal development in both machine learning research and daily life.

---

ACKNOWLEDGMENTS
This work is supported by the National Nature Science Foundation of China (62121002, U23B2028. 62232006, 62032006, 62472395).
We acknowledge the support of the GPU cluster built by the MCC Lab of Information Science and Technology Institution, USTC. We also thank the USTC Supercomputing Center for providing computational resources for this project.
References
[1] Di Chang, Yichun Shi, Quankai Gao, Hongyi Xu, Jessica Fu, Guoxian Song, Qing Yan, Yizhe Zhu, Xiao Yang. and Mohammad Soleymani. Magicpose: Realistic human poses and facial expressions retargeting with identity-aware diffusion. In ICML, 2024.
[2] Jiankang Deng. Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. Retinaface: Single-shot multi-level face localisation in the wild. In CVPR, 2020.
[3] Yue Gao, Yuan Zhou, Jinglu Wang. Xiao Li, Xiang Ming, and Yan Lu. figh-fidelity and freely controllable talking head video generation. In CVPR, pages 5609-560 5619. IEEE, 2023.
[4] Yuan Gong, Yong Zhang, Xiaodong Cun, Fei Yin, Yanbo Fan, Xuan Wang, Baoyuan Wu, and Yujiu Yang. Toontalker: Cross-domain face reenactment. In ICCV, pages 7656-7666. IEEE, 2023.
[5] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS), pages 2672-2680, 2014.
[6] Jiazhi Guan, Zhanwang Zhang, Hang Zhou, Tianshu Hu, Kaisiyuan Wang, Dongliang He, Haocheng Feng, Jingtuo Liu, Errui Ding, Ziwei Liu, and Jingdong Wang. Stylesync: High- fidelity generalized and personalized lip sync in style-based generator. In CVPR, pages 1505- 1515. IEEE, 2023.
[7] Jiazhi Guan, Zhiliang Xu, Hang Zhou, Kaisiyuan Wang. Shengyi He, Zhanwang Zhang, Borong Liang, Haocheng Feng, Errui Ding, Jingtuo Liu, et al. Resyncer: Rewiring style-based generator for unified audio-visualle synced facial performer. In European Conference Computer Vision, pages 348-367. Springer, 2025.
[8] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. In ICLR. OpenReview.net, 2024.
[9] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In NIPS, pages 6626 0886-6637440 2017.
[10] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. CoRR, abs/2207.12592 2022.
[11] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NIPS), 2020.
[12] Fa-Ting Hong, Longhao Zhang, Li Shen, and Dan Xu. Depth-awar generative adversarial network for talking head video generation. In CVPR, pages 3387-3396. IEEE, 2022.
[13] Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. In CVPR. IEEE, 2024.
[14] Ricong Huang, Weizhi Zhong, and Guanbin Li. Audio-driven talking head generation with transformer and 3d morphable model. In ACM Multimedia, pages 7035-7039. ACM, 2022.
10

---

[15] Ricong Huang, Peiwen Lai, Yipeng Qin, and Guanbin Li. Parametric implicit face representation for audio-driven facial reenactment. In CVPR, pages 12759-12768, IEEE, 2023.
[16] Zhichao Huang, Xintong Han, Jia Xu, and Tong Zhang. Few-shot human motion transfer by personalized geometry and texture modeling. In CVPR, pages 2297-2306. Computer Vision Foundation/ / IEEE, 2021.
[17] Ziyao Huang, Fan Tang, Yong Zhang, Xiaodong Cun, Juan Cao, Jintao Li, and Tong- Yee Lee. :ake-your-anccor:: diffusion-based 2d avatar generation framework. In CVPR, pages 6997-7006. IEEE, 2024.
[18] Subin Jeon, Seonghyeon Nam, Seoung Wug Oh, and Seon Joo Kim. Cross-identity motion transfer for arbitrary objects through pose-attentive video reassembling. In ECCV (24), volume 12369 of Lecture Notes in Computer Science, pages 292-308. Springer, 2020.
[19] Johanna Karras, Aleksander Holynski, Ting-Chun Wang, and Ira Gemelmmacerr-Shiizemme Dreampose: Fashion image-to-video synthesis via stable diffusion. In ICCV, pages 22623- 22633, IEEE, 2023.
[20] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, pages 4401-4410. Computer Vision Foundation / IEEE, 2019.
[21] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014.
[22] Fengyuan Liu, Lingyun Yu, Hongtao Xie, Chuanbin Liu, Zhiguo Ding, Quanwei Yang, and Yongdong Zhang. High fidelity face swapping via semantics disentanglement and structure enhancement. In ACM Multimedia, pages 6907-6917, 17. ACM, 2023.
[23] Wen Liu, Zhixin Piao, Zhi Tu, Wenhan Luo, Lin Ma, and Shenghua Gao. Liquid warping GAN with attention: A unified framework for human image synthesis. IEEE Trans. Pattern Anal. Mach. Intell., 44(9):5114 5132, 2022.
[24] Yifeng Ma, Suzhen Wang, Zhipeng Hu, Changjie Fan, Tangjie Lv, Yu Ding. Zhidong Deng, and Xin Yu. Styletalk: One-shot talking head generation with controllable speaking styles. In AAAI, pages 1994-1904. AAAI Press, 2023.
[25] Arun Mallya, Ting-Chur Wang, and Ming- Yu Liu. Implicit warping for animation with image sets. In NeurlPS, 2022.
[26] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV (I), volume 12346 of Lecture Notes in Computer Science, pages 405-421. Springer, 2020.
[27] Arthur Moreau, Jifei Song, Helisa Dhamo, Richard Shaw, Yiren Zhou, and Eduardo Pérez- Pellitero. Human gaussian splatting: Real-time rendering of animatable avatars. In CVPR, pages 788-798. IEFF, 2024.
[28] Georgios pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
[29] Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. Animatable neural radiance fields for modeling dynamic human bodies. In ICCV, pages 14294-14303. IEEE, 2021.
[30] K. R. Prajwal, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, and C. V. Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In ACM Multimedia, pages 484-492. ACM, 2020.
[31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICMI., volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR, 2021.
11

---

[32] Yurui Ren, Ge Li, Yuanqi Chen, Thomas H. Li, and Shan Liu. Pirenderer: Controllable portrait image generation via semantic neural rendering. In ICCV, pages 11739-17748. IEEE. 2021.
[33] Robin Rombach, Andreas Blatmmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pages 10674-10685 IEEE, 2022.
[34] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. In Neur!PS, pages 7135-7145, 2019.
[35] Aliaksandr Siarohin, Oliver J. Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion representations for articulated animation. In CVPR, pages 13653-13662. Computer Vision Foundation/ / IEEE, 2021.
[36] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[37] Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In CVPR, pages 1526-1535. Computer Vision Foundation/ IEEE Computer Society, 2018.
[38] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric challenges. CoRR, (bss/882.011711 2018.
2018.
[39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[40] Jiayu Wang, Kang Zhao, Yifeng Ma, Shiwei Zhang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Facecomposer: A unified model for versatile facial content creation. In NeurlPS, 2023.
[41] Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. Mead: A large-scale audio-visual dataset for emotional talking-facc fakcnng-faceeaa generation. In European Conference on Computer Vision, pages 700-717, Springer, 2020.
[42] Kaisiyuan Wang, Changcheng Liang, Hang Zhou, Jiaxiang Tang, Qianyi Wu, Dongliang He, Zhibin Hong, Jingtuo Liu, Errui Ding, Ziwei Liu, al. Robust video portrait reenactment via personalized representation quantization. In Proceedings of the AAAI Conference on Artificial 2023.
Intelligence, volume 37, pages
[43] Kaisiyuan Wang, Hang Zhou, Qianyi Wu, Jiaxiang Tang, Zhiliang Xu, Borong Liang, Tianshu Hu, Errui Ding, Jingtuo Liu, Ziwei Liu, et al. Efficient video portrait reenactment via grid-based codebook. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1-9, 2023.
[44] Tan Wang. Linjie Li, Kevin Lin, Yuanhao Zhai, Chung-Chinn Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Disco: Disentangled control for realistic human dance generation. In CVPR. IEEE, 2024.
[45] Dongxu Wei, Xiaowei Xu, Haibin Shen, and Kejie Huang. C2F-FWN: coarse-to-fing flow warping network for spatial-temporal consistent motion transfer. In AAAI, pages 2852-2860. AAAI Press, 2021.
[46] Chao Xu, Junwei Zhu, Jiangning Zhang, Yue Han, Wenqing Chu, Ying Tai, Chengjie Wang, Zhifeng Xie, and Yong Liu. High-fidelity generalized emotional talking face generation with multi-modal emotion space learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 6609-6619, 2023.
[47] Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. In CVPR. IEEE, 2024.
12

---

[48] Kewei Yang, Kang Chen, Daoliang Guo, Song-Hai Zhang, Yuan-Chen Guo, and Weidong Zhang. Face2facel: Real-time high-resolution one-shot face reenactment. In ECCV (13), volume 13673 of Lecture Notes in Computer Science, pages 55-71. Springer, 2022.
[49] Quanwei Yang. Xinchen Liu, Wu Liu, Hongtao Xie, Xiaoyan Gu, Lingyun Yu, and Yongdong Zhang. REMOT: A region-to- whhlle framework for realistic human motion transfer. ACM Multimedia, pages 1 1128-1133 1137. ACM, 2022.
[50] Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. Effective whole-body pose estimation with two-stages distillation. In ICCV ( (Workshops), pages 4212-4222. IEEE, 2023.
[51] Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yandong Wen, Timo Bolkart, Dacheng Tao, and Michael J Black. Generating holistic 3d human motion from speech. In CVPR, 2023.
[52] Zhentao Yu, Zixin Yin, Deyu Zhou, Duomin Wang, Finn Wong, and Baoyuan Wang. Talking head generation with probabilistic audio-to-visual diffusion priors. In ICCV, pages 7611-7621. IEEE, 2023.
[53] Bowen Zhang, Chenyang Qi. Pan Zhang, Bo Zhang, Hsiang Tao Wu, Dong Chen, Qifeng Chen, Yong Wang, and Fang Wen. Metaportrait: personalized adaptation. In CVPR, pages 22096-22108 IEEE, 2023.
talking head generation with fast
[54] Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In CVPR, pages 3647-3656. IEEE, 2022.
[55] Jinkai Zheng, Xinchen Liu, Wu Liu, Lingxiao He, Chenggang Yan, and Tao Mei. Gait recognition in the wild with dense 3d representations and A benchmark. In CVPR, pages 10228-20233 2022.
[56] Shenhao Zhu, Junming Leo Chen, Zuozhuo Dai, Yinghui Xu, Xun Cao, Yao Yao, Hao Zhu, and Siyu Zhu. Champ: Controllable and consistent human image animation with 3d parametric guidance, 2024.
13

---

A
Appendix
A.1
Pose Encoder Structure
Fig.
6
shows the detailed network structure of the pose encoder.
A.2 Dataset Details
Tab. gives the number of training and testing clips for the indoor recording dataset and talkshow dataset. The length of each clip is 2-15 seconds.
When preparing the training data, we first produce the DWPose results from each frame and set the center of shoulders as the cropping center. Then we crop the original video frame using an adaptive cropping size, where the cropping size is designed width. This operation forces the human body to lie in a roughly consistent position. Finally, all cropped images resized to 512 x 512.
fixed ratio (we set it 2.65) of the shoulder
In the inference stage, for the cross-identity driven setting, we further bridge the gap between body shapes by scaling the driving poses to match the reference pose. The scaling ratios of width and height are defined as W, and H Ha, where W and H represent the shoulder width and the height of the human body, respectively. The shoulder width is measured as the Euclidean distance between the left and right shoulder key points, while the body height is determined by the Fuclidean distance from the nose key point to the pelvis key point.
A.3 More Ablation Experiments
To further verify the effectiveness of the proposed Key point- based Fine- meneegrained Hand Modeling (HM), we show more ablation experiments in Fig. where Vanilla refers to removing the entire HM module from our proposed ShowMaker, Hand image refers to adopting the cropped hand image to extract texture features through the VAE encoder and then feeding it into the hand attention, and w/o positional encoding refers to not using positional encoding in the HM module. can be seen that using the hand image as compensation information has no obvious effect on improving the hand texture and structure, As for HM, positional encoding can significantly improve the high-frequency details and structural accuracy of the generated hands.
A.4 Temporal consistency
Fig. [ shows two generated video sequences, each sequence contains 3 adjacent video frames. It can be seen that the generated video frames at different times have consistent appearance and no temporal texture changes occur.
A.5
Challenging Examples
Fig. 9 shows two challenging sample generations. In the example on the left, the driving pose is extracted from female video, while the reference image male, and the driving gestures and the reference gesture are very different. In the example on the right, the reference image is not a common frontal image instead of side image. Our proposed ShowMaker can still generate target frames well in these two cases, which shows that our method has satisfied robustness.
14

---

3x512x512
16x512x512
32x256x256
96x128x128
256x64x64
320x64x64
Conv 3x3 16 1
Conv 3x3 16 1
Conv 3x3 32 2
Conv 3x3 96 2
Conv 3x3 96 1
Conv 3x3 256 2
Conv 3x3 320 1
Padding 1
Padding 1
Conv 3x3 32
Padding 1
Padding 1
Padding 1
Pa
Padding
Padding
Padding
Fpa
Driving
Pose
Figure 6: The pose encoder structure consists of convolutional layers, where (Conv x 1) represents the kernel size is 3 x x 16, and the stride is 1. Except for the last convolutional layer, each convolution is followed by GroupNormaiiztion and the activation function silu.
Vanilla
Hand image
w/o positional encoding
Ours
Figure 7: Ablation experiments of hand module.
Table Dataset details. ID1-7 are datasets recorded indoors. Seth and Oliver belong the talkshow dataset. All training and test sets do not overlap.
The number of clips the training
The number of clips
IDs
in test set
ID1
49
5
ID2
57
6
ID3
56
ID4
65
7
IDS
47
5
ID66
64
ID7
55
Seth
3306
100
Oliver
6746
200
15

---

Frame 1
Frame
Frame 9
Figure 8: The generation results of adjacent frames.
Reference Image
Cross- Cross-Driving
Reference Image
Cross Pross-Driving
Ours
Ours
Figure 9: The generated results when the pose difference obvious.
16

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Our main claims in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
17

---

Justification:
We discuss the limitations of the work in Section Conclusion and Limitation
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We fully disclose all the information needed to reproduce the main experimen- tal results.
Guidelines:
18

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We will provide the code in the camera-ready version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
www.:/ubbissioosossoooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https c/iips.cc/public//uue
Codeeuumiiii wettsbbmmssiiioi ssionpolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
19

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We report the training and test details in the paper, including environment and dataset.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: This paper does not include error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide sufficient information on the computer resources.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
20

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps:/ /tttt///it///////rrrra//oa/aaaaa
Answer: [Yes]
Justification:
The research conform, every respect, with the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impacts in the paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [Yes]
Justification: We describe safeguards for responsible release of data or models by requiring that users adhere usage guidelines.
Guidelines:
21

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We explicitly mention and properly respect the license and terms of use paper and the creators original owners of assets used in the paper are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of license (e. CC-BY 4.0) should included for each asset. For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssiippssiiiiaea... phcerssithcode thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details the dataset/code/moddll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [Yes]
22

---

Justification:
We report the full text of instructions.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: Our user study does not apply to it.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
23

---

References for the Appendix
24


Reviews:
[{"review_id": "0UPXuNs5OW", "reviewer": "Reviewer_jdWr", "summary": "This paper represents an early attempt to advance the field of 2D digital human synthesis in real-world scenarios, extending the traditional talking head task to include more complex body movements, such as hand gestures. The authors present a novel framework utilizing dual-stream diffusion models for full-body video synthesis, complemented by two specialized modules designed for fine-grained detail modeling. Specifically, the proposed Key Point-based Fine-grained Hand Modeling effectively addresses the challenge of hand gesture generation through the integration of a key point-based codebook and high-dimensional positional encoding. Additionally, the Face Recapture module enhances facial textures and identity consistency, enabling the generation of more realistic and vivid 2D avatar videos.", "questions": "1) I understand the authors adopted the positional encoding to enhance the information provided by the sparse hand key points, but I’d like to see a further analysis of the positional encoding used in the Hand Modeling module. Since it cooperates with the key point-based codebook, I am wondering if a well-trained codebook is already enough for hand restoration.\n\n2) Have the authors adopted any alignment strategy in cross-driving experiments? In my understanding, cross-driving animation usually suffer from mis-alignment due to the subject differences.", "limitations": "The authors adequately addressed the limitations and negative societal impact.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1) This paper outlines an exciting approach for digital human communities, focusing on the synthesis of 2D controllable human body videos with detailed modeling. Different from existing works like \"Animate Anyone\" that primarily focus on generalized human body movement generation across various subjects, this paper presents a promising direction for digital human communities by focusing on fine-grained modeling and controllable video synthesis of human body videos, especially hand gestures, which has the potential to achieve better human-computer interaction with our body gesture and benefit our demands in real-world scenarios.\n\n2) The paper is well-written and easy to follow. The motivation for the whole task and the two additional modules is clear. The supplementary video provides robust results and convincible comparison with sota methods, which further demonstrate the effectiveness of all the designs proposed in this paper.", "weaknesses": "There are few mistakes and typos in the paper:\n1) The Face Recaption module takes “Reference Pose” as input in Fig 1 (d), which is not mentioned in Fig 2 and Sec 3.3. The description of Face Recaption module should be the same throughout the full manuscript.\n\n2) Eq (5) and Eq(6) are nearly the same. The authors can merge them into one equation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer jdWr\nThanks very much for your careful review comments. We'll answer your questions one by one.\n- Q1: About Fig 1 (d).\n\n  A1. Thanks for pointing out this mistake. We will remove the “Reference Pose” from Figure 1 (d) and refine the whole figure in the revised manuscript.\n  \n- Q2: About Eq (5) and Eq (6).\n\n  A2. Thanks for your suggestion. We will replace Eq (5) and Eq (6) with a unified definition of the attention operation and describe the different inputs used in Eq (5) and Eq (6).\n\n- Q3. The effect of position encoding on generation results.\n\n  A3. Please refer to Figure 1 (the third row) in the pdf file, the comparison results demonstrate that position encoding brings a significant performance improvement in both textural and structural details of human hands. The key points of the hands are distributed in a low-dimensional space and are relatively clustered, directly using the coordinates of the hand key points as inputs of the Hand Modeling module cannot well capture the structure and texture information from the hand. \n\n- Q4. Alignment strategy in cross-driving experiments.\n\n  A4. For the cross-driving setting, we adopt a pose alignment strategy that roughly aligns the driving signals from other subjects with the ones of the target person. The key points of our pose alignment strategy are as follows:\n\n    &emsp; (1) When preparing the training data, we first produce the DWPose results from each frame and set the center of shoulders as the cropping center. Then we crop the original video frame by using an adaptive cropping size, where the cropping size is designed as a fixed ratio of the shoulder width. This operation forces the human body to lie in a roughly consistent position. \n\n    &emsp; (2) In the inference stage, we further try to bridge the gap between body shapes by scaling the driving poses to match the reference pose. The scaling ratios of width and height are defined as $w_r/w_d$, $h_r/h_d$. Here, $w$ represents the shoulder width, and $h$ represents the height of the human body. We will involve a detailed description of $w$ and $h$ in the appendix.\n\n   &emsp; The above strategy enables satisfactory pose alignment between different subjects under our scenario.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YNvZx0odMJ", "reviewer": "Reviewer_HRx2", "summary": "This paper proposes a 2D human video generation framework called ShowMaker, which can generate half-body conversational videos based on 2D keypoints as motion conditions. ShowMaker includes a texture enhancement module for the face and hands, and demonstrates some effectiveness.", "questions": "1. In Table 1, it is unclear what \"One ID\" represents. It is also unclear whether the training set only contains data for the testing IDs. Additionally, it is unclear whether the compared methods, such as Animate Anyone and TPS, were also trained on the testing IDs. More details are needed to clarify these points.\n2.  Are the IDs included in the training set and testing set are completely identical?", "limitations": "Please refer to the questions and weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. ShowMaker achieves good results in generating texture details, with improvements over the compared methods.\n2. The motivation behind ShowMaker is straightforward, and the method design is simple.", "weaknesses": "1. The experimental settings lack clarity, especially regarding the balance of training duration for each ID in the testing examples. It is unclear how much training data corresponds to each ID in the visualized examples, and more details are needed.\n2. The method proposed by the authors has some improvements in details compared to the similar method Make-an-Anchor. However, from Figure 4, it seems that the generation of facial details is unstable, resulting in some color changes (such as in the beard area), which raises concerns about the facial generation capability of the proposed method.\n3. The ablation studies only excluded the face modeling and hand modeling modules. However, introducing region images to enhance the generation results is a straightforward approach in human body generation tasks and could serve as a baseline for comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer HRx2\nThanks very much for taking time out of your busy schedule to review our manuscript.  We reply as follows.\n\n- Q1. The experimental settings. (Response to weakness 1 and question 2)\n\n  A1. Our goal is to extend the talking head task to generate an expressive half-body conversational speaker. Therefore our experimental setting is not a one-shot animation. Following the settings of Make-Your-Anchor, the IDs of the training set and the test set are consistent, and the test set and training set do not overlap.\n\n  &emsp; For detailed dataset information, please refer to Table 1 in the pdf file. Due to space limitations, we will place the additional experimental details in the appendix.\n\n- Q2. About \"One ID\" and Animate Anyone and TPS.\n\n  A2.  Since Make-Your-Anchor provides only one pre-trained model of Seth without the training code before our submission. We failed to reproduce its experiments on other subjects. Therefore, we compared our approach with it only on the Seth data (\"One-ID\") for fairness. \n\n  &emsp; Please also note that the training and test sets of Animate Anyone and TPS are consistent with ours.\n\n- Q3. About some color changes in the results.\n\n  A3. We suppose that the color changes mentioned by the reviewers may come from lighting changes in the facial region. Please refer to the videos in our supplementary videos and additional results in Figure 3 in the pdf file,  our results show temporally consistent texture in the facial region due to the Face Recapture module.\n\n- Q4. About introducing region images as a baseline for comparison. \n\n  A4. Thanks for your suggestion. Actually, we follow the idea of ​​using the regional image to enhance the results. In our Face Recapture Module, we use cropped and aligned face images to enhance the facial details in the generated results. However, when dealing with hand regions, there are two major problems:\n\n  &emsp; (1) Since they only occupy a small proportion of the image, it is difficult to capture enough textured details within the cropped hand images.\n\n  &emsp; (2) Hand poses, expressed in more degrees of freedom, are more complex when compared with head poses. The hand poses from the reference image and driving signals may be very different, which may alleviate the contribution of texture from the cropped hand image.\n  \n  &emsp; Therefore, we map the driving hand key points to a high-dimensional space and adopt a discrete codebook for hand enhancement (refer to A3 for Reviewer jdWr). \n\n  &emsp; We still follow your suggestion to conduct an experiment and the results are illustrated in Figure 1 (the second row). Here, the cropped reference hand image is sent to the VAE encoder for feature extraction and then fed into the hand attention module. The results show that the hand modeling strategy we proposed enables the best synthesis of hand details and structures among all these variants.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S5QB9FxlQH", "reviewer": "Reviewer_R54L", "summary": "This paper proposes a novel conversational human video generation framework named showmaker based on the fine-grained diffusion model. This paper proposes a novel Key Point-based Fine-grained Hand Modeling module and construct a key point-based codebook to handle the challenging hand generation. Meanwhile, this paper extracts facial texture features and global identity features from the aligned face and integrates them into the diffusion process for face enhancement. Sufficient experiments demonstrate the superiority of the proposed framework, and the generated human videos are of high quality.\n\nI have read the comments of other reviewers and the response of the authors, I tend to give a accept score.", "questions": "1. The input for the Face Recapture Module in Figure 1 incorporates a reference pose, whereas the input in Figure 2 lacks such a reference pose.\n2. Shouldn't line 140 on page 3 be replicated F times along the temporal dimension?\n3. What is the rationale behind selecting a VAE encoder as the texture extractor in the Face Recapture Module?\n4. Could you elaborate on the shapes of Gh and Gf? Additionally, why was a discrete codebook chosen to represent hand information? What advantages does this design offer?", "limitations": "Authors adequately discuss the limitations and potential negative societal impact", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper is well written and clearly explains the motivations for the design as well as important technical details.\n2.The human video generation framework introduced in this paper holds significant practical value, particularly in conversational scenarios like TV shows.\n3.﻿The proposed Face Recapture module contributes to generate accurate face regions.\n4.The paper has conducted sufficient experiments that demonstrate the superiority of the proposed model, and the generated videos are highly satisfactory.", "weaknesses": "1.Although the author compares the proposed method with talking head synthesis in the demo, it would be more beneficial to include this comparison in the main text or as an appendix, as it is crucial for understanding the contribution.\n2.The presence of several writing errors in the paper impedes the correct comprehension of its content.\n3.The details regarding some network structures and inference processes, such as the pose encoder, are insufficiently described and require further elaboration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer R54L\nThanks very much for your valuable suggestions and for pointing out the typos in the manuscript. Here are my responses.\n\n- Q1. About the comparison with talking head synthesis.\n  \n  A1. Thanks for your suggestion, we will add the talking head comparison in the demo to the main manuscript to reflect that our task is an extension of the talking head.  \n\n- Q2. About the structure of the pose encoder.\n  \n  A2. For a detailed description of the Pose Encoder, please refer to Figure 2 in the pdf.\n\n- Q3. About some typoes.(Response to Weakness 2 and Questions 1, 2)\n\n  A3. Sorry for the confusion. The \"Reference Pose\" in Figure 1 (d) should be removed. It should be repeated F times instead of L times in line 140 on page 3. We have corrected the known writing errors and will carefully revise the manuscript.\n\n- Q4. The reason for choosing the VAE encoder as the texture extractor in the Face Recapture Module?\n\n  A4. The VAE pre-trained on large-scale datasets has excellent compression capabilities and can capture local texture information of the face region well. Combined with global facial identity features, it can comprehensively represent facial information. Additional results are shown in Figure 3 in the pdf file, it can be seen that our method can generate satisfactory facial details due to the Face Recapture module.\n\n- Q5. About the shapes of $G_h$ and $G_f$. The advantages of adopting a discrete codebook chosen to represent hand information.\n\n  A5. Sorry for our oversight. The shape of $G_h$ is $B\\times F\\times 2\\times 512$ and the shape of $G_f$ is $B\\times F\\times h\\times w\\times 768$. Representing hand information with a discrete codebook is robust. Moreover, the designed codebook consists of a set of orthogonal basis directions, which can fully represent hand structure and texture. (Please refer to A4 for Reviewer HRx2 and A3 for Reviewer jdWr for more motivation.)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6KhXxKWBmS", "reviewer": "Reviewer_xzB4", "summary": "This paper proposed a pipeline for 2D human motion retargeting, based on 2D key points and face encoding via fine-grained diffusion modeling.", "questions": "See weakness. Major questions as following:\n\n1. Examples of Long Video Generation. Ideally, the generated video of the TikTok test set as MagicAnimate shows. (~at least 20 seconds or longer)\n\n2. Motion retargeting visualization while condition and reference have quite different human poses. Please see the previous paper, MagicPose Figure 5,7,17, for reference.\n\n3. Previous work, e.g., MagicPose, has shown that the model can synthesize the back of the human subject. Can this model do the same? Given that the front of the human is the reference image and another pose condition, how can the model learn to distinguish the front/back of the human from such conditions?\n\n4. Can this model provide a satisfying result if the identity reference is a non-frontal image,e.g., the left or right side of the human?\n\n5. Is there any visualization on out-of-domain motion retargeting? E.f. cartoon-style animation?\n\n6. More detailed explanation of the novelty.", "limitations": "The authors include a limitation section.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written, and the demo presents satisfying hand and face modeling results.", "weaknesses": "1. Missing recent baselines in the experimental comparison. E.g. Diffusion-based method: MagicPose from ICML 2024, GAN-based methods: TPS from CVPR2022 etc.\n\n2. The novelty of the proposed method is limited. It seems to me that the only novel component of this paper compared to previous works is the face encoding and key point-based hand modeling. The ReferenceNet for identity control and Pose Encoder for pose control are all from previous works (ReferenceNet from Animate Anyone, MagicAnimate, MagicPose, Champ, and ControlNet from DisCo, MagicAnimate, Animate Anyone, MagicPose). Temporal Attention the same.\n\n3. I would like to see the performance of the proposed method compared to baselines on the TikTok dataset since this is a public dataset, and all previous works reported metrics on it. I'm not implying the authors are hiding something but I think it's necessary to follow previous settings and report the performance on the new dataset (talkshow).\n\n4. Lack of Long Video Generation Results. The presented videos in the supplementary materials are very short compared to MagicAnimate's videos on TikTok dataset. It's necessary to provide longer visualization.\n\n5. Lack of motion retargeting visualization while the condition of human DW-Pose is significantly different than the Human Pose in the reference image. The visualization in both supp and main paper shows only the cases where condition and reference image share similar human poses. This also raises the concern about face encoding and key point-based hand modeling: will such network design preserves human identity and ensure temporal coherence when the poses have a huge gap between reference and condition?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer xzB4 \nThanks for such a detailed review. Here are my responses. \n\nAccording to the provided reviews, we believe there are a few misunderstandings on the task setting that need to be clarified. \n\n__Task \\& Setting Issues__\n\n&emsp; (1) Unlike the approaches concentrating on holistic body movement synthesis (e.g., AnimateAnyone and MagicPose), our ShowMaker focuses on __half-body 2D talking avatar__ creation,  which aims to upgrade the traditional 2D talking head avatar task to the next level with richer movements (e.g., hand gestures and upper body movements). In this scenario, the training data usually contains frontal images with very few side-view cases, therefore learning to distinguish the front/back of the human is not necessary (Question 3).\n\n&emsp; (2) Our approach follows the __person-specific setting__ as Make-Your-Anchor rather than the one-shot animation setting used in AnimateAnyone. We provide our train/test data information in Table 1 in the pdf file. Under the person-specific setting, we focus on self-driving and cross-driving performance on trained IDs instead of out-of-domain data (e.g., cartoon-style images (Question 5)). Considering this person-specific setting and our task scenario, we conduct experiments on the TALKSHOW dataset and our collected dataset instead of the TikTok dataset (Weakness 3).\n\n&emsp; (3) The dual-branch design with a ReferenceNet is an appropriate backbone for generative tasks, but we did not involve them in our contributions. Our technical contributions are summarized below (Weakness 2 and Question 6):\n\n&emsp; &emsp; (a) We propose an effective hand recovery module for fine-grained synthesis by using a positional encoding strategy and a carefully designed key point-based texture codebook. Our approach shows obvious superiority in the hand synthesis over other dual-branch baselines.\n\n&emsp; &emsp; (b) We propose a Face Recapture module to improve facial details and identity-preserving ability, especially under the cross-driving setting with pose and shape gaps. The results in our supplementary video and the pdf file also show the temporally consistent facial texture and identity.\n\nIn addition to the task and setting issues, there are also some experimental issues.\n\n__Experimental Issues__\n\n- Q1. Comparison with TPS from CVPR2022 and MagicPose from ICML 2024. (Response to  Weakness 1)\n\n  A1. Thanks for your suggestion. We compared the TPS method in our paper, please see Table 1 and Figure 3 in the main paper for more details. In terms of MagicPose, since it is a concurrent work that was accepted by ICML in May 2024 (very close to the submission date), we involve another work MagicAnimate (similar to MagicPose) in comparison.\n\n&emsp; Here, we finetune MagicPose on our training data and provide the comparison results in Figure 4 in the pdf file. Similar to AnimateAnyone and MagicAnimate, MagicPose suffers from inaccurate generation and texture degradation in both facial and hand regions. Please zoom in and see the details in the red boxes. The results demonstrate the necessity of our proposed Face Recapture module and Fine-Grained Hand Modeling.\n\n- Q2. About longer visualization. (Response to Question 1)\n\n  A2. In our supplementary video, we have provided demo videos of 10-15 seconds, which is comparable to the length of videos in the TikTok dataset.  During our exploration, we managed to generate videos longer than 30 seconds with high-fidelity and temporally consistent texture. Since we cannot provide external links, we will add longer-generated videos to our supplementary video. \n\n- Q3. Pose gaps between the reference and driving images & Non-frontal images. (Response to Question 2, Question 4)\n\n  A3. We provide two examples in Figure 5 in the pdf file to illustrate how our ShowMaker performs when dealing with challenging cases with pose gaps or non-frontal images.\n\n  &emsp; (1) For pose gaps, we first provide an example by using a reference image from a male and driving signals from a female. As shown in Figure 5 (left part), there is a huge gap in both pose and shape. Please also refer to A4 for Reviewer jdWr, we adopt our pose alignment strategy to first align the driving signals with that of the reference image for better synthesis and the results show the effectiveness of our approach under huge pose gaps.\n  \n  &emsp; (2) Non-frontal images. Since most of the videos in our dataset are in the front view, so we took some time to look for a side-view reference image in the test set, and the generated result is shown in Figure 5 (right part). The results demonstrate that our network design can preserve human identity and hand texture even when processing non-frontal reference images.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# To all reviewers and ACs
Thanks very much for all the reviewers' efforts and suggestions. We appreciate the positive comments on the following:
1. The generation framework holds significant practical value, particularly in conversational scenarios like TV shows. 
2. The paper is well-written and clearly explains the important technical details.
3. The motivation behind ShowMaker is straightforward and the designed model is clear and convincible.
4. The proposed model achieves good results in generating texture details, with improvements over the compared methods, and the demo presents satisfying hand and face modeling results.

We believe the remaining issues can be fully addressed. We will respond to your comments one by one below.
Some relevant figures and tables are provided in the pdf file.

Meta-review: Reviewers mostly agree that the proposed system is well-designed, and provides good results for the less studied problem of half body generation. The authors' rebuttal addressed most of the reviewers' concerns who in many cases increased their score. Overall there seems to be consensus for acceptance.
Decision: Accept (poster)