Kati8/ICLR_2023_2025 huggingface dataset:
authors: Zerong Zheng
reviews: 
Comments: This paper introduces CyberHost, audio-driven human animation framework based on diffusion models. It addresses the less explored area of full-body human animation driven by audio signals, focusing on enhancing the generation quality of critical regions like the hands and face. The authors propose a Region Codebook Attention mechanism, along with a suite of human-prior-guided training strategies. The paper aims to bridge the gap in audio-driven human animation by improving hand clarity and overall natural motion.
Strengths: The authors aimed to tackle two significant challenges in audio-driven body generation and achieved progress in:
1. Improving the synthesis quality of critical regions (hands and face)
2. Reducing motion uncertainty caused by weak correlations.

Specifically, this paper successfully addresses the challenge of generating high-quality hand and facial features using proposed modules including RAM.

In addition, comprehensive experiments were conducted. Comparisons were made to evaluate not only audio-to-body generation methods but also video-to-video and audio-to-face methods, demonstrating its expandability.
Weaknesses: While most parts are understandable, some details and explanations are missing. The questions regarding the missing information are listed under "Questions." Additionally, as the methods utilize many well-known architectures and frameworks while introducing several modules—including the Latent Bank, Pose Encoder, Heatmap Estimator, and Mask Predictor—some missing information limits the paper’s reproducibility and clarity of the paper. If the concerns or questions listed on "Questions" are addressed, this paper would be worthy of a higher rating.
Recommendation: 8
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper proposes a novel and elegant one-stage audio-driven human diffusion model. The authors primarily focus on the most challenging problems of existing body animation models, which are details underfitting and motion uncertainty. To address details underfitting, the authors introduce a region attention module, and to tackle motion uncertainty, they design a series of human-prior-guided conditions. The paper is well-written and enjoyable to read. The final video results demonstrate high-quality rendering and natural motion driving.
Strengths: 1. This paper addresses two important and challenging problems in the body animation field, and the proposed approaches are novel and effective.
2. The proposed method supports multi-modalities driving
3. The driving results show really good rendering quality and natural motion fidelity.
4. The paper is well-organized and well-written.
Weaknesses: 1. Some details are not provided:  
a) Which specific layers of Wav2Vec features were used? (line 191)  
b) How to constrain the basis vectors of the latents bank to be orthogonal? (line 242)  
c) There is a lack of loss description for regional mask predictor. (line 260)  
2. Authors claim that the hand clarity score can enhance the model's robustness to blurry hands during training and enable control over the clarity of hand images during inference. They conducted ablations on hand clarity, but they did not demonstrate to what extent this score can control hand clarity during inference. I would like to know this result.
3. The explanation of how the proposed 'Pose-aligned Reference Feature' works has not convinced me for two reasons:  
a) Although the ablation on pose-aligned ref shows a lower HKC score compared with Cyberhost, this method was proposed to solve the case of challenging initial poses, and the authors did not demonstrate its effectiveness in that scenario.  
b) The authors claimed that the skeleton map provides topological structure information, which improves the quality of hand generation. However, they did not explain how this structural information actually contributes to generating higher-quality hand images.  
4. Some spelling mistakes: 'feference' should be corrected to 'reference' in line 313.
Recommendation: 10
Reviewer confidence: 5
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper proposes a one-stage audio-driven talking body video generation framework, addressing issues in half-body video generation such as blurred hand details, inconsistent identity, and unnatural motion. Specifically, it introduces a Region Attention Module (RAM) to enhance the quality of local regions. Additionally, it proposes a human-prior-guided condition to improve motion stability in generated videos. A new dataset was collected for experimentation, with results verifying the effectiveness of the proposed method and the improvements contributed by each component.
Strengths: 1.	The proposed method demonstrates a certain degree of generalization, allowing it to adapt to multiple tasks, such as video-driven generation or multimodal-driven generation, while also enabling open-set generation.
2.	Based on the experimental results, the proposed method surpasses both the baseline and state-of-the-art methods across multiple metrics.
Weaknesses: 1.	Although the proposed method achieves promising results overall, it introduces many components. As shown in Table 1, there are nine components, but the experiments lack in-depth analysis of these. For example, the impact of the size of the latent bank in RAM. The results of using alternatives in the Region Attention Module (RAM), such as not using spatial latents, were not examined. Additionally, the effect of not decoupling the latent bank into spatial and temporal latents—instead using a single 3D latent bank—was not investigated. Furthermore, it remains unclear what specific aspects of video information are captured by the spatial and temporal latents, lacking justification and explanation.
3.	The use of the Laplacian operator to compute the hand clarity score requires justification, as the rationale behind this choice is not explicitly discussed. Additionally, the influence of the hand clarity score on the experimental results is not demonstrated in the experiments. It is essential to clarify whether this score is necessary and how it contributes to the overall performance of the proposed method.
4.	The method [1] is also a one-stage audio-driven half-body video generation model, but this paper does not discuss or compare it.

5.	The dataset used in [2] was not employed in experiments for comparison with previous methods. Additionally, the beat consistency metric [3] was not reported in the experiments.

6.	Some typos, such as in line 313 feference -> reference

reference:

[1] Liu X, Wu Q, Zhou H, Du Y, Wu W, Lin D, Liu Z. Audio-driven co-speech gesture video generation.

[2] Qian S, Tu Z, Zhi Y, Liu W, Gao S. Speech drives templates: Co-speech gesture synthesis with learned templates.

[3] Li R, Yang S, Ross DA, Kanazawa A. Ai choreographer: Music conditioned 3d dance generation with aist++
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: The paper introduces an end-to-end audio-driven human animation framework, which is designed to generate realistic and natural upper body human videos from a single image and control signals such as audio, ensuring hand integrity, identity consistency, and natural motion.
Strengths: 1. Cyberhost can generate cospeech videos with very natural motions and clear hand/body structures.
2. It employs various control training methods, including codebook , hand clarity, pose-aligned reference, and also key point supervision. Experimental results indicate that these methods effectively enhance the clarity of hands and the correctness of body structures in the generated objects.
Weaknesses: 1. The generated videos exhibit insufficient facial clarity and detail, resulting in a noticeable discrepancy between the generated object and the characteristic features of the person in the reference image.
2. Unlike the codebook in VQ-VAE, which is specifically used for the reconstruction of designated features, the codebook in Cyberhost lacks supervisory signals during training, making it unable to ensure that the codebook effectively guides the model to generate correct hand shapes and facial features.
3. It would be good to visualize the ablation study for the two main contribution components: “Motion codebook” and "ID Descriptor".
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 2
reviews: 
Comments: The paper introduces CyberHost, an innovative one-stage audio-driven framework for generating talking body animations, addressing common issues such as hand integrity, identity consistency, and natural motion. Unlike multi-stage methods using intermediate representations like poses or meshes, CyberHost works end-to-end and supports zero-shot generation.

Key innovations like Region Attention Module and the usage of Human-Prior-Guided Conditions are proposed to improve the generation quality of local human regions and to address the motion uncertainty problem.

Experiments show CyberHost outperforms existing methods both qualitatively and quantitatively and works well in audio-driven, video-driven, and hybrid scenarios.
Strengths: 1. CyberHost introduces the first one-stage approach for audio-driven talking body generation, avoiding the complexity and inefficiencies of multi-stage systems that rely on intermediate representations.

2. The proposed Region Attention Module component effectively enhances critical areas such as hands and faces, improving the quality of local details and maintaining identity consistency.

3. By integrating motion constraints and structural priors via human-prior-guided conditions, the model mitigates the challenge of motion uncertainty, resulting in more stable and natural body animations.

4. The qualitative results in the supplementary materials are impressive. Also, compared to the previous state-of-the-art audio-driven half-body generation method, VLOGGER, CyberHost produces visibly superior results.

5. The paper is well-written and clearly presents its objectives, methodology, and findings.
Weaknesses: 1. Detailed Failure Analysis: The paper would benefit from a discussion of failure cases or limitations where CyberHost struggles, such as specific types of input audio or complex poses. This would provide a more balanced view of the model's capabilities.

2. Scalability to Full-Body Generation: The paper focuses on half-body animation, but it does not discuss how well the architecture scales to full-body animation or if there are significant challenges in extending the framework.

3. Lack of User Study for Subjective Evaluation: The paper does not include user studies or subjective evaluations to gather feedback on the perceived naturalness and quality of the generated videos. Such evaluations would provide valuable insights into how well the model meets human expectations for lifelike animation.
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper introduces an innovative single-stage diffusion-based model for generating human animations from audio inputs. The work tackles two critical limitations in current body animation approaches: insufficient detail representation and inherent motion variability. The authors propose a region-focused attention mechanism to improve detail fidelity and incorporate human-prior constraints to reduce motion ambiguity. The manuscript is clearly structured and engaging to follow. The resulting animations exhibit impressive visual quality and realistic motion synthesis driven by audio signals.
Strengths: 1. The paper effectively identifies and addresses two significant challenges in the body animation domain with innovative and well-executed solutions.
2. The proposed approach enables driving animations from multiple input modalities.
3. The animation results demonstrate excellent visual fidelity and convincing motion realism.
4. The manuscript is logically organized and clearly presented.
Weaknesses: 1. Several technical details require clarification:
a) Which specific layers from the Wav2Vec feature extraction were selected? (line 191)
b) What methodology ensures orthogonality among the basis vectors in the latents bank? (line 242)
c) The loss function formulation for the regional mask prediction component is missing. (line 260)
2. While the authors propose that hand clarity scoring improves model robustness to hand blur during training and offers inference-time control, the ablation study does not quantify the degree of hand clarity control achievable during inference. Additional empirical evidence on this controllability would strengthen the claims.
3. The mechanism underlying 'Pose-aligned Reference Feature' requires further justification:
a) Despite showing reduced HKC scores relative to Cyberhost in ablations, the method was designed to handle difficult initial poses, yet its effectiveness in such challenging scenarios is not empirically demonstrated.
b) The authors argue that skeleton maps convey topological information that enhances hand generation quality, but the specific mechanisms through which this structural information improves output quality remain unexplained.
4. Typographical error present: 'feference' requires correction to 'reference' at line 313.
Recommendation: 10
Reviewer confidence: 5
Soundness/correctness: 3
Clarity: 3