Kati8/ICLR_2023_2025 huggingface dataset:
authors: Haiyang Liu
authors: Xingchao Yang
authors: Tomoya Akiyama
authors: Yuantian Huang
authors: Qiaoge Li
authors: Shigeru Kuriyama
authors: Takafumi Taketomi
reviews: 
Comments: The paper introduces TANGO, a framework for generating high-fidelity co-speech gesture videos using a motion graph-based retrieval approach. It addresses audio-motion misalignment and visual artifacts in previous methods by implementing two key innovations: the AuMoCLIP, a hierarchical audio-motion joint embedding space for improved cross-modal alignment, and ACInterp, a diffusion-based interpolation network for generating high-quality transition frames. TANGO effectively synchronizes body gestures with speech audio, outperforming existing methods on datasets like Talkshow-Oliver and YouTube Business. The framework represents a significant advancement in gesture video generation, providing an open-source solution that integrates these novel components.
Strengths: 1. Contributions to Retrieval, Interpolation, and Datasets: The paper introduces TANGO, which enhances co-speech gesture video generation through improved retrieval methods, a diffusion-based interpolation model, and the introduction of the YouTube Business dataset.
2. Contribution to Open Source Gesture Generation: By making their code, pretrained models, and datasets publicly available, the authors significantly advance the field of gesture generation and encourage collaboration among researchers.
3. Comprehensive Experiments: The extensive evaluation of TANGO using various quantitative metrics and user studies demonstrates its superiority over existing state-of-the-art methods in generating realistic and audio-synchronized gesture videos.
Weaknesses: 1. Clarification on Graph Pruning Methodology: The paper does not provide sufficient detail on how the strongly connected component (SCC) subgraphs are merged, which is a critical operation in the Graph Pruning section.
2. Inconsistencies in Visual Representation: In Figure 3, the use of blue and green to represent different video motion clips lacks rigorous color correspondence during transitions, potentially leading to misunderstandings; similarly, Figure 5 uses yellow for both the Wav2Vec2 transformer feature and input audio, which could cause confusion.
5. Typographical Errors: There is a typographical error in Figure 4 where "merge" should be corrected to "merging."
4. Testing Results: The generated gesture movements exhibit stuttering, which undermines the fluidity and realism of the videos.
Recommendation: 10
Reviewer confidence: 5
Soundness/correctness: 4
Clarity: 4