TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio Motion Embedding and Diffusion Interpolation
Authors: Haiyang Liu
Xingchao Yang
Tomoya Akiyama
Yuantian Huang
Qiaoge Li
Shigeru Kuriyama
Takafumi Taketomi
Venue: ICLR 2025 Oral (2025)
Keywords: co-speech video generation
cross-modal retrieval
audio repsentation learning
motion repsentation learning
video frame interpolation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: We present TANGO, a framework for generating co-speech body-gesture videos. Given a few-minute, single-speaker reference video and target speech audio, TANGO produces high-fidelity videos with synchronized body gestures. TANGO builds on Gesture Video Reenactment (GVR), which splits and retrieves video clips using a directed graph structure - representing video frames as nodes and valid transitions as edges. We address two key limitations of GVR: audio-motion misalignment and visual artifacts in GAN-generated transition frames. In particular, i) we propose retrieving gestures using latent feature distance to improve cross-modal alignment. To ensure the latent features could effectively model the relationship between speech audio and gesture motion, we implement a hierarchical joint embedding space (AuMoClip); ii) we introduce the diffusion-based model to generate high-quality transition frames. Our diffusion model, Appearance Consistent Interpolation (ACInterp), is built upon AnimateAnyone and includes a reference motion module and homography background flow to preserve appearance consistency between generated and reference videos. By integrating these components into the graph-based retrieval framework, TANGO reliably produces realistic, audio-synchronized videos and outperforms all existing generative and retrieval methods. Our code, pretrained models, and datasets are publicly available at https://github.com/CyberAgentAILab/TANGO.

Published as a conference paper at ICLR 2025
TANGO: CO-SPEECH GESTURE VIDEO REENACT- MENT WITH HIERARCHICAL AUDIO MOTION EMBED- DING AND DIFFUSION INTERPOLATION
Haiyang Liu' Xingchao Yang*
Tomoya Akiyama
Yuantian Huang"
Qiaoge Li
Shigeru Kuriyama Takafumi Taketomi
'The University of Tokyo
CyberAgent
TToyohashi University of Technology
Generated
Figure I: TANGO is framework designed to generate co-speech gody-gesture videos using motion graph-baser retrieval approach. It first retrieves most the reference video clips that match the target speech audio by utilizing implicit hierarchical audio-motion embedding space. Then, adopts diffusion-based interpolation network to generate the remaining transition frames and smooth the discontinuities at clip boundaries.
ABSTRACT
We present TANGO, framework for generating co-speech body-gesture videos. Given a few-minute, single-speaker reference video and target speech audio, TANGO produces high-fidelity videos with synchronized body gestures. TANGO builds on Gesture Video Reenactment (GVR), which splits and retrieves video clips using a directed graph structure representing video frames as nodes and valid transitions edges. We address two key limitations of GVR: audio-motion misalignment and visual artifacts in GAN-generated transition frames. In partic- ular, (i) we propose retrieving gestures using latent feature distance to improve cross-modal alignment. To ensure the latent features could effectively model the relationship between speech audio and gesture motion, implement hierar- chical joint embedding space (AuMoCIiP)): (ii) we introduce the diffusion-based model generate high-quality transition frames. Our diffusion model, Appear- ance Consistent Interpolation (ACInterp), is built upon AnimateAnyone and in- cludes reference motion module and homography background flow to preserve appearance consistency between generated and reference videos. By integrating these components into the graph-based retrieval framework, TANGO reliably pro- duces realistic, syndhoisnccrr videos and outperforms all existing generative and retrieval methods. Our code, pretrained models, and datasets are publicly available at https: //tpthubttbtttt/
INTRODUCTION
This paper addresses the problem of generating high texture quality co-speech body gesture videos from reference speaker's talking video. Since significant progress has been made in talking face generation (Prajwal et al., 2020), our goal is to synchronize the body gestures in video with new, unseen speech audio. Successfully generating cesture-synccronizzer talking videos significantly reduce production costs in real-world applications, such as news broadcasting and virtual YouTube content creation.

---

Published as a conference paper at ICLR 2025
Generating gesture-synchromizee videos from audio is promising but presents challenges, as humans are sensitive to both the video textural quality and the relationship between gestures and the audio's s acoustic and semantic proper- ties. Existing methods could be broadly catego- rized into two groups: generative and retrieval. Generative methods (Ginosar et al., 2019; Qian al., 2021) generate all frames from given au- dio audio estimated 2D pose using video generation neural networks (Chan et al., 2019), while retrieval methods (Zhou et al., 2022), re- combine existing frames to match the audio and generate few transition frames for recom- bination boundaries. Generative methods fre-
Audio-Mottoo Misalignment
Blending Artfact
Figure 2: Limitations of GVR (Zhou et al., 2022). quently suffer from artifacts such as temporal blur in hand and cloth textures. This limitation motivates our choice of the retrieval method, which ensures higher video quality for real-world applications.
Audio-Driven Gesture Video Reenactment (GVR) (Zhou et al., 2022), to the best of our knowledge, the first and only retrieval-based method for gesture video generation. GVR splits videos into cqual-length sub-clips and reassembles them in motion graph-based (Kovar et al., 2008) approach. However, as shown Figure 2, GVR has characteristic artifacts in two key components. First, the alignment between the target speech audio and the retrieved gesture video is limited, as the retrieval naively relies on audio onset features and keyword matching. Second, the performance of its GAN- based interpolation network is limited by its ability to predict accurate optical flow (Fleet & Weiss, 2006; 1Ig et al., 2017), resulting in artifacts such as distorted hands.
To address these, we reproduce GVR's ss motion graph-based framework and introduce two improve- ments: an implicit feature distance-based gesture retrieval method and a diffusion- beffusioo-bbaed interpo- lation network. The former (AuMoCLIP) introduces a hicrarchical audio-motion joint embedding space to encode paired audio and motion modality data into a close latent space. The training pipeline is designed to split the low-level and high-level joint embedding space for learning local and global associations. After training, this joint embedding is adopted to retrieve gestures from the target's unseen audio. The latter, Appearance Consistent Interpolation (ACInterp), a diffusion- based interpolation network, leverages the power of existing video generation diffusion models, AnimateAnyone (Hu al., 2023), to eliminate the blur and ghost artifacts found in traditional flow- based interpolation methods (Huang et al., 2022; Kong et al., 2022; Reda et al., 2022; Lu et al., 2022; Zhou et al., 2022), and proposes utilizing homography background flow and reference motion module to preserve appearance consistency between generated and reference videos. By integrating these improvements, our method, TANGO, could produce plausible videos while accurately aligning gestures with audio inputs. Our contributions can be summarized as follows:
We propose hierarchical audio-motion joint emhedding space, AuMoCLIP, for accurately retrieving gestures hased on target speech audio. To the best of our knowledge, AuMoCLIP is the first work to present CLII--lkke embedding space between audio-motion modalities.
We introduce a diffusion-based interpolation network, ACInterp, reducing spatial and tem- poral video artifacts and generating uppearance-consister video clips.
We present reproduced and improved motion graph-based gesture retrieval framework featuring a graph pruning method to generate co-speech gesture videos of infinite length.
We release a small-scale, background- co-speech video dataset, YouTube Business, including data from 12 speakers to validate gesture video generation models.
We integrate the above components into TANGO; it outperforms existing generative and retrieval methods, both quantitatively and qualitatively, on the existing Kalkshow-Olivee and the newly introduced YouTube Business dataset.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
Our methodology is related to prior research on generative and retrieval- based co-speech video generation, cross-modal retrieval, and video frame interpolation.
Generative Co-Speech Video Generation. Generative approaches (Qian et al., 2021; Liu et al., 2022a; Yoon et al., 2020; Liu et al., 2022b; Yang et al., 2023a;b; Zhu al., 2023; Yi et al., 2023; Pang et al., 2023; Nyatsanga et al., 2023; He et al., 2024b) generate all frames from given audio via a two-stage pipeline. These methods, such as speech2gesture and speech-driven template (Ginosar et al., 2019; Qian et al., 2021), initially map audio poses through specialized networks, followed by employing a separate GAN-based pose2video pre-trained model (Chan et al., 2019) to transform these poses into video frames. The audio2Pose stage has been improved by emotion-aware architec- ture (Qi et al., 2023) and diffusion models (Mughal et al., 2024). Recent literature has improved the performance of pose2video with diffusion models. For instance, Animate (Hu et al., 2023) utilizes& reference-net attention-basee motion module (Guo et al., 2023) for spatial and temporal consistency. Overall, the skeleton-level results from Generative methods are typically aligned with the audio, the pose2video stage (Chan et al., 2019; Zhang et al., 2023b; Hu et al., 2023; Rombach et al., 2022) is the bottleneck which often suffers from artifacts such as temporal blur in hands and cloth textures. This is due to the network's need to handle higher resolutions, long-term, accurate temporal consistency, and varied body deformations. This limitation shows the benefit of our ap- proach, which reuses existing video frames. In method, pose2video is only required for short clips with start and end frames; this allows to maintain high-quality results with minimal artifacts.
Retrieval Co-Speech Video Generation. Gesture Video Reenactment (GVR) (Zhou et al., 2022) represents the first attempt to retrieve gesture motion from speech audio using a motion graph- based (Kovar et al., 2008) framework. It has three key steps: (i) creating a motion graph based on 3D motion and 2D image domain distances, (ii) retrieval of the optimal path within this graph for the target speech by audio onset and keyword matching, (iii) blending the discontinues frames by an interpolation network based on flow warping and GAN, Our method improves GVR by incorpo- rating learned feature-based retrieval and diffusion- based interpolation modules, resulting in better cross-modal alignment and high-quality transitions.
Cross-Modal Retrieval. Cross-modal retrieval aligns associations between different modalities within a learned feature space. The CLIP series (Radford et al., 2021; Li et al., 2022; 2023a) align text and images using contrastive learning. In the text-motion domain, MotionCLIP (Tevet et al., 2022) aligns motion with the frozen pretrained CLIPP text space. GestureDiffCI..j (Ao et al.) aligns gesture motion with speech transcripts using max pooling. However, directly using text-only fea- tures to retrieve gesture motion is challenging due to the lack of timing information. Unlike previous methods, we propose a joint embedding space directly between audio and motion modalities.
Video Frame Interpolation. Video frame interpolation (VFI) aims to create intermediate frames between two existing frames. The integration of optical flow-based techniques with deep learning is the mainstream approach for VFI (Liu et al., 2017: Jiang al., 2018; Niklaus & Liu, 2018; Xue et al., 2019; Niklaus & Liu, 2020; Park et al., 2020; 2021; Sim et al., 2021; Wu ct al., 2022; Danier et al., 2022; Kong et al., 2022; Reda et al., 2022; Huang al., 2022: L. et al., 2023b). Optical flow meth- ods estimate pixel movement between frames to guide the interpolation process. However, these methods still face challenges such as handling frequeryy details, large fast motions, occlu- sions, and balancing memory requirements. Hybrid models that combine CNNs with transformers (Lu et al., 2022: Zhang et al., 2023a; Park al., 2023) and diffusion models (Voleti et al., 2022; Danier et al., 2024; Jain et al., 2024) have recently demonstrated more consistent and sharper re- sults but require high memory capacity, e.g., 76GB training memory for 256x256 images (Lu al., 2022; Danier et al., 2024), making the extension of these methods to real-world high-resolution videos challenging. Our method addresses these challenges by integrating latent diffusion-based architectures and temporal priority to improve visual fidelity and computational efficiency.
Lip Synchronization in Co-Speech Video Generation. Similar to previous co-speech gesture video generation works (Zhou et al., 2022; He et al., 2024a), our method only focuses on body gestures and employs post-processing using Wav2Lip (Prajwal et al., 2020) for lip synchronization. The reason for separating lip-sync and body gesture generation is performance-drivee pipeline will have higher SyncNet score (Prajwal et al., 2020). Audio correlates more strongly with lip movements than with body gestures, making it beneficial to handle them separately.
i.e., separated
3

---

Published as a conference paper at ICLR 2025
Create Video Motion Graph
2. Retrival Motion Path by Audio
3. Interpolate Discominuous Playback Path
leumeed joint embecking
existing
O dPoll
directly playback
reference video
0.17
|0
reference video
motion
0.92
distance
noW path
Denerslive play
Figure 3: System Pipeline of TANGO. TANGO generates gesture video in three steps. Firstly, creates directed motion graph to represent video frames nodes and valid transitions edges. Each sampled path (in bold) dictates the selected playback order. Secondly, an audio-conditioned gesture retrieval module aims minimize cross-modal feature distance find path where ges- tures best match target audio. Lastly, a diffusion-basee interpolation model generates appearance- consistent connection frames when the transition edges do exist in the original reference video.
TANGO
Our TANGO, as shown in Figure is a mo- tion graph- based framework for reenacting ges- ture videos based on target speech audio. Ini- tially, we build upon the implementation of the VideoMotionGrraph baseline (Zhou al., 2022) and introduce graph pruning create directed Gesture Video Motion Graph (Section 3.1). In this graph. each node represents both the au- dio and image frames of the video, while each edge denotes valid transition between frames. Given a target audio, its temporal features are pre-trained audio-motion joint embedding network (AuMoCLIP). These fea- are then utilized retrieve a subset of video playback paths (Section 3.2). When transition edge does not exist within the original reference video, a frame interpolation network (ACInterp), is employed to ensure smooth tran- sitions (Section 3.3). This enables each transi- tion on the retrieved path to consist of realistic video frames. After the above three steps, TANGO reliably produces realistic, audio-synchronized gesture videos.
mades:3526 edger 7982 Video Marion Graphi WHO Prusing
Andor-2618 edacs
Video Modioo Graga Will PresiiR
Figure 4: Graph Pruning. We delete paths with dead endpoints by merging SCC subgraphs. i.e. those ending with node without out-degree in the initial Gesture Video Graph (left), and obtain strongly connected subgraph (right). Each node in the pruned graph is reachable from any other node within this subgraph, enabling efficient sampling of long video. The color of the paths represents different reference video clips for one speaker.
extracted
3.1 GRAPH CONSTRUCTION
Graph initalization. TANGO is represented as graph structure G(N, E) with nodes and edges. Similar to Gesture Video Reenactment (GVR) (Zhou al., 2022), nodes (nj nz...... n. are defined as I-frame, non-overlapping clips from reference videos, containing both RGB im- age frames and audio waveforms. The existence of valid transitions between nodes (edges)
Algorithm Graph Pruning Method for Enhancing Connectivity
Graph G Enhanced Graph G" Collect all SCC subgraphs in as Goce = (Go, GI...., Gn. and m = argmax, /Gy|, where denotes the size of subgraph
for each subgraph Gz(A Gm) in G.oo do if any nodes in G, not G, then for each node u in Gm do for each node in G; do Calculate the distance d(u, v) between nodes u and (i,j) arg min,, d(u, v)
Add bi-directional edges C;, and oj.á to G
4

---

Published as a conference paper at ICLR 2025
Proj. Layer
Eniodaa
Pretrained Feature
Concationat
Stop Gindient
Waveform
Doo edacto
D0O Fiih Cliggn
IP Self- Attentionn
CLS
High
D00
Doo
JP Selt- Atleation
Sumilazzy Matrix
CES
Hiersshiicl Enbiekding
Framo-Level InONCC
Sensence Level InfoNCE
Figure 5: AuMoCLIP. AuMoCLIP is pipeline to train hierarchical joint embedding. The audio waveform and extracted 3D motions are encoded in learned embedding space where paired audio and motion have closer distance than non con-paired samples. It employs dual-tower encoder architec- ture; each encoder split into low and high-level sub-encoder. Besides, it includes the pretrained Wav2Vec2 and BERT features make it work. The embedding is trained with frame-wise and clip-wise contrastive loss for local and global cross-modal alignment, respectively. We design the frame-wise loss by frames within close temporal window (i = t) are positive, while distant frames (i kt, i t) and (i + t, i + kf) are negative.
} is determined on the basis of the similarities from both 3D motion space and 2D image space. We calculate the similarity in 3D space from the positions of full body 3D> joints. The 3D pose extracted using a state-of-the-art open-source SMPL-X (Pavlakos et al., 2019) estimation method (Yi al., 2023). The pose dissimilarity D pose (n; n2) between any pair of clips is determined by taking the average the Euclidean distances for their positions and velocities across all joints.
(fe1.1
e........ e1.
The similarity in 2D image space is the Intersection- -ver- Union (IoU) for body segmentation and hand boundary boxes. The body segmentation represents the visible foreground area in the image, computed by MM-Segmentation (Contributors, 2020), and the bounding boxes for hands are ob- tained from MediaPipe (Lugaresi al., 2019). The (1 ToU) of their visible surface areas then estimates the image space dissimilarity for each pair of frames as Diou (n., ny).
By employing the distance d D pose (n,, n2) + Dion (n, ny), ) for any pair of nodes n,, nj. an edge eijj exists if their distances di fall below predefined thresholds. We leverage an adaptive threshold by averaging the transition distances 4,j (d,,-1 d,, + d1,++1/33 in the original video.
Graph Pruning. The initial motion graph obtained from GVR is limited connectivity, as shown in Figure 4, which reduces the efficiency of sampling longer-length path. Search algorithms such as Beam search and dynamic programming typically encounter dead endpoints-non nodees without outgo- ing edges-ing in the motion graph. In particular, the probability of sampling a path that ends at a dead endpoint increases with sample length, reaching 75.9% for randomly sampling 10-second video and 98.66 68.6% for a 30-second video. To address this issue, we introduce graph pruning method by merging the strongly connected component (SCC) subgraphs. A strongly connected component is maximal subgraph where each node is reachable from any other node within the subgraph. Specifi- cally, we employ Algorithm to obtain strongly connected component that sample videos of any length starting from any point in the graph. More details for graph merging are in Appendix.
3.2 AUDIO-CONDITIONER GESTURE RETRIEVAL
The original Gesture Video Reenactment utilized onset and hard-coded keywords for audio-based path searching. However, this method has several limitations: speakers may not move syn- chronously with the audio onset; ii) the binary nature of onset results in weak distinction among similar samples; iii) there is no matching result a keyword is not present in the reference video clips. These limitations lead to misaligned results. Therefore, we introduce a learning method to implicitly model the temporal association between audio and motion. As shown in Figure 5, mitiioommoionnoooom joint embedding to consider short audio-motion
approach learns hierarchical beat alignment and longer-term content similarity simultancously. To the best of our knowledge, AuMoCLIP is the first pipeline to learn CLIP-like features between gesture audio and motion
5

---

Published as a conference paper at ICLR 2025
modalities. We discuss our design in three key aspects: i) model architecture, ii) loss design, and iii)
training schedule.
Architecture of AuMoCLIP. Inspired by the CLIP-based contrastive learning framework and Mo- CoV2 (Chen et al., 2020), we start from a dual-tower architecture trained with a global InfoNCE loss. Our key design for audio-motion modalities is the split between low-level and high-level en- coders. Following Wav2 (Baevski et al., 2020), we represent audio as a raw waveform and use 7-1ayer CNN (low-level) and a I-layer Transformer (high-level) for the audio encoder. For motion, inspired by NeMF (Guo et al., 2022), we use a 1SD representation and a motion encoder consisting of a 28- 28-layer CNN (modified from TM2T (Guo et al., 2022)) and 1-layer Transformer. The first 10 layers of the CNN are used as the low-level motion encoder. As shown in Figure 5, we use a Projection MLP to map low-level features, while a Projection Self-Attention to obtain the CLSS token to summarize high-level features.
Since Wav2Vec2 is trained on large-scale human speech audio, we concatenate the frozen pretrained low-level and high-level features from Vec2 to enhance performance. However, encoding only audio waveforms is insufficient for high-level mapping between speech audio and gesture mo- tion because gestures are often related to speech transcripts, while Vac2 and our audio en- coder focus on "audio texture." " To address this, we include aaiinne BERT features using the Wey2Nc2CCTC model and pretrained BERT. We design word timing alignment method to align BERT features correctly without relying on MFA (see Appendix for details). This allows the audio branch to contain the necessary features for training the joint embedding.
Local and Global Contrastive Loss. We retain the InfoNCE loss for the CLS token in the mini- batch as the global contrastive loss and introduce a local contrastive learning task. Specifically, as shown in Figure 5, we define the frame-wise loss where frames within a close temporal window + t) are considered positive. while distant frames (i kt, t, i + t, i + kt) are considered negative. In this paper, we set f = 4 and &s = 4 for 30 FPS motion. This design proposes an casier learning task by accounting for slight misalignments in natural talking scenarios.
Stop Gradient for Low-Level Encoders. We aim to maximize both low-level and high-level re- trieval accuracy during training. Our observation shows that directly optimizing both losses de- creases the performance of the low-level encoder but improves the high-level encoder. This suggests that: i) the low-level encoder should not be trained jointly with the high-level encoder, and ii) in- cluding low-level features benefits the high-level encoder. Therefore, we stop the gradient from the global contrastive loss to the low-level encoder. This operation enables us to maximize the perfor- mance of both feature sets.
Teature-Based Gesture Retrieval. After training, we obtain two types of features: low-level fea- tures that can distinguish whether the current 8 8-frame features that can evaluate if the current 4-second audio-motion clip is paired. We leverage these features for retrieval by combine these two features, First, for each 4-second clip groundtruth mo- tion, we pre-calculate the high-level feature and repeat it for each frames. Next, we pre-calculate low-level feature and directly use its per-frame feature value. We then search for the best-matched path Pooth.match by maximizing the both low-level and high-level similarity between the motion and the target audio over the entire path via Dynamic Programming (DP).
pair is matched, and high-level
3.3 DIFFUSION-BAND VIDEO FRAME INTERPOLATION
The transition frames, synthesized from previous flow-based methods, often suffer from blur arti- facts. To improve this, as shown in Figure 6, we leverage the power of the two-stage (pose2image and image2video) video generation diffusion model, AnimateAnyone (Hu et al., 2023). Our method, ACInterp, generates target interpolation frames & E (i. using the existing start frames / E (i k i] and end frames t e [j.j + k), along with linear-blended 2D pose images and homography back- ground offsets.
Homography Offset-Refineh Pose2Image Stage. As shown in Figure 6, The Pose2Image stage aims to sample a random noise 2, and denoises it for estimated image latent âg. As same as An- imateAnyoneeHHu et al., 2023), ACInterp i) implements denosing progress in a latent space with pretrained VAE Encoder and Decoder EyAE, DvAE: ii) adds pose features from PoseGuider g to noisy latent space as the input DenoisingNet DD iii) incorporates ReferenceNet R. and CLIP Im-

---

Published as a conference paper at ICLR 2025
YAE,
x Reference Net
Powe2Image Stage
Image2Videe Stinge
DOR
image
Clli
Pase Guider
2 Denoising Net
all stapel imputs Video
M Medion Module
derooised Tattents
pose image
|
Animate Anyune
Self-Att Attestion nlong
ACTnterp (Qurs)
backgrond
offiel
Badigruund Guider
000 0000000000 SA with reference from start and and GT frames
000
Figure 6: ACInterp. Our Appearance Consistent Interpolation (ACInterp) model generates target interpolation frames (i.j) using the existing start frames (i k. i] and end frames t [i_j + k), along with linear-blended 2D pose images and homography background offsets. Based on AnimateAnyone ACInterp enhances appearance consistency in two ways. First, during the pose-to- image stage, estimated background pixel offsets are introduced to produce background-stable image results. Second, it uses the start and end frames as temporal priorities for the Motion Module to ensure human identity consistency. Achieving appearance consistency in transition frames is crucial for making Gesture Video Graph results appear natural.
age Encoder preserve consistent objects' appearances. Hierarchical features from the merged reference latent are concatenated to corresponding layers of the target DenoisingNet D to embed identity information. iv) calculates v-prediction (Rombach et al., 2022) loss for training.
Different from AnimateAnyone, we introduce additional homography background offset flow to eliminate artifacts due to camera parame- ter changes. As shown in Figure 7, the gen- erated images often cause the drift of objects in background, ignoring their fixed posi- tion in the reference background image. This is caused by overfitting the camera parameter changes in in-the-wild videos. To address this, we calculate an image-level background off- set flow Hi Rhxwx2 and add by Back- groundGuider B, the B has the same archi- tecture with g. Specifically, we compute the homography matrix between the reference and target images to calculate the pixel movement Ar, Ay for the background region. In particu- lar, we masked the foreground by human seg- mentation results from DeepLabv3, then ap- plied SIFT, FLANN, and RANSAC to keypoint detection, keypoint matching, and homography matrix computation, respectively.
Figure 7: Appearance Artifacts in Animate one (Hu al., 2023). The generated results (in the second frame) show i) position jitters due to training data camera movement, and ii) identity inconsistencies after the Motion Module refine- ment. Best viewed with Acrobat Reader. Click the images to play the clip.
Reference Motion badule-based Image2Videu Stage. As shown in Figure 6, the Image2Video stage captures temporal dependencies among video frames to mitigate the jitter effects in the Pose2Image stage. AnimateAnyone (Hu et al., 2023), as same as AnimateDifff (Guo et al., 2023), op- timizes residual selffattention--aae based motion module M within Denoising ' in this stage. The motion module reshapes feature map E I Bbxtxhxuxx to E [(xxxxx)xxxx and then performs temporal attention along the t dimension. However, as illustrated in Figure 7, this approach tends to produce averaged appearances across image sequences, resulting in diminished identity consis- tency. While this artifact may be negligible for other tasks where human identity is not important, substantially degrades the realism of gesture video graph.
Our analysis identifies that the key issue is the expansive solution space for the motion module, as self-attention applied exclusively along the temporal dimension t. To address this, we introduce additional conditioning to constrain the solution space. We train the motion module by randomly selecting start and end pixels to effectively reduce uncertainty. During training, for the feature map x IR (bxxhxx))xxx
we introduce probability p incorporate 4-frame ground truth latent

---

Published as a conference paper at ICLR 2025
Table 1: Evaluation for co-speech video generation on Show-Oliver and YuuTube Video dataset.
Show-Oliver
You'Tuhe Talking Video
FVD
FGD
BC
Diversity
FVD
FGD 4
BC1
Diversity
Ground Truth
0.326
3.514
0.435
3.746
SpeechDrivea Templatte (Qian et al., 2021)
2.239
5.722
0.401
1.950
7.612
5.559
0.461
2.081
ANGIE (Liu et al., 2022c)
2.079
5.112
0.359
2.577
D2G-Diffusion (He al., 2024a)
2.007
4.799
0.393
3.398
5.835
5.011
0.439
2.625
GVR (Zhou et al,, 2022)
1.615
4.246
0.270
4.623
4.027
2,900
0.331
3,573
TANGO (Ours)
1.379
3.714
0.375
5.393
3.133
2.068
0.479
4.128
features as reference. Leveraging classifier-free guidance, our conditional diffusion based motion module supports inference with or without these reference frames. The 4-frame segment alignment corresponds to the node length in the Gesture Video Reenactment. Finally, During inference, we introduce X a and 4 x 9 start and end conditional frames, generating intermediate 8 frames for transition edges.
4 EXPERIMENTS
4.1 DATASET
Our experiments are conducted on the open-source Show dataset (Yi al., 2023) and newly collected YouTube Video dataset. The Show dataset comprises 26 hours of talking videos featuring four speakers with varying backgrounds and irregular camera movements. We selected the speaker, Oliver, as these videos contain fewer interactions with the background. The YouTuue Video data is a scall-scale, less than one hour dataset from in-the-wild You'Tube videos characterized by clean backgrounds and fixed camera positions. More details of split for each dataset is in the APPENDIX.
4.2 EVALUATION OF GENERATED VIDEOS
We compare our method with the previous state-of-the-art Generative method SDT (Qian et al., 2021). ANGIE (Liu et al., 2022c), S2G-Diffusion (He et al., 2024a) and reassemble-batee method GVR (Zhou et al., 2022). We use the pertained weights from SDT and finetune the pose2ing stage with specific cloth. We evaluate ANGIE with the original paper test samples provided on Show- Oliver. For GVR, we reproduce the onset-based graph search and pose-aware neural rendering according to the implementation details in their paper.
Objective Evaluation We employ both video and kinematic Feature Distance (FVD (Carreira Zisserman, 2017) and FGD (Yoon al.,
Table 2: User Study on Talkshow-Oli Oliver.
SDT
GVR
TANGO
GT
Video Texture Quality
2020)) quantify feature-level discrepancies. Additionally, we utilize Beat Consistency (BC)
Audio-Mition Alignient 1
29.2%
10.9%
28.7%
31.2%
Overall Preference
4.9%
20.6%
36.9%
37.6%
(Liu et al., 2022d) and Diversity (Li et al., 2021) metrics to evaluate audio-motion synchronization and gesture diversity, respectively. As shown in Table 1, our method outperforms GVR and SDT across all metrics except for BC. The fully generated baseline exhibits greater flexibility in the out- put motion space, which results in better BC performance. However, compared to SDT, our method significantly improves video quality. Furthermore, compared GVR, our approach consistently shows improvement across all metrics, demonstrating that TANGO generates more realistic and suddo--sycchooizeed videos.
Subjective Evaluation As shown in Table 2, we conducted a user study across four results. 47 Participants were asked to assess each video based on i) which video is more physically accurate, ii) which video's content aligns more closely with the audio, and iii) overall, which video is more like a real video. Users compared videos from all four results
single row, with the order of videos randomly shuffled. A total of 60 video clips, each spanning 6 sec. We didn't include ANGIE in the user study due to not having enough result video clips. Some snapshots of a transition period shown in Figure 8, and the supplementary material includes video results. Our method scored comparable to the ground truth video and outperforms the existing generative method SDT and retrieval method GVR with clear margin.
8

---

Published as a conference paper at ICLR 2025
4.3 EVALUATION OF AUMOCLIP
We evaluate the effectiveness of AuMoCLIP by retrieving gesture motion sequences using target audio features and measure performance using retrieval accuracy. As shown in Table 3, we compute low-level and high-level retrieval accuracy. Low-level retrieval accuracy is calculated by randomly selecting an audio feature at frame and finding the motion frame within (i (i- 16, i + 16) with the highest cosine similarity. If this frame lies within (i - 4, + 4), it is marked as accurate. The final accuracy is averaged over 16K random pairs, so the random search will have an accuracy of 25%. Similarly, high-level retrieval accuracy is measured by selecting an audio high-level feature and comparing it against 256 motion candidates (1 paired motion + 255 negatives). If the highest cosine similarity corresponds to the paired motion, it is marked as accurate. This accuracy is averaged over 3K random pairs, and the random search will perform 0.391%. If the performance beats the random search, that means the model works correctly. The onset and keyword matching in GVR has a performance of 35.38% (low-level) and 1 1.288% (high-level), which is better than the random search. We then discuss the performance roadmap to the final AuMoCL.I.
Generative Features. One straightforward approach is to directly train an audio-to-motion network (Liu et al., 2022b) and compute the joint level distance between the generated motion and motion candidates for retrieval. However, this method yields lower-than-expected performance, achieving only 29.03% on low-level retrieval and 1.403% on high-level retrieval.
MaxPooling or Token, We then switch to a MoCoV2-based dual-tower contrastive learn- ing framework, starting with high-level features only. MoCoV2 is originally designed for im- ages, not sequential data like motion. One so- lution is to apply max pooling along the time axis for the global token, similar to (Ao et al.). However, while max pooling focuses on ac- curate local alignment, it is less effective for global retrieval, resulting in a performance of only 5.312%. To address this, we adopt adaptive global feature merge approach using
Table Comparison of features for audio-motion retrieval
Low Level
High Level
Rundom Seurch
25.00% +000055
0.391% +00.00%
Generative Features
29.03% +16 10%
1,403%
258.8%
Keyword (Zhou
2022)
1,288%
Onset (Zhou et al., 2022)
35.38%
+11.51%
Baseline (Max Pooling)
5.112%
1360%
Baseline (CLS Tokea)
11.84%
3028%
-War2Ve22
12.73% 3255%.
BERT
15.68%
1010%%
Wav2Ve22kBBERT
16.40%
4194S
Split (Low High)
47.94%
99.76%
17.83%
+4460%
Split (Low only)
65.57%
162. 2%
AuMoCLIP (4 Stop Grad.)
68.68%
163.8%
19.54% 4897%
the CLS token, which significantly improves performance to 11.84% We keep CLS token in the remained experiments.
Pretrained Audio Features from Wav2Vee2 and BERT. As shown in Table 3, incorporating pre- trained audio features, specifically time-aligned BERT features, significantly enhances the baseline's s performance from 11.84% to 15.68%. This improvement occurs because BERT captures high- dimensional language semantics rather than just "audio textures.", which is critical for co-speech gesture retrieval task.
Discussion of Low-Level Contrastive Learning, Including the low-level contrastive learning task consistently benefits high-level retrieval performance, suggesting that adding a more robust low- level feature improves high-level performance. Interestingly, we found that training only the low- level contrastive learning task achieves significantly better performance, reaching 65.57% These observations suggest that we should: i) incorporate learned low-level features into high-level em- bedding learning, and ii) avoid the influence of high-level learning on low-level features. Therefore, propose simply stopping the gradient to achieve the best performance for both features.
4.4 EVALUATION OF VIDEO BLENDING METHODS
We compare the effectiveness of the pro- posed diffusion-based video frame interpo- lation method by evaluating the quality of blended videos. The test set is from the same videos as other sections. Since the test videos in other sections vary in length (e.g., 3 to 10 seconds). In this section, we evenly sampled 8- frame clips, resulting in a 368-video test set for evaluating blending. Our approach is compared with the Pose Aware Neural Rendering in the original GVR (Zhou et al., 2022), the state-of-the-
Table 4: Comparison of video blending methods.
PSNR
LPDS
MOVIE
IVD
FRLM (Reda et al., 202)
35.43
0.072
74.85
1.358
VFIF (1 et al., 2022)
$4.91
0.077
79.42
1.777
AnimateAnyane (itu et al. 2023)
32.63
0.127
86.06
1.421
PANR (2hou et al., 2022)
35.18
0.071
75.02
1
AClmesp (Ours)
35.63
0.065
72.65
9

---

Published as a conference paper at ICLR 2025
Animate Anyone
FILM
VFIFomer
Pose Aware
Ours
Ground Truth
Neural Rendering
Figure 8: Comparison for Transition Frames Generation. From top to bottom, we show four snapshots in the same frames across different methods. Our method shows fewer artifacts in the hand regions and maintains appearance consistency with the GT frames. AnimateAnyone can recover the hands but loses appearance consistency. The flow-based methods FiLM and VFIFormer fail to estimate the flow for complex motions. resulting in the disappearance of hands. Pose Aware Neural Rendering shows better hand results but still suffers from artifacts such as blurring.
art flow-based blending method FiLM (Reda et al., 2022), VFIFormerr (Lu et al., 2022), and the diffusion- pose-guides video generation method AnimateAnyone (Hu et al., 2023). All meth- ods are re- ve-trained on the Show-Oliver dataset with a training and inference resolution of 768 x 768. We evaluate both image-level and video-level quality. For single images, utilize Image Error (L.1 pixel-level distance), Learned Perceptual Image Patch Similarity (LPIPS), and Peak Signal-to- Noise Ratio (PSNR). For videos, adopt the Mean Opinion Video Quality Estimation (MOVIE) and feature-level distance (FVD). The features for calculating LPIPS and FVD are obtained from pretrained AlexNet and I3D networks, respectively. The objective comparisons are shown in Ta- ble 4. Overall, our diffusion-be based interpolation model outperforms both previous flow-based and diffusion-barr based methods by clear margin, e.g., FVD 0.922 vs. the previous best 1.190. See Figure 8 and supplementary material for the resulting video.
CONCLUSION
We presented TANGO, a framework for generating high-fidelity videos where body gestures align with the target speech audio. To the best of our knowledge, TANGO is the first work to present CLIP- Like contrastive learning audio and motion modalities, and it is the first open-source motion graph and audio-driven video generation pipeline. In the future, we aim to extend the gesture video graph on general human motion videos, such as dance, sports and more.
Disclosures. The authors acknowledge that they are solely responsible for ensuring the legal com- pliance of their collected benchmark data. Specifically. Show-Oliver is derived from the publicly available ShokShow dataset (Yi et al., 2023). To ensure compliance with YouTube's 's terms of service, we provide the original video URLs along with the data preprocessing scripts for YouTube Business Dataset, allowing researchers to reconstruct the dataset under the same conditions.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Tenglong Ao, Zeyi Zhang, and Libin Liu. Gesturediffuclip: Gesture diffusion model with clip latents. ACM Trans. Graph. doi: 10. 145/3592097
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, wav2vec 2.0: frame- work for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 12449-122426, 2020.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 6299-6308, 2017.
Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. Everybody dance now. In Pro- ceedings of the IEEE/CVF international conference on computer vision, pp. 5933-5942, 2019.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 2020.
MMSegmentation Contributors, Mmsegmentation Openmmlab semantic segmentation toolbox and benchmark, 2020.
Duolikun Danier, Fan Zhang, and David R. Bull. St-mfnet: A spatio-temporal multi-flow network for frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 351 1-3521, 2022.
Duolikun Danier, Fan Zhang, and David R. Bull. LDMVFI: video frame interpolation with latent diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 1472- 1480, 2024.
David Fleet and Yair Weiss. Optical flow estimation. pp. 237-257, 2006.
Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, and Jitendra Malik. Learn- ing individual styles of conversational gesture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3497-3506, 2019.
Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In European Conference on Computer Vision, pp. 580-597. Springer, 2022.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023.
Chengan He, Jun Saito, James Zachary, Holly Rushmeier, and Yi Zhou. Nemf: Neural motion fields for kinematic animation. Advances in Neural Information Processing Systems, 35:4244-4256, 2022.
Xu He, Qiaochu Huang, Zhensong Zhang, Zhiwei Lin, Zhiyong Wu, Sicheng Yang, Minglei Li, Zhiyi Chen, Songeen Xu, and Xiaofci Wu. Co-speech gesture video generation via motion- decoupled diffusion model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2263-2273, 2024a.
Xu He, Qiaochu Huang, Zhensong Zhang, Zhiwei Lin, Zhiyong Wu, Sicheng Yang, Minglei Li, Zhiyi Chen, Songcen Xu. and Xiaofei Wu. Co-speech gesture video generation via motion- decoupled diffusson model. arXiv preprint arXiv:2404.018. 018622 2024b.
Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. arXiv preprint 1.Xiv:2311.1711 2023.
Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. Real-time intermediate flow estimation for video frame interpolation. European Conference on Computer Vision, pp- 624 642. Springer, 2022.
11

---

Published as a conference paper at ICLR 2025
Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet2.00 Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2462 2470, 2017.
Siddhant Jain, Daniel Watson, Eric Tabellion, Aleksander Holynski, Ben Poole, and Janne Kon- tkanen. Video interpolation with diffusion models. CoRR. abs/2404.01203, 2024. URL
https:// /ttps://doi... wrggro.ggooooooo 1t11211..88555 48550/ wrxiv.2400. 01203.
Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik G. Learned-Miller, and Jan Kautz. Super slomo: High quality estimation of multiple intermediate frames for video interpola- tion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 9000-9008, 2018.
Lingtong Kong, Boyuan Jiang, Donghao Luo, Wenqing Chu, Xiaoming Huang, Ying Tai, Chengjie Wang. and Jie Yang. Ifrnet: Intermediate feature refine network for efficient frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1969-1978, 2022.
Lucas Kovar, Michael Gleicher, and Frédéric H. Pighin. Motion graphs. pp. 51:1-51:10, 2008
Jing Li, Di Kang. Wenjie Pei, Xuefei Zhe, Ying Zhang, Zhenyu He. and Linchao Bao. Au- dio2gestures: Generating diverse gestures from speech audio with conditional variational au- toencoders. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 11293-11302. 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision- vision-language understanding and generation. In International conference on machine learning. pp. 12888-18888 12900. PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19744 19742. PMLR, 2023a.
Zhen Li, Zuo-Liang Zhu, Linghao Han, Qibin Hou, Chun-Le Guo, and Ming-Ming Cheng. AMT: all all-pairs multi-field transforms for efficient frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9801 19010, 2023b.
Haiyang Liu, Naoya Iwamoto, Zihao Zhu, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng. Disco: Disentangled implicit content and rhythm learning for diverse co-speech gestures synthe- sis. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 3764-3773, 2022a.
Haiyang Liu, Zihao Zhu, Naoya Iwamoto, Yichen Peng, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng, Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis. arXiv preprint arXiv:2203.05297, 2022b.
Xian Liu, Qianyi Wu, Hang Zhou, Yuanqi Du, Wayne Wu, Dahua Lin, and Ziwei Liu. Audio-driven co-speech gesture video generation. Advances in Neural Information Processing Systems, 35: 21386-21399, 2022c.
Xian Liu, Qianyi Wu, Hang Zhou, Yinghao Xu, Rui Qian, Xinyi Lin, Xiaowei Zhou, Wayne Wu, Bo Dai, and Bolei Zhou. Learning hierarchical cross-modal association for co-speech gesture generation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recog- nition, pp. 10462-10472, 2022d.
Ziwei Liu, Raymond A. Yeh, Xiaoou Tang, Yiming Liu, and Aseem Agarwala. Video frame synthe- sis using deep voxel flow. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4473-4481, 2017.
Liying Lu, Ruizheng Wu, Huaijia Lin, Jiangbo Lu, and Jiaya Jia. Video frame interpolation with transformer. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recog- nition, pp. 3532-3542, 2022.
12

---

Published as a conference paper at ICLR 2025
Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, et al. Mediapipe: A framework for building perception pipelines. arXiv preprint 1rXiv:1906.081 081.0, 2019.
Muhammad Hamza Mughal, Rishabh Dabral, Ikhsanul Habibie, Lucia Donatelli, Marc Habermann, and Christian Theobalt. Convofusion: Multi-modal conversational diffusion for co-speech ges- ture synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 2024.
Recognition, pp.
Simon Niklaus and Feng Liu. Context-aware synthesis video frame interpolation. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 1701- 1710, 2018.
Simon Niklaus and Feng Liu. Softmax splatting for video frame interpolation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5436-5445, 2020.
Simbarashe Nyatsanga, Taras Kucherenko, Chaitanya Ahuja, Gustav Fje Henter, and Michael Neff. A comprehensive review of data-driven co-speech gesture generation. Comput. Graph. Forum, 42 (2):669559,, 2023.
Kunkun Pang, Dafei Qin, Yingruo Fan, Julian Habekost, Takaaki Shiratori, Junichi Yamagishi, and Taku Komura. Bodyformer: Semantics-guided 3d body gesture synthesis with transformer. ACM Transactions Graphics (TOG), 42(4):1-12, 2023.
Junheum Park, Keunsoo Ko, Chul Lee, and Chang-Su Kim. BMBC: bilateral motion estimation with bilateral cost volume for video interpolation. In European Conference on Computer Vision, volume 12359, pp. 125, 2020.
Junheum Park, Chul Lee, and Chang-Su Kim. Asymmetric bilateral motion estimation for video frame interpolation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14519-14528 2021.
Junheum Park, Jintae Kim, and Chang-Su Kim. Biformer: Learning bilateral motion estimation via bilateral transformer for 4k video frame interpolation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 1568 1568-1577, 2023.
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dim- itrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from single image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 10975-10985, 2019.
KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multimedia, pp. 484 492, 2020.
Xingqun Qi, Jiahao Pan, Peng Li, Ruibin Yuan, Xiaowei Chi, Mengfei Li, Wenhan Luo, Wei Xue, Shanghang Zhang, Qifeng Liu, et al. Suaely-s emotion transition learning for diverse 3d co-speech gesture generation. arXiv preprint vrXiv:2311.17533 2023.
Shenhan Qian, Zhi Tu, Yihao Zhi, Wen Liu, and Shenghua Gao. Speech drives templates: Co- speech gesture synthesis with learned templates. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 11077-11086. 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, and Brian Curless. FILM: frame interpolation for large motion. In European Conference on Computer Vision, pp- 250-266. Springer, 2022.
13

---

Published as a conference paper at ICLR 2025
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10695, 2022.
Hyeonjun Sim, Jihyong Oh, and Munchurl Kim. XVFI: extreme video frame interpolation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14469- 14478, 2021.
Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen Or. Motionelip: Ex- posing human motion generation to clip space. In European Conference Computer Vision, pp. 358-374. Springer, 2022.
Vikram Voleti, Alexia diffusion for prediction, generation, and interpolation. In NeurlPS 2022, 2022.
and Chris Pal. MCVD masked conditional video
Yue Wu, Qiang Wen, and Qifeng Chen. Optimizing video prediction via video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17814-17823, 2022.
Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T. Freeman. Video enhancement with task-oriented flow. International Journal of Computer Vision, 127(8):1106-11) 125, 2019.
Sicheng Yang. Zhiyong Wu, Minglei Li, Zhensong Zhang, Lei Hao, Weihong Bao, Ming Cheng, and Long Xiao. Diffusestylegesture: Stylized audio-driven co-speech gesture generation with diffusion models. arXiv preprint arXiv:2305. 04919, 2023a.
Sicheng Yang. Zhiyong Wu, Minglei Li. Zhensong Zhang, Lei Hao, Weihong Bao, and Haolin Zhuang. Qpgesture: Quantization- based and phase-guided motion matching for natural speech- driven gesture generation. In IEEF/CVF Conference on Computer Vision and Puttern Recognition, CVPR, pp. 2321-2330 IEEE, June 2023b.
Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yandong Wen, Timo Bolkart, Dacheng Tao, and Michael Black. Generating holistic 3d human motion from speech. In CVPR, 2023.
Youngwoo Yoon, Bok Cha, Joo-Haeng Lee, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Geehyuk Lee. Speech gesture generation from the trimodal context of text, audio, and speaker identity. ACM Transactions on Graphics (TOG), 3966):1-16, 2020.
Guozhen Zhang, Yuhan Zhu, Haonan Wang, Youxin Chen, Gangshan Wu, and Limin Wang. Ex- tracting motion and appearance via inter-frame attention for efficient video frame interpolation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pp. 06822-662,.21 2023a.
Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023b.
Yang Zhou, Jimei Yang. Dingzeyu Li, Jun Saito, Deepali Aneja, and Evangelos Kalogerakis. Audio- driven neural gesture reenactment with video motion graphs. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3418-3428, 2022.
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations neural networks. In Proceedings of the IEFE/CVF conference on computer vision and pattern recognition, pp. 5745-5753 2019.
Lingting Zhu, Xian Liu, Xuanyu Liu, Rui Qian. Ziwei Liu, and Lequan Yu. Taming diffusion models for audio-driven co-speech gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10544-10553, 2023,
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.I DATASETS
Show-Oliver. The Show dataset comprises 26 hours of talking videos featuring four speakers with varying backgrounds and irregular camera movements. We selected the speaker, Oliver, as these videos contain fewer interactions with the background. The full Show-Oliver dataset contains 6546 video clips, ranging from 3 to 10 seconds. We evaluate our approach using multiple few-shot sets, each derived from a total 10 minutes randomly selected video clips with consistent clothing. These sets are divided into 80%, 10%, and 10% splits for training, validation, and testing, respec- tively.
YouTube Video Dataset. We further collect and process a small-scale, few-shot dataset from in in-the- wild YouTube videos characterized by clean backgrounds and fixed camera positions. These videos feature 12 speakers delivering presentations lasting 1 to minutes. We select four speakers of them and picks to 10-second video subsets as the test set and use the remaining videos for training and constructing the Gesture Video Reenactment.
We first collect raw YouTube videos featuring 12 different identities. These videos are then seg- mented into multiple clips based on the detected sentence boundaries in the audio. Face detector is utilized to ensure that all clips contain clear faces. Subsequently, post-processing is employed to eliminate background obstructions and automatically adjust the camera position for consistency. Fi- nally, we obtained 304 clips for different identities, with from total duration of 1 hour data. For each identity, the longer video clips are used for the validation set, while the remaining clips constitute the training set.
A.2 TIMING-ALIGNED BERT FEATURES WITHOUT MFA
To achieve time-aligned BERT features for audio without forced alignment (MFA), we combine Wav2Vec2 and BERT models through the following steps:
Transcription (ASR). We use Wav2Vec2 with a Connectionist Temporal Classification (CTC) head to obtain the logits, which represent the model's confidence scores for each possible to- ken at each time step. The logits are processed to generate sequence of predicted to- ken IDs for the audio input. These token IDs are then decoded into a transcription us- ing the Vec2 processor's vocabulary. For example, we have the alphabet sequence pn un "T", nn on "h", "e", nn """","""" """ """,""""""""""""" ""","""") in this step.
BERT Embedding. The transcription is tokenized using the BERT tokenizer, and the embeddings are obtained from the BERT model. The tokenizer in BERT conver the alphabet sequence into word sequence CLLS" "The", "First", "POS"].
Time Alignment. We align the Wav2Vec2-generatee tokens with the BERT tokens using character- level matching. In particular, for each audio frame, we assign the aligned BERT embedding as its feature. a match is not found, we fill the gap by using the nearest neighboring non-zero features, ensuring smooth transition in the feature sequence over time.
A.3 15D MOTION REPRESENTATION
We refer to NeMF (He et al., 2022) represent the motion at each time step t using a 15-dimensional (15D) feature vector for each joint. This representation captures only local motion. The 15D motion representation, X, E /x15 includes:
Firstly, the joint positions x E J23 represent the 3D coordinates of each joint relative to the root joint, providing the skeletal pose at time Secondly, the joint velocities x R capture the rate of change of joint positions.
Additionally, the representation includes joint rotations x; E /x6 which encode the orientation of each joint in a 6D rotation format (Zhou et al., 2019), allowing for a more robust and unambiguous rotation representation. Lastly, the angular velocities x / describe the rotational speed of each joint. This unified representation enables detailed and comprehensive modeling of both the position and movement dynamics of each joint.
15

---

Published as a conference paper at ICLR 2025
A.4 LIMITATIONS
The ACInterp inputs 2D pose images from linear 2D pose blending. During inference, the image- level pose guidance is obtained by linearly blending detected 2D pose sequences. Since the linear blending is applied independently to each axis, the result of potential 3D blending (#1, M/, =1) - (I2. 92. 22) is equivalent to blending 2D (21, Vi) + (22, y2) for the x- and y-axes. However, when the GT motion between 8-interpolated frame is non-linear, the generated results is slightly differ from the GT. We calculate the linear blending could work, ie., with 2D pose error smaller than threshold 0.005, on 83% clips for Talkshow-Olive Dataset.
Besides, our method requires reference videos with low-dynamic backgrounds, such as TalkShow Oliver and YouTubeeTalk datasets. Our methods do not work well on speakers with high-dynamic backgrounds. For example, other speakers in TalkShow often interact with the background black- board or move around while talking. resulting in non-stable backgrounds. The blending of highly different backgrounds within very short durations, egg., half a second, makes the results unnatural.
A.S TRAINING DETAILS AND SETTINGS
We trained AuMoCLIP with a learning rate of 5 x 10-1, batch size of 64, on a single Nvidia L4 (24 GB) GPU for 30 hours. For ACInterp we leveraged the pre-trained weights from the reproduced " AnyoateAnyone by Moore-Thread, then was finetuned with learning rate of 1 10 a batch size of 16 for the image stage, and batch size of for the video stage, using 4 A100 (80 GB) GPUs. The image and video stages were trained for 30k and 20k iterations, respectively, requiring 5-6 days.
A.6 DETAILS OF MERGING SMALLER SCCs TO THE LARGEST SCC
The graph pruning methodology enhances the connectivity of the motion graph G by merging its strongly connected components (SCCs). We firstly decompose the graph G into strongly connected components (SCCs), which are maximal subgraphs where every node is reachable from every other node within the same subgraph, denoted as Gscc {Go, GI...., Gn], where |G& represents the size (number of nodes) of the k-th SCC. Then, we select the largest SCC Gm as the primary component for merging.
Each smaller SCC G; (G; * Gm) is analyzed to determine whether any of its nodes are not in G, We will try to merge the G; Gm, If 1) disconnected nodes are found and there is more than 30 disconnected nodes (1 (1-second video), and 2) if the number of nodes in an SCC is smaller than n (set to n 100 in our implementation). These rules are to prevent merging small and isolated nodes into the main SCC.
Then, for each node u in Gm and each node v Gi we compute the distance d(u, v), where the distance is the similarity for pose positions 3D space and IOU distance on 2D space. We found the closest pair of nodes (u, v) that minimizes the distance. After determining the closest pair, we add bi-directional edges cu,v and Cr,u the original graph G. for effectively merging G; into G..- Finally, this iterative process produces an enhanced graph G' where paths of any desired length could be sampled from any starting node.
16


Reviews:
[{"review_id": "FXR5DbPidQ", "reviewer": "Reviewer_8mqD", "summary": "The paper introduces TANGO, a framework for generating high-fidelity co-speech gesture videos using a motion graph-based retrieval approach. It addresses audio-motion misalignment and visual artifacts in previous methods by implementing two key innovations: the AuMoCLIP, a hierarchical audio-motion joint embedding space for improved cross-modal alignment, and ACInterp, a diffusion-based interpolation network for generating high-quality transition frames. TANGO effectively synchronizes body gestures with speech audio, outperforming existing methods on datasets like Talkshow-Oliver and YouTube Business. The framework represents a significant advancement in gesture video generation, providing an open-source solution that integrates these novel components.", "questions": "See weaknesses above.", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Contributions to Retrieval, Interpolation, and Datasets: The paper introduces TANGO, which enhances co-speech gesture video generation through improved retrieval methods, a diffusion-based interpolation model, and the introduction of the YouTube Business dataset.\n2. Contribution to Open Source Gesture Generation: By making their code, pretrained models, and datasets publicly available, the authors significantly advance the field of gesture generation and encourage collaboration among researchers.\n3. Comprehensive Experiments: The extensive evaluation of TANGO using various quantitative metrics and user studies demonstrates its superiority over existing state-of-the-art methods in generating realistic and audio-synchronized gesture videos.", "weaknesses": "1. Clarification on Graph Pruning Methodology: The paper does not provide sufficient detail on how the strongly connected component (SCC) subgraphs are merged, which is a critical operation in the Graph Pruning section.\n2. Inconsistencies in Visual Representation: In Figure 3, the use of blue and green to represent different video motion clips lacks rigorous color correspondence during transitions, potentially leading to misunderstandings; similarly, Figure 5 uses yellow for both the Wav2Vec2 transformer feature and input audio, which could cause confusion.\n5. Typographical Errors: There is a typographical error in Figure 4 where \"merge\" should be corrected to \"merging.\"\n4. Testing Results: The generated gesture movements exhibit stuttering, which undermines the fluidity and realism of the videos.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T8sD0gtOP6", "reviewer": "Reviewer_NVti", "summary": "The authors present a co-speech gesture generation approach through a gesture video reenactment framework. To align video segments with audio, the author proposes an audio-motion joint embedding space using contrastive learning.\n\nFurthermore, an existing diffusion-based, pose-driven video generation model is employed to produce interpolated frames. By incorporating an additional reference motion module and a homography-based background flow, the method achieves improved appearance consistency.\n\nBoth qualitative and quantitative comparisons with existing state-of-the-art methods demonstrate that the proposed method outperforms alternatives.", "questions": "1. The appendix mentions, “We evaluate our approach using multiple few-shot sets.” Are each of the proposed models (AuMoCLIP, ACInterp) trained separately on only one of these few-shot sets?\n\n2. The author states, “we leverage the power of the two-stage (pose2image and image2video) video generation diffusion model, AnimateAnyone.” However, there are no details on what \"leverage\" means in this context. Are they merely using the model’s structural concept, or are they initializing with a pre-trained version of AnimateAnyone?\n\n3. Why does Section 4.4 use a different test set for evaluation?\n\n4. Why are there missing results in Table 3?\n\n5. In Figure 5, why is there a white segment within the low-level and high-level feature representations? According to the paper, it should include only the Wav2vec CNN feature and the Wav2Vec transformer feature with the BERT feature.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Compared to existing works, the video results of the proposed method are strong in terms of both visual quality and audio-gesture alignment.\n2.  The authors conduct thorough experiments to validate the effectiveness of the proposed method.\n3. In terms of novelty, I would categorize this work as moderate. Although using contrastive learning for audio-motion retrieval is not new, the author deserves credit for applying it within the speech gesture retrieval domain. Beyond this, other contributions seem more incremental, such as adding graph pruning to GVR and incorporating an additional reference motion module and homography-based background flow to the animation process.", "weaknesses": "1. Some claims are not sufficiently justified. For example:\n\n    The author claims, “BERT captures high-dimensional language semantics rather than just ‘audio textures,’ which is critical for the co-speech gesture retrieval task.” However, there is no analysis showing that the addition of BERT features actually generates language-semantics-aware gestures. It’s possible that the performance improvement is due to alignment information alone. If the author could conduct an experiment comparing the use of BERT features with using only CTC alignment information, this might demonstrate the necessity of BERT. Additionally, if the author could provide video examples showing that the generated gestures become more language-semantics-aware with the addition of BERT, it would further support the claim.\n\n2. Since the authors compare their work to other co-speech gesture video generation methods, it would be beneficial to include recent state-of-the-art works, such as *Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model (S2GMM).*”If the authors could add this to their evaluation, it would make the proposed work's claim to be state-of-the-art more convincing.\n\n3. Both the objective and subjective evaluations use only one character, Oliver, making it difficult to verify whether the proposed approach would outperform other methods in different circumstances.The authors could perform a comparison experiment with an additional character to demonstrate the generalizability of their approach. Furthermore, if the proposed method does not perform well on characters commonly used by other state-of-the-art methods (e.g., TALKSHOW, PATS), it would be essential to discuss these potential limitations.\n\n4. Writing and formatting improvements are needed for easier comprehension. For example:\n\n- 4.1 Abbreviations:\n    \n    - GVR stands for Gesture Video Reenactment in the abstract, but GVR is later used to refer to the specific work (Zhou et al., 2022).\n        \n    - The abbreviation for (Liu et al., 2022c) should be ANGIE, not ANGLE.\n        \n- 4.2 Tables:\n    \n    - In Table 2, methods are listed as columns, while in Table 1, they are listed as rows.\n        \n    - There are no captions to explain some non-trivial numbers in the tables, which could be helpful for interpretation.\n        \n\n5. Some explanations and implementation details are unclear or missing. Please refer to the question section for further clarification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kH6eNPKkzw", "reviewer": "Reviewer_UxDj", "summary": "This paper proposes a framework for generating co-speech body-gesture videos, dubbed TANGO. TANGO integrates a specifically designed Gesture Video Reenactment (GVR) module, which facilitates the realistic portrayal of gestures, alongside a hierarchical joint embedding space known as AuMoCLIP, which enhances the quality of video generation. The authors conducted extensive experiments that demonstrate the method's superiority over existing approaches in terms of well-rounded metrics and visualization results.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper proposes a framework for generating co-speech body-gesture videos, dubbed TANGO. TANGO integrates a specifically designed Gesture Video Reenactment (GVR) module and a hierarchical joint embedding space known as AuMoCLIP to facilitate high-quality video generation. The strengths of this work are summarized as follows:\n1. The authors present a novel and robust technical contribution through the introduction of a graph network baseline that adaptively incorporates the GVR and AuMoCLIP modules.\n2. The motivation for this research is clearly articulated, and the insights provided are compelling. It is important to highlight that a unified framework for co-speech gesture modeling from audio to video sequences holds significant relevance in this community.\n3. The dataset introduced in this paper is designed to facilitate ongoing research in gesture generation, providing valuable resources for future studies.\n4. The visualization results are well-executed, allowing reviewers to thoroughly evaluate the findings and methodologies presented.", "weaknesses": "However, there are still some questions for me. I encourage the authors to give reasonable responses. The final rating would be raised if the solution can solve my concerns.\n1. Some important related works might be missed, and I encourage the authors to discuss with them. For example, both of these two methods leverage the audio signal as input to generate human postures.\n \n[1] Qi, X., Pan, J., Li, P., Yuan, R., Chi, X., Li, M., ... & Guo, Y. (2024). Weakly-Supervised Emotion Transition Learning for Diverse 3D Co-speech Gesture Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10424-10434).\n\n[2] Mughal, M. H., Dabral, R., Habibie, I., Donatelli, L., Habermann, M., & Theobalt, C. (2024). ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1388-1398).\n\n2. In the experimental setting, it is very interesting to explore the frame interval with different settings rather than 4. Please note that this is not an ask for additional experiments in the rebuttal, and this does not impact the paper score. I encourage authors to conduct this discussion in the future.\n\n3. In the user study, did the authors assess participant engagement, specifically regarding whether responses were submitted too quickly or too slowly, and whether any such responses should be discounted? For example, whether the authors considered using response time thresholds or including control questions to check for consistent responses. I encourage authors to add these discussions in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BGV3p2p1hq", "reviewer": "Reviewer_JX53", "summary": "The paper presents TANGO, a novel framework for generating co-speech body-gesture videos that effectively synchronizes gestures with speech audio. The key contributions of TANGO include:\n\nMotion Graph-Based Retrieval: TANGO enhanced the motion graph-based model with more fine-grained designed AuMoCLIP to obtain relevant body gestures that align with the audio input, enhancing the realism and relevance of the generated gestures.\n\nDiffusion-Based Interpolation Network: The framework utilizes a diffusion-based approach for video frame interpolation, which improves the quality and fluidity of the generated videos, addressing limitations found in previous methods.\n\nThe authors conduct extensive evaluations, including both objective metrics and user studies, demonstrating that TANGO outperforms existing methods like Gesture Video Reenactment (GVR) and Speech-Driven Templates (SDT) in terms of video quality, audio-motion alignment, and overall user preference.", "questions": "1. Representing the gesture as the motion graph: While it is good to represent the gesture patterns from the video as the graph, the retrived features are discrete tokens not uniformly sampled. Will this reduce the quality of gesture generation as the speech audio in natural should be continuous?\n\n2. It seems like for high-level representation, the author only used one transformer layer upon the low-level features from CNN encoders. Is it possible the limited number of transformer level leads to a low performance for the global retrieval? I have this doubt because the author draw inspiration from Wav2Vec 2 for the model design. However, Wav2Vec ustilized 11 transformer layers upon the CNN encoder and the self-supervised learning is also based on the transformer layer feature. Is it possible adding additional layers of transformers can help improve the high-level retrieval?\n\n3. For one technical detail of the paper, why random search of low-level retrieval has a 25% accuracy? In addition, can the author include some other metrics like F1 score, recall? There should be mostly negative samples for the retrieval if my understanding is correct.\n\n4. One further question based on weakness 3, why the temporal contrastive learning is used for the low-level feature instead of applied to high-level features from transformer layers?\n\n5. Refer to Weakness 5, I am assuming the the camera during inference will be kept the same. One claim the author mentioned is \"homography background offset flow to eliminate artifacts due to camera parameter changes\". Does this only apply for the training but not inference for stablizing training?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "For the method design:\n\n1. This work presents a retrieval framework for gesture generation. It resolved a significant issue by the former work GVR (retrieval naively relies on audio onset features and keyword matching) \n\n2. The author represent the gesture patterns within the video as graph and present an efficient pruning strategy to achieve long-sequence representation. This strategy seems effective.\n\n3. The author propose the AuMoCLIP for aligning the two modalities and demonstrate an effective merging of text semantics, high-level audio, and low-level audio features. The ablation study for low-level and high-level retrieval is insightful with very detailed analysis and design. The author analyzed how the temporal alignment of two modalities can be achieved and also the high level semantics.\n\n4. While uplifting the geature pattern generation to the pixel-level video synthesis, the author did not simply reply on commonly used ReferenceNet-like modules but in addition propose homography background offset flow for the consideration of camera changes for realistic videos. The author did very deep analysis of temporal dependencies and resolved the background and jittering problems introduced by camera motion commonly seen in video generations.\n\n\nFor potential insight and benefits for Future studies:\n1. Auto-Evaluation: The AuMoCLIP seems to be a very good generalizable evaluation method for automatic evaluation of gesture motion naturalness conditioned on speech. Unlike FGD, diversity or Beat Alignment (BAS), which cannot precisely include the naturalness of motion patterns, the semantics or beat alignment of audio and speech, this model might be very beneficial for future auto-evaluations, though the author did not propose this point.\n\n2. Audio and Gesture Motion Modality alignment: Previous works in this domain have very limited analysis of the how the speech signal functions as the trigger for gesture patterns. Many previous works, (like DiffSheg, CaMN, TalkShow, EMAGE, AMUSE, etc) only designed the generator by incorporating the audio features or text as the control signals to drive the gesture generation but never analyzed how the inner relationships of different modality should be. GestureDiffuCLIP did preliminary studies in aligning the speech word context and gesture motion patterns but ignored the temporal alignment between modalities. In this work, the author did very detailed analysis of the low-level retrieval and high-level retrieval, reconsider the temporal alignment and semantic alignment (low-level indicates the temporal alignment, I consider the local temporal contrastive loss might functions for the beat-dynamic of the gestures, high-level indicates the global semantic alignment of the two modalities. The ablation of the retrieval is very insightful)", "weaknesses": "1. The dataset for comparison are limited. The author only selected Oliver in the Show dataset and a small scale in-the-wild YouTube videos. (However, as far as I know, this filed lacks of high-quality data, PATS is another option, but it contains too many low-quality/low-resolution videos and blurry frames for large shoulder and hand motions)\n\n2. It seems like the temporal contrastive learning is based on the low-level features obtained from the CNN encoder. I am not quite sure if I understand it correctly. It will be better if the author can include this information in the final draft for easy understanding.\n\n3. For the method design, in case I am uncertain of some technical details, I will present some potential weakness in the Question section instead of here.\n\n4. For the gesture motion representation, the author utilized 3D SMPL/SMPL-X template model parameters. I assume the author based on the first frame of the given image, do the tracking to obtain the camera and based on the camera to project the generated 3D joints onto the 2D-space for image-level pose guided generation. I think the author could add this information to the final draft.\n\n5. Some of the training details of the model is missing, like what are the GPUs for training, time for training, learning rate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)