TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio Motion Embedding and Diffusion Interpolation

## Abstract

Abstract We present TANGO, a framework for generating co-speech body-gesture videos. Given a few-minute, single-speaker reference video and target speech audio, TANGO produces high-fidelity videos with synchronized body gestures. TANGO builds on Gesture Video Reenactment (GVR), which splits and retrieves video clips using a directed graph structure - representing video frames as nodes and valid transitions as edges. We address two key limitations of GVR: audio-motion misalignment and visual artifacts in GAN-generated transition frames. In particular, (i) we propose retrieving gestures using latent feature distance to improve cross-modal alignment. To ensure the latent features could effectively model the relationship between speech audio and gesture motion, we implement a hierarchical joint embedding space (AuMoCLIP); (ii) we introduce the diffusion-based model to generate high-quality transition frames. Our diffusion model, Appearance Consistent Interpolation (ACInterp), is built upon AnimateAnyone and includes a reference motion module and homography background flow to preserve appearance consistency between generated and reference videos. By integrating these components into the graph-based retrieval framework, TANGO reliably produces realistic, audio-synchronized videos and outperforms all existing generative and retrieval methods. Our codes and pretrained models are available: https://pantomatrix.github.io/TANGO/

## 1 Introduction

This paper addresses the problem of generating high texture quality co-speech body gesture videos from a reference speaker’s talking video. Since significant progress has been made in talking face generation (Prajwal et al., ), our goal is to synchronize the body gestures in video with new, unseen speech audio. Successfully generating gesture-synchronized talking videos can significantly reduce production costs in real-world applications, such as news broadcasting and virtual YouTube content creation.

Figure: Figure 2: Limitations of GVR (Zhou et al., ).
Refer to caption: x2.png

Generating gesture-synchronized videos from audio is promising but presents challenges, as humans are sensitive to both the video textural quality and the relationship between gestures and the audio’s acoustic and semantic properties. Existing methods could be broadly categorized into two groups: generative and retrieval. Generative methods (Ginosar et al., ; Qian et al., ) generate all frames from given audio or audio estimated 2D pose using video generation neural networks (Chan et al., ), while retrieval methods (Zhou et al., ), recombine existing frames to match the audio and generate a few transition frames for recombination boundaries. Generative methods frequently suffer from artifacts such as temporal blur in hand and cloth textures. This limitation motivates our choice of the retrieval method, which ensures higher video quality for real-world applications.

Audio-Driven Gesture Video Reenactment (GVR) (Zhou et al., ), to the best of our knowledge, is the first and only retrieval-based method for gesture video generation. GVR splits videos into equal-length sub-clips and reassembles them in a motion graph-based (Kovar et al., ) approach. However, as shown in Figure 2, GVR has characteristic artifacts in two key components. First, the alignment between the target speech audio and the retrieved gesture video is limited, as the retrieval naively relies on audio onset features and keyword matching. Second, the performance of its GAN-based interpolation network is limited by its ability to predict accurate optical flow (Fleet & Weiss, ; Ilg et al., ), resulting in artifacts such as distorted hands.

To address these, we reproduce GVR’s motion graph-based framework and introduce two improvements: an implicit feature distance-based gesture retrieval method and a diffusion-based interpolation network. The former (AuMoCLIP) introduces a hierarchical audio-motion joint embedding space to encode paired audio and motion modality data into a close latent space. The training pipeline is designed to split the low-level and high-level joint embedding space for learning local and global associations. After training, this joint embedding is adopted to retrieve gestures from the target’s unseen audio.
The latter, Appearance Consistent Interpolation (ACInterp), a diffusion-based interpolation network, leverages the power of existing video generation diffusion models, AnimateAnyone (Hu et al., ), to eliminate the blur and ghost artifacts found in traditional flow-based interpolation methods (Huang et al., ; Kong et al., ; Reda et al., ; Lu et al., ; Zhou et al., ), and proposes utilizing homography background flow and reference motion module to preserve appearance consistency between generated and reference videos. By integrating these improvements, our method, TANGO, could produce plausible videos while accurately aligning gestures with audio inputs. Our contributions can be summarized as follows:

- •
We propose a hierarchical audio-motion joint embedding space, AuMoCLIP, for accurately retrieving gestures based on target speech audio. To the best of our knowledge, AuMoCLIP is the first work to present CLIP-like embedding space between audio-motion modalities.
- •
We introduce a diffusion-based interpolation network, ACInterp, reducing spatial and temporal video artifacts and generating appearance-consistent video clips.
- •
We present a reproduced and improved motion graph-based gesture retrieval framework featuring a graph pruning method to generate co-speech gesture videos of infinite length.
- •
We release a small-scale, background-clean co-speech video dataset, YouTube Business, including data from 12 speakers to validate gesture video generation models.
- •
We integrate the above components into TANGO; it outperforms existing generative and retrieval methods, both quantitatively and qualitatively, on the existing Talkshow-Oliver and the newly introduced YouTube Business dataset.

## 2 Related Work

Our methodology is related to prior research on generative and retrieval-based co-speech video generation, cross-modal retrieval, and video frame interpolation.

Generative Co-Speech Video Generation. Generative approaches (Qian et al., ; Liu et al., ; Yoon et al., ; Liu et al., ; Yang et al., ; ; Zhu et al., ; Yi et al., ; Pang et al., ; Nyatsanga et al., ; He et al., ) generate all frames from given audio via a two-stage pipeline. These methods, such as speech2gesture and speech-driven template (Ginosar et al., ; Qian et al., ), initially map audio to poses through specialized networks, followed by employing a separate GAN-based pose2video pre-trained model (Chan et al., ) to transform these poses into video frames. Recent literature has improved the performance of pose2video with diffusion models. For instance, AnimateAnyone (Hu et al., ) utilizes a reference-net attention-based motion module (Guo et al., ) for spatial and temporal consistency. Overall, the skeleton-level results from Generative methods are typically aligned with the audio, the pose2video stage (Chan et al., ; Zhang et al., ; Hu et al., ; Rombach et al., ) is the bottleneck which often suffers from artifacts such as temporal blur in hands and cloth textures. This is due to the network’s need to handle higher resolutions, long-term, accurate temporal consistency, and varied body deformations. This limitation shows the benefit of our approach, which reuses existing video frames. In our method, pose2video is only required for short clips with start and end frames; this allows our method to maintain high-quality results with minimal artifacts.

Retrieval Co-Speech Video Generation. Gesture Video Reenactment (GVR) (Zhou et al., ) represents the first attempt to retrieve gesture motion from speech audio using a motion graph-based (Kovar et al., ) framework. It has three key steps: (i) creating a motion graph based on 3D motion and 2D image domain distances, (ii) retrieval of the optimal path within this graph for the target speech by audio onset and keyword matching, (iii) blending the discontinues frames by an interpolation network based on flow warping and GAN. Our method improves GVR by incorporating learned feature-based retrieval and diffusion-based interpolation modules, resulting in better cross-modal alignment and high-quality transitions.

Cross-Modal Retrieval. Cross-modal retrieval aligns associations between different modalities within a learned feature space. The CLIP series (Radford et al., ; Li et al., ; ) align text and images using contrastive learning. In the text-motion domain, MotionCLIP (Tevet et al., ) aligns motion with the frozen pretrained CLIP text space. GestureDiffCLIP () aligns gesture motion with speech transcripts using max pooling. However, directly using text-only features to retrieve gesture motion is challenging due to the lack of timing information. Unlike previous methods, we propose a joint embedding space directly between audio and motion modalities.

Video Frame Interpolation. Video frame interpolation (VFI) aims to create intermediate frames between two existing frames. The integration of optical flow-based techniques with deep learning is the mainstream approach for VFI (Liu et al., ; Jiang et al., ; Niklaus & Liu, ; Xue et al., ; Niklaus & Liu, ; Park et al., ; ; Sim et al., ; Wu et al., ; Danier et al., ; Kong et al., ; Reda et al., ; Huang et al., ; Li et al., ). Optical flow methods estimate pixel movement between frames to guide the interpolation process.
However, these methods still face challenges such as handling high-frequency details, large or fast motions, occlusions, and balancing memory requirements. Hybrid models that combine CNNs with transformers (Lu et al., ; Zhang et al., ; Park et al., ) and diffusion models (Voleti et al., ; Danier et al., ; Jain et al., ) have recently demonstrated more consistent and sharper results but require high memory capacity, e.g., 76GB training memory for 256x256 images (Lu et al., ; Danier et al., ), making the extension of these methods to real-world high-resolution videos challenging. Our method addresses these challenges by integrating latent diffusion-based architectures and temporal priority to improve visual fidelity and computational efficiency.

Lip Synchronization in Co-Speech Video Generation. Similar to previous co-speech gesture video generation works (Zhou et al., ; He et al., ), our method only focuses on body gestures and employs post-processing using Wav2Lip (Prajwal et al., ) for lip synchronization. The reason for separating lip-sync and body gesture generation is performance-driven, i.e., separated pipeline will have a higher SyncNet score (Prajwal et al., ). Audio correlates more strongly with lip movements than with body gestures, making it beneficial to handle them separately.

Figure: Figure 3: System Pipeline of TANGO. TANGO generates gesture video in three steps. Firstly, it creates a directed motion graph to represent video frames as nodes and valid transitions as edges. Each sampled path (in bold) dictates the selected playback order. Secondly, an audio-conditioned gesture retrieval module aims to minimize cross-modal feature distance to find a path where gestures best match target audio. Lastly, a diffusion-based interpolation model generates appearance-consistent connection frames when the transition edges do not exist in the original reference video.
Refer to caption: x3.png

## 3 TANGO

Figure: Figure 4: Graph Pruning. We delete paths with dead endpoints by merge SCC subgraphs. i.e., those ending with a node without out-degree in the initial Gesture Video Graph (left), and obtain a strongly connected subgraph (right). Each node in the pruned graph is reachable from any other node within this subgraph, enabling efficient sampling of long video. The color of the paths represents different reference video clips for one speaker.
Refer to caption: x4.png

Our TANGO, as shown in Figure 3, is a motion graph-based framework for reenacting gesture videos based on target speech audio. Initially, we build upon the implementation of the VideoMotionGraph baseline (Zhou et al., ) and introduce graph pruning to create a directed Gesture Video Motion Graph (Section 3.1). In this graph, each node represents both the audio and image frames of the video, while each edge denotes a valid transition between frames. Given a target audio, its temporal features are extracted via a pre-trained audio-motion joint embedding network (AuMoCLIP). These features are then utilized to retrieve a subset of video playback paths (Section 3.2). When a transition edge does not exist within the original reference video, a frame interpolation network (ACInterp), is employed to ensure smooth transitions (Section 3.3). This enables each transition on the retrieved path to consist of realistic video frames. After the above three steps, TANGO reliably produces realistic, audio-synchronized gesture videos.

Figure: Algorithm 1 Graph Pruning Method for Enhancing Connectivity

### 3.1 Graph Construction

Graph initalization.
TANGO is represented as a graph structure $\mathbf{G}(\mathbf{N},\mathbf{E})$ with nodes and edges. Similar to Gesture Video Reenactment (GVR) (Zhou et al., ), nodes $\mathbf{N}=\{\mathbf{n}_{1},\mathbf{n}_{2},\ldots,\mathbf{n}_{i}\}$ are defined as 4-frame, non-overlapping clips from reference videos, containing both RGB image frames and audio waveforms. The existence of valid transitions between nodes (edges) $\mathbf{E}=\{\mathbf{e}_{1,1},\mathbf{e}_{1,2},\ldots,\mathbf{e}_{i,j}\}$ is determined on the basis of the similarities from both 3D motion space and 2D image space.We calculate the similarity in 3D space from the positions of full body 3D joints. The 3D pose is extracted using a state-of-the-art open-source SMPL-X (Pavlakos et al., ) estimation method (Yi et al., ). The pose dissimilarity $\mathcal{D}_{\text{pose}}(\mathbf{n}_{i},\mathbf{n}_{j})$ between any pair of clips is determined by taking the average of the Euclidean distances for their positions and velocities across all joints.

The similarity in 2D image space is the Intersection-over-Union (IoU) for body segmentation and hand boundary boxes. The body segmentation represents the visible foreground area in the image, computed by MM-Segmentation (Contributors, ), and the bounding boxes for hands are obtained from MediaPipe (Lugaresi et al., ). The ($1-IoU$) of their visible surface areas then estimates the image space dissimilarity for each pair of frames as $\mathcal{D}_{\text{iou}}(\mathbf{n}_{i},\mathbf{n}_{j})$.

By employing the distance $d_{i,j}=\mathcal{D}_{\text{pose}}(\mathbf{n}_{i},\mathbf{n}_{j})+\mathcal{D}_{
\text{iou}}(\mathbf{n}_{i},\mathbf{n}_{j})$, for any pair of nodes $\mathbf{n}_{i},\mathbf{n}_{j}$, an edge $\mathbf{e}_{i,j}$ exists if their distances $d_{i,j}$ fall below predefined thresholds. We leverage an adaptive threshold by averaging the transition distances $t_{i,j}=(d_{i,i-1}+d_{i,i}+d_{i,i+1})/3$ in the original video.

Graph Pruning.
The initial motion graph obtained from GVR is limited in connectivity, as shown in Figure 4, which reduces the efficiency of sampling a longer-length path. Search algorithms such as Beam search and dynamic programming typically encounter dead endpoints—nodes without outgoing edges—in the motion graph. In particular, the probability of sampling a path that ends at a dead endpoint increases with sample length, reaching 75.9% for randomly sampling a 10-second video and 98.6% for a 30-second video. To address this issue, we introduce a graph pruning method by merging the strongly connected component (SCC) subgraphs. A strongly connected component is a maximal subgraph where each node is reachable from any other node within the subgraph. Specifically, we employ Algorithm 1 to obtain a strongly connected component that can sample videos of any length starting from any point in the graph.

Figure: Figure 5: AuMoCLIP. AuMoCLIP is a pipeline to train hierarchical joint embedding. The audio waveform and extracted 3D motions are encoded in a learned embedding space where paired audio and motion have a closer distance than non-paired samples. It employs dual-tower encoder architecture; each encoder is split into low and high-level sub-encoder. Besides, it includes the pretrained Wav2Vec2 and BERT features to make it work. The embedding is trained with a frame-wise and clip-wise contrastive loss for local and global cross-modal alignment, respectively. We design the frame-wise loss by frames within a close temporal window ($i\pm t$) are positive, while distant frames ($i-kt$, $i-t$) and ($i+t$, $i+kt$) are negative.
Refer to caption: x5.png

### 3.2 Audio-Conditioned Gesture Retrieval

The original Gesture Video Reenactment utilized onset and hard-coded keywords for audio-based path searching. However, this method has several limitations: i) speakers may not move synchronously with the audio onset; ii) the binary nature of onset results in weak distinction among similar samples; iii) there is no matching result if a keyword is not present in the reference video clips. These limitations lead to misaligned results. Therefore, we introduce a learning method to implicitly model the temporal association between audio and motion. As shown in Figure 5, our approach learns a hierarchical audio-motion joint embedding to consider short audio-motion beat alignment and longer-term content similarity simultaneously.
To the best of our knowledge, our AuMoCLIP is the first pipeline to learn CLIP-like features between gesture audio and motion modalities. We discuss our design in three key aspects: i) model architecture, ii) loss design, and iii) training schedule.

Architecture of AuMoCLIP. Inspired by the CLIP-based contrastive learning framework and MoCoV2 (Chen et al., ), we start from a dual-tower architecture trained with a global InfoNCE loss. Our key design for audio-motion modalities is the split between low-level and high-level encoders. Following Wav2Vec2 (Baevski et al., ), we represent audio as a raw waveform and use a 7-layer CNN (low-level) and a 1-layer Transformer (high-level) for the audio encoder. For motion, inspired by NeMF (Guo et al., ), we use a 15D representation and a motion encoder consisting of a 28-layer CNN (modified from TM2T (Guo et al., )) and a 1-layer Transformer. The first 10 layers of the CNN are used as the low-level motion encoder. As shown in Figure 5, we use a Projection MLP to map low-level features, while a Projection Self-Attention to obtain the CLS token to summarize high-level features.

Since Wav2Vec2 is trained on large-scale human speech audio, we concatenate the frozen pretrained low-level and high-level features from Wav2Vec2 to enhance performance. However, encoding only audio waveforms is insufficient for high-level mapping between speech audio and gesture motion because gestures are often related to speech transcripts, while Wav2Vec2 and our audio encoder focus on ”audio texture.” To address this, we include timing-aligned BERT features using the Wav2Vec2CTC model and pretrained BERT. We design a word timing alignment method to align BERT features correctly without relying on MFA (see Appendix for details). This allows the audio branch to contain the necessary features for training the joint embedding.

Local and Global Contrastive Loss. We retain the InfoNCE loss for the CLS token in the mini-batch as the global contrastive loss and introduce a local contrastive learning task. Specifically, as shown in Figure 5, we define the frame-wise loss where frames within a close temporal window ($i\pm t$) are considered positive, while distant frames ($i-kt$, $i-t$, $i+t$, $i+kt$) are considered negative. In this paper, we set $t=4$ and $k=4$ for 30 FPS motion. This design proposes an easier learning task by accounting for slight misalignments in natural talking scenarios.

Stop Gradient for Low-Level Encoders. We aim to maximize both low-level and high-level retrieval accuracy during training. Our observation shows that directly optimizing both losses decreases the performance of the low-level encoder but improves the high-level encoder. This suggests that: i) the low-level encoder should not be trained jointly with the high-level encoder, and ii) including low-level features benefits the high-level encoder. Therefore, we stop the gradient from the global contrastive loss to the low-level encoder. This operation enables us to maximize the performance of both feature sets.

Feature-Based Gesture Retrieval. After training, we obtain two types of features: low-level features that can distinguish whether the current 8-frame audio-motion pair is matched, and high-level features that can evaluate if the current 4-second audio-motion clip is paired. We leverage these features for retrieval by combine these two features, First, for each 4-second clip groundtruth motion, we pre-calculate the high-level feature and repeat it for each frames. Next, we pre-calculate low-level feature and directly use its per-frame feature value. We then search for the best-matched path $P_{both_match}$ by maximizing the both low-level and high-level similarity between the motion and the target audio over the entire path via Dynamic Programming (DP).

Figure: Figure 6: ACInterp. Our Appearance Consistent Interpolation (ACInterp) model generates target interpolation frames $t\in(i,j)$ using the existing start frames $t\in(i-k,i]$ and end frames $t\in[j,j+k)$, along with linear-blended 2D pose images and homography background offsets. Based on AnimateAnyone, ACInterp enhances appearance consistency in two ways. First, during the pose-to-image stage, estimated background pixel offsets are introduced to produce background-stable image results. Second, it uses the start and end frames as temporal priorities for the Motion Module to ensure human identity consistency. Achieving appearance consistency in transition frames is crucial for making Gesture Video Graph results appear natural.
Refer to caption: x6.png

### 3.3 Diffusion-Based Video Frame Interpolation

The transition frames, synthesized from previous flow-based methods, often suffer from blur artifacts. To improve this, as shown in Figure 6, we leverage the power of the two-stage (pose2image and image2video) video generation diffusion model, AnimateAnyone (Hu et al., ). Our method, ACInterp, generates target interpolation frames $t\in(i,j)$ using the existing start frames $t\in(i-k,i]$ and end frames $t\in[j,j+k)$, along with linear-blended 2D pose images and homography background offsets.

Homography Offset-Refined Pose2Image Stage. As shown in Figure 6, The Pose2Image stage aims to sample a random noise $z_{t}$ and denoises it for estimated image latent $\hat{z}_{0}$. As same as AnimateAnyone(Hu et al., ), ACInterp i) implements denosing progress in a latent space with pretrained VAE Encoder and Decoder $\mathcal{E}_{\text{VAE}},\mathcal{D}_{\text{VAE}}$; ii) adds pose features from PoseGuider $\mathcal{G}$ to noisy latent space as the input to DenoisingNet $\mathcal{D}$; iii) incorporates a ReferenceNet $\mathcal{R}$ and CLIP Image Encoder $\mathcal{C}$ to preserve consistent objects’ appearances. Hierarchical features from the merged reference latent are concatenated to corresponding layers of the target DenoisingNet $\mathcal{D}$ to embed identity information. iv) calculates v-prediction (Rombach et al., ) loss for training.

Figure: Figure 7: Appearance Artifacts in AnimateAnyone (Hu et al., ). The generated results (in the second frame) show i) position jitters due to training data camera movement, and ii) identity inconsistencies after the Motion Module refinement. Best viewed with Acrobat Reader. Click the images to play the clip.

Different from AnimateAnyone, we introduce additional homography background offset flow to eliminate artifacts due to camera parameter changes. As shown in Figure 7, the generated images often cause the drift of objects in a background, ignoring their fixed position in the reference background image. This is caused by overfitting the camera parameter changes in in-the-wild videos. To address this, we calculate an image-level background offset flow $H_{i,k}\in\mathbb{R}^{h\times w\times 2}$ and add it by BackgroundGuider $\mathcal{B}$, the $\mathcal{B}$ has the same architecture with $\mathcal{G}$. Specifically, we compute the homography matrix between the reference and target images to calculate the pixel movement $\Delta x,\Delta y$ for the background region. In particular, we masked the foreground by human segmentation results from DeepLabv3, then applied SIFT, FLANN, and RANSAC to keypoint detection, keypoint matching, and homography matrix computation, respectively.

Reference Motion Module-based Image2Video Stage. As shown in Figure 6, the Image2Video stage captures temporal dependencies among video frames to mitigate the jitter effects in the Pose2Image stage. AnimateAnyone (Hu et al., ), as same as AnimateDiff (Guo et al., ), optimizes a residual self-attention-based motion module $\mathcal{M}$ within DenoisingNet $\mathcal{D}$ in this stage. The motion module reshapes a feature map $x\in\mathbb{R}^{b\times t\times h\times w\times c}$ to $x\in\mathbb{R}^{(b\times h\times w)\times t\times c}$ and then performs temporal attention along the $t$ dimension. However, as illustrated in Figure 7, this approach tends to produce averaged appearances across image sequences, resulting in diminished identity consistency. While this artifact may be negligible for other tasks where human identity is not important, it substantially degrades the realism of our gesture video graph.

**Table 1: Evaluation for co-speech video generation on Show-Oliver and YouTube Video dataset.**
|  | Show-Oliver | YouTube Talking Video |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | FVD $\downarrow$ | FGD $\downarrow$ | BC $\uparrow$ | Diversity $\uparrow$ | FVD $\downarrow$ | FGD $\downarrow$ | BC $\uparrow$ | Diversity $\uparrow$ |
| Ground Truth | - | - | 0.326 | 3.514 | - | - | 0.435 | 3.746 |
| SpeechDrivenTemplate (Qian et al., ) | 2.239 | 5.722 | 0.401 | 1.950 | 7.612 | 5.559 | 0.461 | 2.081 |
| ANGLE (Liu et al., ) | 2.079 | 5.112 | 0.359 | 2.577 | - | - | - | - |
| GVR (Zhou et al., ) | 1.615 | 4.246 | 0.270 | 4.623 | 4.027 | 2.900 | 0.331 | 3.573 |
| TANGO (Ours) | 1.379 | 3.714 | 0.375 | 5.393 | 3.133 | 2.068 | 0.479 | 4.128 |

Our analysis identifies that the key issue is the expansive solution space for the motion module, as self-attention is applied exclusively along the temporal dimension $t$. To address this, we introduce additional conditioning to constrain the solution space. We train the motion module by randomly selecting start and end pixels to effectively reduce uncertainty. During training, for the feature map $x\in\mathbb{R}^{(b\times h\times w)\times t\times c}$, we introduce a probability $p$ to incorporate 4-frame ground truth latent features as reference. Leveraging classifier-free guidance, our conditional diffusion-based motion module supports inference with or without these reference frames. The 4-frame segment alignment corresponds to the node length in the Gesture Video Reenactment. Finally, During inference, we introduce $4\times\alpha$ and $4\times\beta$ start and end conditional frames, generating intermediate 8 frames for transition edges.

## 4 Experiments

### 4.1 Dataset

Our experiments are conducted on the open-source Show dataset (Yi et al., ) and a newly collected YouTube Video dataset. The Show dataset comprises 26 hours of talking videos featuring four speakers with varying backgrounds and irregular camera movements. We selected the speaker, Oliver, as these videos contain fewer interactions with the background. The YouTube Video data is a small-scale, less than one hour dataset from in-the-wild YouTube videos characterized by clean backgrounds and fixed camera positions. More details of split for each dataset is in the APPENDIX.

### 4.2 Evaluation of Generated Videos

We compare our method with the previous state-of-the-art Generative method SDT (Qian et al., ), ANGLE (Liu et al., ) and reassemble-based method GVR (Zhou et al., ). We use the pertained weights from SDT and finetune the pose2img stage with a specific cloth. We evaluate ANGLE with the original paper test samples provided on Show-Oliver. For GVR, we reproduce the onset-based graph search and pose-aware neural rendering according to the implementation details in their paper.

**Table 2: User Study on Talkshow-Oliver.**
|  | SDT | GVR | TANGO (Ours) | GT |
| --- | --- | --- | --- | --- |
| Video Texture Quality $\uparrow$ | 4.1% | 26.7% | 33.8% | 35.5% |
| Audio-Motion Alignment $\uparrow$ | 29.2% | 10.9% | 28.7% | 31.2% |
| Overall Preference $\uparrow$ | 4.9% | 20.6% | 36.9% | 37.6% |

Objective Evaluation
We employ both video and kinematic Feature Distance (FVD (Carreira & Zisserman, ) and FGD (Yoon et al., )) to quantify feature-level discrepancies. Additionally, we utilize Beat Consistency (BC) (Liu et al., ) and Diversity (Li et al., ) metrics to evaluate audio-motion synchronization and gesture diversity, respectively. As shown in Table 1, our method outperforms GVR and SDT across all metrics except for BC. The fully generated baseline exhibits greater flexibility in the output motion space, which results in better BC performance. However, compared to SDT, our method significantly improves video quality. Furthermore, compared to GVR, our approach consistently shows improvement across all metrics, demonstrating that TANGO generates more realistic and audio-synchronized videos.

Subjective Evaluation
As shown in Table 2, we conducted a user study across four results. 47 Participants were asked to assess each video based on i) which video is more physically accurate, ii) which video’s content aligns more closely with the audio, and iii) overall, which video is more like a real video. Users compared videos from all four results in a single row, with the order of videos randomly shuffled. A total of 60 video clips, each spanning 6 sec. We didn’t include ANGLE in the user study due to not having enough result video clips. Some snapshots of a transition period are shown in Figure 8, and the supplementary material includes video results. Our method scored comparable to the ground truth video and outperforms the existing generative method SDT and retrieval method GVR with a clear margin.

### 4.3 Evaluation of AuMoCLIP

We evaluate the effectiveness of AuMoCLIP by retrieving gesture motion sequences using target audio features and measure performance using retrieval accuracy. As shown in Table 3, we compute low-level and high-level retrieval accuracy. Low-level retrieval accuracy is calculated by randomly selecting an audio feature at frame $i$ and finding the motion frame within $(i-16,i+16)$ with the highest cosine similarity. If this frame lies within $(i-4,i+4)$, it is marked as accurate. The final accuracy is averaged over 16K random pairs, so the random search will have an accuracy of 25%. Similarly, high-level retrieval accuracy is measured by selecting an audio high-level feature and comparing it against 256 motion candidates (1 paired motion + 255 negatives). If the highest cosine similarity corresponds to the paired motion, it is marked as accurate. This accuracy is averaged over 3K random pairs, and the random search will perform 0.391%. If the performance beats the random search, that means the model works correctly. The onset and keyword matching in GVR has a performance of 35.38% (low-level) and 1.288% (high-level), which is better than the random search. We then discuss the performance roadmap to the final AuMoCLIP.

Generative Features. One straightforward approach is to directly train an audio-to-motion network (Liu et al., ) and compute the joint level distance between the generated motion and motion candidates for retrieval. However, this method yields lower-than-expected performance, achieving only 29.03% on low-level retrieval and 1.403% on high-level retrieval.

**Table 3: Comparison of features for audio-motion retrieval**
|  | Low Level | High Level |
| --- | --- | --- |
| Random Search | 25.00% +00.00% | 0.391% +00.00% |
| Generative Features | 29.03% +16.10% | 1.403% + 258.8% |
| Keyword (Zhou et al., ) | - | 1.288% + 229.4% |
| Onset (Zhou et al., ) | 35.38% +41.51% | - |
| Baseline (Max Pooling) | - | 5.312% + 1360% |
| Baseline (CLS Token) | - | 11.84% + 3028% |
| + Wav2Vec2 | - | 12.73% + 3255% |
| + BERT | - | 15.68% + 4010% |
| + Wav2Vec2$\&$BERT | - | 16.40% + 4194% |
| + Split (Low + High) | 47.94% + 99.76% | 17.83% +4460% |
| + Split (Low only) | 65.57% + 162.2% | - |
| AuMoCLIP (+ Stop Grad.) | 65.68% + 163.8% | 19.54% + 4897% |

MaxPooling or CLS Token. We then switch to a MoCoV2-based dual-tower contrastive learning framework, starting with high-level features only. MoCoV2 is originally designed for images, not sequential data like motion. One solution is to apply max pooling along the time axis for the global token, similar to (). However, while max pooling focuses on accurate local alignment, it is less effective for global retrieval, resulting in a performance of only 5.312%. To address this, we adopt an adaptive global feature merge approach using the CLS token, which significantly improves performance to 11.84%. We keep CLS token in the remained experiments.

Pretrained Audio Features from Wav2Vec2 and BERT. As shown in Table 3, incorporating pretrained audio features, specifically time-aligned BERT features, significantly enhances the baseline’s performance from 11.84% to 15.68%. This improvement occurs because BERT captures high-dimensional language semantics rather than just ”audio textures.”, which is critical for co-speech gesture retrieval task.

Discussion of Low-Level Contrastive Learning. Including the low-level contrastive learning task consistently benefits high-level retrieval performance, suggesting that adding a more robust low-level feature improves high-level performance. Interestingly, we found that training only the low-level contrastive learning task achieves significantly better performance, reaching 65.57%. These observations suggest that we should: i) incorporate learned low-level features into high-level embedding learning, and ii) avoid the influence of high-level learning on low-level features. Therefore, we propose simply stopping the gradient to achieve the best performance for both features.

**Table 4: Comparison of video blending methods.**
|  | PSNR $\uparrow$ | LPIPS $\downarrow$ | MOVIE $\downarrow$ | FVD $\downarrow$ |
| --- | --- | --- | --- | --- |
| FiLM (Reda et al., ) | 35.43 | 0.072 | 74.85 | 1.358 |
| VFIF (Lu et al., ) | 34.91 | 0.077 | 79.42 | 1.777 |
| AnimateAnyone (Hu et al., ) | 32.63 | 0.127 | 86.06 | 1.421 |
| PANR (Zhou et al., ) | 35.18 | 0.071 | 75.02 | 1.190 |
| ACInterp (Ours) | 35.63 | 0.065 | 72.65 | 0.922 |

Figure: Figure 8: Comparison for Transition Frames Generation. From top to bottom, we show four snapshots in the same frames across different methods. Our method shows fewer artifacts in the hand regions and maintains appearance consistency with the GT frames. AnimateAnyone can recover the hands but loses appearance consistency. The flow-based methods FiLM and VFIFormer fail to estimate the flow for complex motions, resulting in the disappearance of hands. Pose Aware Neural Rendering shows better hand results but still suffers from artifacts such as blurring.
Refer to caption: x7.png

### 4.4 Evaluation of video blending methods

We compare the effectiveness of the proposed diffusion-based video frame interpolation method by evaluating the quality of blended videos. The evaluation is conducted on a separate test set from Show-Oliver, consisting of 368 video clips, each with 8 frames. Our approach is compared with the Pose Aware Neural Rendering in the original GVR (Zhou et al., ), the state-of-the-art flow-based blending method FiLM (Reda et al., ), VFIFormer (Lu et al., ), and the diffusion-based pose-guided video generation method AnimateAnyone  (Hu et al., ). All methods are re-trained on the Show-Oliver dataset with a training and inference resolution of $768\times 768$. We evaluate both image-level and video-level quality. For single images, we utilize Image Error (L1 pixel-level distance), Learned Perceptual Image Patch Similarity (LPIPS), and Peak Signal-to-Noise Ratio (PSNR). For videos, we adopt the Mean Opinion Video Quality Estimation (MOVIE) and feature-level distance (FVD). The features for calculating LPIPS and FVD are obtained from pretrained AlexNet and I3D networks, respectively.

The objective comparisons are shown in Table 4.
The flow-based interpolation method (Reda et al., ) tends to have lower average pixel-level error. The AnimateAnyone  (Hu et al., ) performs worse due to appearance inconsistency.
Overall, our diffusion-based interpolation model outperforms both previous flow-based and diffusion-based methods by a clear margin, e.g., FVD 0.922 vs. the previous best 1.190. See Figure 8 and supplementary material for the resulting video.

## 5 Conclusion

We presented TANGO, a framework for generating high-fidelity videos where body gestures align with the target speech audio. TANGO is evaluated on the Show-Oliver and YouTube Video datasets to demonstrated its ability to produce realistic videos, outperforming state-of-the-art generative and retrieval based methods. Besides, to the best of our knowledge, TANGO is the first work to present CLIP-Like contrastive learning on audio and motion modalities, and it is the first open-source motion graph and audio-driven video generation pipeline. In the future, we aim to extend the gesture video graph on general human motion videos, such as dance, sports and more.

## Appendix A Appendix

### A.1 Datasets

Show-Oliver.
The Show dataset comprises 26 hours of talking videos featuring four speakers with varying backgrounds and irregular camera movements. We selected the speaker, Oliver, as these videos contain fewer interactions with the background.
The full Show-Oliver dataset contains 6546 video clips, ranging from 3 to 10 seconds. We evaluate our approach using multiple few-shot sets, each derived from a total of 10 minutes of randomly selected video clips with consistent clothing. These sets are divided into 80%, 10%, and 10% splits for training, validation, and testing, respectively.

YouTube Video Dataset.
We further collect and process a small-scale, few-shot dataset from in-the-wild YouTube videos characterized by clean backgrounds and fixed camera positions. These videos feature 12 speakers delivering presentations lasting 1 to 2 minutes. We select four speakers of them and picks 6 to 10-second video subsets as the test set and use the remaining videos for training and constructing the Gesture Video Reenactment.

We first collect raw YouTube videos featuring 12 different identities. These videos are then segmented into multiple clips based on the detected sentence boundaries in the audio. Face detector is utilized to ensure that all clips contain clear faces. Subsequently, post-processing is employed to eliminate background obstructions and automatically adjust the camera position for consistency.
Finally, we obtained 304 clips for different identities, with from total duration of 1 hour data. For each identity, the longer video clips are used for the validation set, while the remaining clips constitute the training set.

### A.2 Timing-Aligned BERT Features without MFA

To achieve time-aligned BERT features for audio without forced alignment (MFA), we combine Wav2Vec2 and BERT models through the following steps:

Transcription (ASR). We use Wav2Vec2 with a Connectionist Temporal Classification (CTC) head to obtain the logits, which represent the model’s confidence scores for each possible token at each time step. The logits are processed to generate a sequence of predicted token IDs for the audio input. These token IDs are then decoded into a transcription using the Wav2Vec2 processor’s vocabulary. For example, we have the alphabet sequence $["","","T","","","h","e","","F","i","r","s","t"]$ in this step.

BERT Embedding. The transcription is tokenized using the BERT tokenizer, and the embeddings are obtained from the BERT model. The tokenizer in BERT conver the alphabet sequence into word sequence $["CLS","The","First","POS"]$.

Time Alignment. We align the Wav2Vec2-generated tokens with the BERT tokens using character-level matching. In particular, for each audio frame, we assign the aligned BERT embedding as its feature. If a match is not found, we fill the gap by using the nearest neighboring non-zero features, ensuring a smooth transition in the feature sequence over time.

### A.3 15D Motion Representation

We refer to NeMF (He et al., ) represent the motion at each time step $t$ using a 15-dimensional (15D) feature vector for each joint. This representation captures only local motion. The 15D motion representation, $\mathbf{X}_{t}\in\mathbb{R}^{J\times 15}$, includes:

Firstly, the joint positions $\mathbf{x}_{t}^{p}\in\mathbb{R}^{J\times 3}$ represent the 3D coordinates of each joint relative to the root joint, providing the skeletal pose at time $t$. Secondly, the joint velocities $\dot{\mathbf{x}}_{t}^{p}\in\mathbb{R}^{J\times 3}$ capture the rate of change of joint positions.

Additionally, the representation includes joint rotations $\mathbf{x}_{t}^{r}\in\mathbb{R}^{J\times 6}$, which encode the orientation of each joint in a 6D rotation format (Zhou et al., ), allowing for a more robust and unambiguous rotation representation. Lastly, the angular velocities $\dot{\mathbf{x}}_{t}^{r}\in\mathbb{R}^{J\times 3}$ describe the rotational speed of each joint.
This unified representation enables a detailed and comprehensive modeling of both the position and movement dynamics of each joint.

### A.4 Limitations

The ACInterp inputs 2D pose images from linear 2D pose blending. When the GT motion between 8-interpolated frame is non-linear, the generated results is slightly differ from the GT. We calculate the linear blending could work, i.e., with a 2D pose error smaller than threshold 0.005, on 83% clips for Talkshow-Oliver Dataset.

## Peer Reviews

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: See weaknesses above.
soundness: 4
strengths: 1. Contributions to Retrieval, Interpolation, and Datasets: The paper introduces TANGO, which enhances co-speech gesture video generation through improved retrieval methods, a diffusion-based interpolation model, and the introduction of the YouTube Business dataset.
2. Contribution to Open Source Gesture Generation: By making their code, pretrained models, and datasets publicly available, the authors significantly advance the field of gesture generation and encourage collaboration among researchers.
3. Comprehensive Experiments: The extensive evaluation of TANGO using various quantitative metrics and user studies demonstrates its superiority over existing state-of-the-art methods in generating realistic and audio-synchronized gesture videos.
summary: The paper introduces TANGO, a framework for generating high-fidelity co-speech gesture videos using a motion graph-based retrieval approach. It addresses audio-motion misalignment and visual artifacts in previous methods by implementing two key innovations: the AuMoCLIP, a hierarchical audio-motion joint embedding space for improved cross-modal alignment, and ACInterp, a diffusion-based interpolation network for generating high-quality transition frames. TANGO effectively synchronizes body gestures with speech audio, outperforming existing methods on datasets like Talkshow-Oliver and YouTube Business. The framework represents a significant advancement in gesture video generation, providing an open-source solution that integrates these novel components.
weaknesses: 1. Clarification on Graph Pruning Methodology: The paper does not provide sufficient detail on how the strongly connected component (SCC) subgraphs are merged, which is a critical operation in the Graph Pruning section.
2. Inconsistencies in Visual Representation: In Figure 3, the use of blue and green to represent different video motion clips lacks rigorous color correspondence during transitions, potentially leading to misunderstandings; similarly, Figure 5 uses yellow for both the Wav2Vec2 transformer feature and input audio, which could cause confusion.
5. Typographical Errors: There is a typographical error in Figure 4 where "merge" should be corrected to "merging."
4. Testing Results: The generated gesture movements exhibit stuttering, which undermines the fluidity and realism of the videos.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 2
questions: 1. The appendix mentions, “We evaluate our approach using multiple few-shot sets.” Are each of the proposed models (AuMoCLIP, ACInterp) trained separately on only one of these few-shot sets?

2. The author states, “we leverage the power of the two-stage (pose2image and image2video) video generation diffusion model, AnimateAnyone.” However, there are no details on what "leverage" means in this context. Are they merely using the model’s structural concept, or are they initializing with a pre-trained version of AnimateAnyone?

3. Why does Section 4.4 use a different test set for evaluation?

4. Why are there missing results in Table 3?

5. In Figure 5, why is there a white segment within the low-level and high-level feature representations? According to the paper, it should include only the Wav2vec CNN feature and the Wav2Vec transformer feature with the BERT feature.
soundness: 3
strengths: 1. Compared to existing works, the video results of the proposed method are strong in terms of both visual quality and audio-gesture alignment.
2.  The authors conduct thorough experiments to validate the effectiveness of the proposed method.
3. In terms of novelty, I would categorize this work as moderate. Although using contrastive learning for audio-motion retrieval is not new, the author deserves credit for applying it within the speech gesture retrieval domain. Beyond this, other contributions seem more incremental, such as adding graph pruning to GVR and incorporating an additional reference motion module and homography-based background flow to the animation process.
summary: The authors present a co-speech gesture generation approach through a gesture video reenactment framework. To align video segments with audio, the author proposes an audio-motion joint embedding space using contrastive learning.

Furthermore, an existing diffusion-based, pose-driven video generation model is employed to produce interpolated frames. By incorporating an additional reference motion module and a homography-based background flow, the method achieves improved appearance consistency.

Both qualitative and quantitative comparisons with existing state-of-the-art methods demonstrate that the proposed method outperforms alternatives.
weaknesses: 1. Some claims are not sufficiently justified. For example:

    The author claims, “BERT captures high-dimensional language semantics rather than just ‘audio textures,’ which is critical for the co-speech gesture retrieval task.” However, there is no analysis showing that the addition of BERT features actually generates language-semantics-aware gestures. It’s possible that the performance improvement is due to alignment information alone. If the author could conduct an experiment comparing the use of BERT features with using only CTC alignment information, this might demonstrate the necessity of BERT. Additionally, if the author could provide video examples showing that the generated gestures become more language-semantics-aware with the addition of BERT, it would further support the claim.

2. Since the authors compare their work to other co-speech gesture video generation methods, it would be beneficial to include recent state-of-the-art works, such as *Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model (S2GMM).*”If the authors could add this to their evaluation, it would make the proposed work's claim to be state-of-the-art more convincing.

3. Both the objective and subjective evaluations use only one character, Oliver, making it difficult to verify whether the proposed approach would outperform other methods in different circumstances.The authors could perform a comparison experiment with an additional character to demonstrate the generalizability of their approach. Furthermore, if the proposed method does not perform well on characters commonly used by other state-of-the-art methods (e.g., TALKSHOW, PATS), it would be essential to discuss these potential limitations.

4. Writing and formatting improvements are needed for easier comprehension. For example:

- 4.1 Abbreviations:
    
    - GVR stands for Gesture Video Reenactment in the abstract, but GVR is later used to refer to the specific work (Zhou et al., 2022).
        
    - The abbreviation for (Liu et al., 2022c) should be ANGIE, not ANGLE.
        
- 4.2 Tables:
    
    - In Table 2, methods are listed as columns, while in Table 1, they are listed as rows.
        
    - There are no captions to explain some non-trivial numbers in the tables, which could be helpful for interpretation.
        

5. Some explanations and implementation details are unclear or missing. Please refer to the question section for further clarification.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: Please refer to Weaknesses.
soundness: 4
strengths: This paper proposes a framework for generating co-speech body-gesture videos, dubbed TANGO. TANGO integrates a specifically designed Gesture Video Reenactment (GVR) module and a hierarchical joint embedding space known as AuMoCLIP to facilitate high-quality video generation. The strengths of this work are summarized as follows:
1. The authors present a novel and robust technical contribution through the introduction of a graph network baseline that adaptively incorporates the GVR and AuMoCLIP modules.
2. The motivation for this research is clearly articulated, and the insights provided are compelling. It is important to highlight that a unified framework for co-speech gesture modeling from audio to video sequences holds significant relevance in this community.
3. The dataset introduced in this paper is designed to facilitate ongoing research in gesture generation, providing valuable resources for future studies.
4. The visualization results are well-executed, allowing reviewers to thoroughly evaluate the findings and methodologies presented.
summary: This paper proposes a framework for generating co-speech body-gesture videos, dubbed TANGO. TANGO integrates a specifically designed Gesture Video Reenactment (GVR) module, which facilitates the realistic portrayal of gestures, alongside a hierarchical joint embedding space known as AuMoCLIP, which enhances the quality of video generation. The authors conducted extensive experiments that demonstrate the method's superiority over existing approaches in terms of well-rounded metrics and visualization results.
weaknesses: However, there are still some questions for me. I encourage the authors to give reasonable responses. The final rating would be raised if the solution can solve my concerns.
1. Some important related works might be missed, and I encourage the authors to discuss with them. For example, both of these two methods leverage the audio signal as input to generate human postures.
 
[1] Qi, X., Pan, J., Li, P., Yuan, R., Chi, X., Li, M., ... & Guo, Y. (2024). Weakly-Supervised Emotion Transition Learning for Diverse 3D Co-speech Gesture Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10424-10434).

[2] Mughal, M. H., Dabral, R., Habibie, I., Donatelli, L., Habermann, M., & Theobalt, C. (2024). ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1388-1398).

2. In the experimental setting, it is very interesting to explore the frame interval with different settings rather than 4. Please note that this is not an ask for additional experiments in the rebuttal, and this does not impact the paper score. I encourage authors to conduct this discussion in the future.

3. In the user study, did the authors assess participant engagement, specifically regarding whether responses were submitted too quickly or too slowly, and whether any such responses should be discounted? For example, whether the authors considered using response time thresholds or including control questions to check for consistent responses. I encourage authors to add these discussions in the appendix.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: 1. Representing the gesture as the motion graph: While it is good to represent the gesture patterns from the video as the graph, the retrived features are discrete tokens not uniformly sampled. Will this reduce the quality of gesture generation as the speech audio in natural should be continuous?

2. It seems like for high-level representation, the author only used one transformer layer upon the low-level features from CNN encoders. Is it possible the limited number of transformer level leads to a low performance for the global retrieval? I have this doubt because the author draw inspiration from Wav2Vec 2 for the model design. However, Wav2Vec ustilized 11 transformer layers upon the CNN encoder and the self-supervised learning is also based on the transformer layer feature. Is it possible adding additional layers of transformers can help improve the high-level retrieval?

3. For one technical detail of the paper, why random search of low-level retrieval has a 25% accuracy? In addition, can the author include some other metrics like F1 score, recall? There should be mostly negative samples for the retrieval if my understanding is correct.

4. One further question based on weakness 3, why the temporal contrastive learning is used for the low-level feature instead of applied to high-level features from transformer layers?

5. Refer to Weakness 5, I am assuming the the camera during inference will be kept the same. One claim the author mentioned is "homography background offset flow to eliminate artifacts due to camera parameter changes". Does this only apply for the training but not inference for stablizing training?
soundness: 4
strengths: For the method design:

1. This work presents a retrieval framework for gesture generation. It resolved a significant issue by the former work GVR (retrieval naively relies on audio onset features and keyword matching) 

2. The author represent the gesture patterns within the video as graph and present an efficient pruning strategy to achieve long-sequence representation. This strategy seems effective.

3. The author propose the AuMoCLIP for aligning the two modalities and demonstrate an effective merging of text semantics, high-level audio, and low-level audio features. The ablation study for low-level and high-level retrieval is insightful with very detailed analysis and design. The author analyzed how the temporal alignment of two modalities can be achieved and also the high level semantics.

4. While uplifting the geature pattern generation to the pixel-level video synthesis, the author did not simply reply on commonly used ReferenceNet-like modules but in addition propose homography background offset flow for the consideration of camera changes for realistic videos. The author did very deep analysis of temporal dependencies and resolved the background and jittering problems introduced by camera motion commonly seen in video generations.


For potential insight and benefits for Future studies:
1. Auto-Evaluation: The AuMoCLIP seems to be a very good generalizable evaluation method for automatic evaluation of gesture motion naturalness conditioned on speech. Unlike FGD, diversity or Beat Alignment (BAS), which cannot precisely include the naturalness of motion patterns, the semantics or beat alignment of audio and speech, this model might be very beneficial for future auto-evaluations, though the author did not propose this point.

2. Audio and Gesture Motion Modality alignment: Previous works in this domain have very limited analysis of the how the speech signal functions as the trigger for gesture patterns. Many previous works, (like DiffSheg, CaMN, TalkShow, EMAGE, AMUSE, etc) only designed the generator by incorporating the audio features or text as the control signals to drive the gesture generation but never analyzed how the inner relationships of different modality should be. GestureDiffuCLIP did preliminary studies in aligning the speech word context and gesture motion patterns but ignored the temporal alignment between modalities. In this work, the author did very detailed analysis of the low-level retrieval and high-level retrieval, reconsider the temporal alignment and semantic alignment (low-level indicates the temporal alignment, I consider the local temporal contrastive loss might functions for the beat-dynamic of the gestures, high-level indicates the global semantic alignment of the two modalities. The ablation of the retrieval is very insightful)
summary: The paper presents TANGO, a novel framework for generating co-speech body-gesture videos that effectively synchronizes gestures with speech audio. The key contributions of TANGO include:

Motion Graph-Based Retrieval: TANGO enhanced the motion graph-based model with more fine-grained designed AuMoCLIP to obtain relevant body gestures that align with the audio input, enhancing the realism and relevance of the generated gestures.

Diffusion-Based Interpolation Network: The framework utilizes a diffusion-based approach for video frame interpolation, which improves the quality and fluidity of the generated videos, addressing limitations found in previous methods.

The authors conduct extensive evaluations, including both objective metrics and user studies, demonstrating that TANGO outperforms existing methods like Gesture Video Reenactment (GVR) and Speech-Driven Templates (SDT) in terms of video quality, audio-motion alignment, and overall user preference.
weaknesses: 1. The dataset for comparison are limited. The author only selected Oliver in the Show dataset and a small scale in-the-wild YouTube videos. (However, as far as I know, this filed lacks of high-quality data, PATS is another option, but it contains too many low-quality/low-resolution videos and blurry frames for large shoulder and hand motions)

2. It seems like the temporal contrastive learning is based on the low-level features obtained from the CNN encoder. I am not quite sure if I understand it correctly. It will be better if the author can include this information in the final draft for easy understanding.

3. For the method design, in case I am uncertain of some technical details, I will present some potential weakness in the Question section instead of here.

4. For the gesture motion representation, the author utilized 3D SMPL/SMPL-X template model parameters. I assume the author based on the first frame of the given image, do the tracking to obtain the camera and based on the camera to project the generated 3D joints onto the 2D-space for image-level pose guided generation. I think the author could add this information to the final draft.

5. Some of the training details of the model is missing, like what are the GPUs for training, time for training, learning rate.

Meta-review: The paper introduces a framework for generating high-fidelity co-speech gesture videos using a motion graph-based retrieval approach. It addresses audio-motion misalignment and visual artifacts by introducing two key innovations: a hierarchical audio-motion joint embedding space for improved cross-modal alignment, and a diffusion-based interpolation network for generating high-quality transition frames. All the reviewers gave very positive feedback on this paper. AC agrees with the recommendation of the reviewers and decided to accept this work.

Decision: Accept (Oral)