Co$^{\mathbf{3}}$Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusion
Authors: Xingqun Qi
Yatian Wang
Hengyuan Zhang
Jiahao Pan
Wei Xue
Shanghang Zhang
Wenhan Luo
Qifeng Liu
Yike Guo
Venue: ICLR 2025 Spotlight (2025)
Keywords: 3D co-speech gesture generation
human motion modeling
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We introduce the new task of concurrent co-speech gesture generation cooperating with one newly collected large-scale dataset named GES-Inter.

Abstract: Generating gestures from human speech has gained tremendous progress in animating virtual avatars. While the existing methods enable synthesizing gestures cooperated by people self-talking, they overlook the practicality of concurrent gesture modeling with two-person interactive conversations. Moreover, the lack of high-quality datasets with concurrent co-speech gestures also limits handling this issue. To fulfill this goal, we first construct a large-scale concurrent co-speech gesture dataset that contains more than 7M frames for diverse two-person interactive posture sequences, dubbed $\textbf{GES-Inter}$. Moreover, we propose Co$^{\mathbf{3}}$Gesture, a novel framework that enables concurrent coherent co-speech gesture synthesis including two-person interactive movements. Our framework is built upon two cooperative generation branches conditioned on decomposed speaker audio. Specifically, to enhance the coordination of human postures w.r.t corresponding speaker audios while interacting with the conversational partner, we present a Temporal-Interaction Module ($\textbf{TIM}$). TIM can effectively model the temporal association representation between two speakers' gesture sequences as interaction guidance and fuse it into the concurrent gesture generation. Then, we devise a mutual attention mechanism to further boost learning dependencies of interacted concurrent motions, thereby enabling us to generate vivid and coherent gestures. Extensive experiments demonstrate that our method outperforms the state-of-the-art models on our newly collected GES-Inter dataset.

Published as a conference paper at ICLR 2025
GESTURE::: TOWARDS COHERENT CONCURRENT CO-SPEECH 3D GESTURE GENERATION WITH INTER- ACTIVE DIFFUSION
Xingqun Qi Yatian Wang"; Hengyuan Zhang", Jiahao Pan"
Wei Xue", Shanghang Zhang", Wenhan Luo', Qifeng Liu Yike Guo
1 The Hong Kong University of Science and Technology
2 Peking University
SPEAKER tis the left
Gesture
SPEAKER rapatt No, you diaitt soe ouninn..
Figure 1: Diverse exemplary clips sampled by method from our newly collected GES-Inter dataset. The vital frames are visualized to demonstrate the concurrent upper body dynamics two speakers generated by Gesture framework displaying temporal coherent interaction with each other, respectively. Best view on screen.
ABSTRACT
Generating gestures from human speech has gained tremendous progress in ani- mating virtual avatars. While the existing methods enable synthesizing gestures cooperated by individual self-talking, they overlook the practicality of concurrent gesture modeling with two-person interactive conversations. Moreover, the lack of high-quality datasets with concurrent co-speech gestures also limits handling this issue. To fulfill this goal, we first construct large-scale concurrent co-speech gesture dataset that contains more than 7M frames for diverse two-person interac- tive posture sequences, dubbed GFt--Intr. Additionally, we propose Co Gesture, a novel framework that enables coherent concurrent co-speech gesture synthesis including two-person interactive movements. Considering the asymmetric body dynamics of two speakers, our framework is built upon two cooperative generation branches conditioned on separated speaker audio. Specifically, enhance the coordination human postures w.r.tcorrreponding speaker audios while interact- ing with the conversational partner, we present Temporal Interaction Module (TIM). TIM can effectively model the temporal association representation between two speakers' gesture sequences interaction guidance and fuse it into the con- current gesture generation. Then, we devise mutual attention mechanism to further holistically boost learning dependencies of interacted concurrent motions, thereby enabling us to generate vivid and coherent gestures. Extensive experiments demonstrate that our method outperforms the state-of-the-a models on our newly collected GES-Inter dataset. The dataset and source code are publicly available at https://mmatie- www.ggitttbbcccCcc io/Co3/
'These authors contributed equally to this work. x Corresponding authors.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
The generation of co-speech gestures seeks to create expressive and diverse human postures that align with audio input. These nen-verbal behaviors play crucial role in human communication, signifi- cantly enhancing the effectiveness of speech delivery. Meanwhile, modeling co-speech gestures has broad applications in embodied AI, including human-machine interaction (Liu et al.) (023), robotic assistance (Farouk] 2022), and virtual/augmenntdd reality (AR/VR) (Fu et al. 2022). Traditionally, researchers have primarlyy ly concentrated on synthesizing upper body gestures that correspond to spoken audio (Liu et al. 2022b; Yi et al. 2023
These methods usually focus on synthesizing single-speaker gestures following people's self- talking (Liu et al. 2024b a Qi et al. 2024b] Yang et al., 2023). Although some researchers model the single human postures via conversational speech corpus (Mughal et al. 2024a Ng al. 2024), they mostly overlook generating the concurrent long sequence gestures with interactions. Besides, others generate the single speaker gesture from conversational corpus incorporated with interlocuter reaction movements (Kucherenko et al. 2023) Zhao et al. (023). However, few researchers have devoted themselves constructing datasets with interactive concurrent gestures. For example, the interactive movement of two speakers may include waving arms when saying "hello" during conversation. In this work, we therefore introduce the new task of two-speaker concurrent gestures generation under the condition of conversational human speeches, as displayed in Figure
There are two main challenges in this task: 1) Datasets of concurrent 3D co-speech gestures syn- chronized with conversation audios of two speakers are scarce. Creating such a dataset containing large-scale 3D human postures is difficult due to complex motion capture systems and expensive labor for actors. 2) Modeling the plausible and temporal coherent co-speech gestures of two speakers is difficult, especially involving the frequent interactions in long sequences.
To address the issue of data scarcity, we construct new large-scale whole-body meshed 3D co- speech gesture dataset that includes concurrent speaker postures within more than 7M frames, dubbed GES-Inter. In particular, we first leverage the advanced 3D pose estimator Zhhag al. 2023a to obtain high-quality poses (i.e., SMPL-
X Pavlakos al. (2019) and FLAME Li et al.) (201)) from in-the-wild talk show videos. To obtain the individual sound signals of each speaker in the conversation while preserving the identity consistency with the posture movement, we employ the pyannote- audio Bredin et al.
Mixed Audio Separated Andio
Yeah, all shodied
light,, welen came.......
Sevaraion
(2020) to separate the mixed speech, as shown in Figure 2 Afterward, by utilizing the automatic speech recognition techniques Whisper-X X [ain al. (202)), acquire the consistent text tran- script and speech phoneme with speaker audio. In this fashion, our GES-Inter dataset covers wide range of two-person interactive concurrent co-speech gestures, from daily conversations to formal interviews. Moreover, the multi-modality annotation and common meshed human postures pave the potential for various downstream tasks like
funny!
Figure 2: Illustration of our audio separation and alignment with speakers.
human behaviors analysis (Liang et al. 2024b; Xu et al., 2024) and talking face generation (Peng al. 2024 Ng al. 2024), etc.
Based on our GES-Inter dataset, we propose novel framework, named Co GoGture to model the coherent concurrent co-speech 3D gesture generation. The key insight of our framework is to carefully build the interactions between concurrent gestures. Here, we observe that the motions of two speakers are asymmetric (e.g., when one speaker moves talking, the other could be quiet static or moving slowly). Directly producing the concurrent gestures in a holistic manner may lead to unnatural results. Therefore, we establish two cooperative transformer- mansforeer-basedd diffusion branches to generate corresponding gestures of two speakers. performing the specific denoising process, respectively. This bilateral paradigm encourages our framework to yield diverse interactive movements while effectively preventing mode collapse.
2

---

Published as a conference paper at ICLR 2025
Moreover, ensure the motions of the one speaker temporally consistent with the corresponding audio signal and display coherent interaction with the conversational partner, we devise a Temporal Interaction Module (TIM). Specifically, first incorporate the separated human voices to produce single-speaker gesture features, respectively. Then, we model the joint embedding of the current speaker features and the integrated conversational motion clues guided by mixed speech audio. Here, the learned joint embedding is leveraged as the soft weight to balance the interaction dependence of the generated current speaker gesture dynamics with other ones. Then, we conduct the mutual attention of the fused bilateral gesture denoisers to further facilitate high-fidelity concurrent gesture generation with desirable interactive properties. Extensive experiments conducted on our newly collected Gnt-Inter dataset verify the effectiveness of our method, displaying diverse and vivid concurrent co-speech gestures.
Overall, our contributions are summarized as follows:
We introduce the new task of concurrent co-speech gesture generation cooperating with one newly collected large-scale dataset named GES- Inter. It contains more than 7M high-quality co-speech postures of the whole body, significantly facilitating research on diverse gesture generation.
We propose novel framework named Co Gesture upon the bilateral cooperative diffusion branches to produce realistic concurrent co-speech gestures. Our Co Gesture includes the tailor-designed Temporal Interaction Module (TIM) ensure the temporal synchronization of gestures w.r.t.he corresponding speaker voices while preserving desirable interactive dynamics.
Extensive experiments show that our framework outperforms various state-of-the-art counterparts on the GES-Inter dataset, producing diverse and coherent concurrent co-speech gestures given conversational speech audios.
RELATED WORK
Co-speech Gesture Synthesis. Synthesizing the diverse and impressive co-speech gestures displays significant role in the wide range of applications like human-machine interaction (Cho et al. 2023
Guo et al.] (021), robot (De Wit et al.] 2023; Sahoo et al., 2023). and embodied AT (Li et al. 2023
Benson et al.] 2022). Numerous works were proposed address this task that can be roughly divided into rule-based approaches, machine learning designed methods, and deep learning based ones. Rule-based research depends on linguistic experts' pre-defined corpus bridge human speech and gesture movements (Cassell et al.) 1994 Poggi et al. 2005). The others usually leverage machine learning techniques with mutually constructed speech features to generate co-speech gestures (Levine) et al. 2010; (Sargin et al. (208). However, these methods heavily rely efforts on pre- pre-processing which may cause expensive labor consumption.
Recently deep learning based methods gained much development directly modeling co-speech gesture synthesis via deep neural networks. Most of them usually leverage the multi-modality cues to generate
postures incorporated with individual self-talking audio (Li et al.
202
a Zhu et al. 2023 Yi et al.
2023
Qi et al. 2024c), such as speaker identity (Liu et al. 2024a bj Chen et al., 2024), emotion (Qi
et al. 2024b Liu et al. 2022a), and transcripts (Zhang et al. 2024| Ao et al. 2023 2022 Zhi
et al. speech guidance (Ng al.,
2023). Only a few counterparts propose to synthesize the single gesture under conversational
2024 Mughal et al., (024b)). Besides, the GENEA challenge holds the most similar settings to us. The participants aim to generate the single-person gesture from
the conversational corpus incorporated with interlocuter reaction movements (Kucherenko al.) 2023). However, they overlook the concurrent co-speech gesture modeling of two speakers during the conversation is much more practical in the real scenes. Few of the above methods could be directly adapted to this new thought.
Co-speech Gesture Datasets. Co-speech gesture datasets are roughly divided into two types: pseudo-label based gestures and motion-capture based ones. For pseudo-label approaches, researchers usually utilize the pre-trained pose estimator to obtain upper body postures from in-the-wild News or
talk show videos (Yoon et al. 2020 Ahuja et al. 2020 Habibie et al. 2021). Thanks to the recent advanced parametric whole- body meshed 3D model SMPL-X Pavlakos al. (2019) and FLAME Li
et al. (2017). some high-quality shole-body based 3D co-speech gesture datasets are emerged (Qi
2024c Yi al. 2023 Qi et al. 2024h a). Meanwhile, it significantly promotes the construction
3

---

Published as a conference paper at ICLR 2025
Table 1: Statistical comparison of our GES-Inter with existing datasets. The dotted line separates
whether the speech content in the dataset is built based on the conversational corpus.
Concurrent Gestures
Duration (hours)
Additional Attributee
Joint Annotation
Datasets
Facial
Mesh
Phonme
Text
TED Yoon et al. 2020 mOG
X
106.1
x
x
X
pseudo
TED-EX (Lau ct al. 2022h
CVPR
X
100.8
X
x
X
pseudo
EGGS Ghorhani of al 2023
X
2
X
X
X
pseudo
BEAT [iu et al. 2022a
X
76
/
X
V
V
mo-cap
SHOW Yl al. 2023
X
26.9
/
J
X
X
pseudo
TWAI6.2 [Lee el al. 2019
1OCV
17
T
mo-cap
BEAT2 (Liu
2024a
VPR
60
/
mo-cap
DND Mugghll
2024b
OVPR
X
x
mo-cap
Photoreal [Ng al. 2024
OVPR
mo-cap
GES-Inter (ours)
70
V
pseudo
2022a 2024a Ghorbani et al. 2023 Mughal et al. 2024b; Ng et al. 2024). Although some of them are built upon conversational corpora, they only provide gestures single speakers (Liu et al. 2024a Ng al. 2024). The TWH16.2 (Lee et 2019) dataset displays the pioneer exploration of concurrent gestures via joint-based representation. However, overlooks the significance of the facial expression data in conversation. Meanwhile, the SMPL-X mesh-based whole-body data in our dataset is more convenient for avatar rendering and downstream applications (e.g., talking face) compared to TWH16.2. Besides, the DND GROUP GESTURE dataset Mughal et al. (2024b) is built upon multi-performer group talking scene, which can not be directly applied to our task. Therefore, a 3D co-speech dataset including concurrent gestures of two speakers with the meshed whole body is required for further research.
of motion-capture based co-speech datasets (Liu et al.
3D Human Motion Modeling, Human motion modeling aims to generate natural and realistic coherent posture sequences from multi-modality conditions, which contains co-speech gesture synthesis as a sub-task (Liang al. 2024a] Tevet et al. 2023). One of the hottest tasks is generating human movements from the input action descriptions (Jiang et al. 2023 Zhang et al. 2023b Lin accurate semantic expression aligned with text prompts. The other one that shares modality guidance similar to our task is the Al choreographer (Li et al. 2020 (202b]] Siyao al., 2022 Le et al., (023). While retaining analogous interactive human motion modeling with the approaches mentioned above, our newly introduced work differs from them significantly. Both of the aforementioned topics follow the symmetrical fact that exchanging the identities of performers during interactions does not change the semantics coherence of motions. We take the asymmetric body dynamics of concurrent human movements into consideration, motivating us to design the bilateral diffusion branches.
et al. 2024 Xu et al., 2024). It needs to enforee the results by displaying
PROPOSED METHOD
3.1 INTERACTIVE GESTURE DATASET CONSTRUCTION
Preliminary. Due to the expensive labor and complex motion capture system establishment during the frequent interactive conversations, similar to Yi et al. (2023); Liu et al. (20226); Qi et al. (2024c), we intend to obtain the high-quality 3D pscudo human postures of our dataset. Synthesizing datasets conducive to our task focuses on ensuring high-fidelity and smooth gesture movements, authority speaker voice separation, and identity consistent
alignment.
Estimation of 3D Posture. Firstly, we exploit the state-of-the-art 3D pose estimator Pymaf- X ZZhag et al.) (2023a) to acquire the meshed whole-body parametric human postures based on SMPL-X MMMLaXXPavlakos Pavlakos et al. (2019). In particular, the body dynamics are denoted by the unified SMPL model Loper et al. (203)) that collaborated with the MANO hand model Boukhayma et al. (019)). Meanwhile, we adopt the FLAME face model Li et al. (2017) to present the facial expressions of speakers. The corpora are collected from the in-the-wild talk show or formal interview videos that contain high-resolution frames and unobstructed sitting postures. Then, conduct extensive
4

---

Published as a conference paper at ICLR 2025
PELL
Tumporal Interaction Module (TD3)
X
of
X Natual Alhaniia
Iruppea Inloaation Soslale
Naasal
Dot Product
Additio
Figure 3: The overall pipeline of our Co Gesture. Given conversational speech audios, our framework generates concurrent co-speech gestures with coherent interactions.
data processing filter the unnatural and jittery poses, thereby ensuring the high-quality of the datase("] Our GnS-Inter includes more than 7M validated gesture frames with 70 hours. To the best of our knowledge, this is the first large-scale co-speech dataset that includes mesh-based whole-body concurrent postures of two speakers, as reported in Table
Separation of Speaker Audio. To obtain the identity-specific speech audios of each speaker in the mixed conversation corpus, we leverage the advanced sound source separation technique pyannote-audii Bredin et al. (2000) to conduct human voice separation. Here, we enforce the number of separated speakers
for assigning each speech segment to the corresponding speaker. Then, we utilize the superior speech recognition model WhisperX [Bin et al. (203)) to acquire accurate word-level text transcripts. Once acquire high-fidelity transcripts, we utilize the Montreal Forced Aligner (MFA) McAuliffe et al. (2017) to obtain phoneme-leve timestamps associated with facial expressions. Such extensive multi-modality attributes of our dataset enable the research of various downstream tasks like talking face (Peng et al., 2024: Ng et al. 2024). and human behavior
analysis (Park al. 2023
Qi al. 2023
Dubois al. 2024) etc.
Alignment of Audio-Posture Pair. Once obtain the separated audio signals for each speaker, we assign them to the corresponding body dynamics. We recruit professional inspectors to manually annotate the separated audio signals with their corresponding speaker identities. To ensure the accuracy of the aligned audio-posture pairs, different inspectors double-check these results. In this dataset offers high-quality concurrent gestures with aligned
way, our newly constructed multi-modal annotations.
3.2 PROBLEM FORMULATION
Given a sequential collection of conversational audio signal Cmiz as the condition, our goal is to generate the interactive concurrent gesture sequences of two speakers x. Where Cmiz C, + C and x (Xa, )b) denote the corresponding audios and postures of two speakers. The sequence length is the fixed number Specifically, each pose single person presented as J joints with 3D representation. Note that only generate the upper body including fingers in this work.
3.3 BILATERAL COOPERATIVE DIFFUSION MODEL
Considering the asymmetric body dynamics of two speakers, we aim to address the concurrent co- speech gesture generation a bilateral cooperation manner, as depicted in Figure The framework takes the two noisy human motions input for producing denoised ones, which is conditioned on diffusion time step f, mixed conversational audio signal Cmüzz and separated speaker voices C. or Ct. We leverage separated human speech guidance for bilateral branches to generate
Please refer supplementary material for more details about data processing.
5

---

Published as a conference paper at ICLR 2025
corresponding gestures. Moreover, we utilize the original mixed audio signal of two speakers to indicate the interaction information to ensure the synthesized posture retains rhythm with specific audio while preserving interactive coherency with the conversation partner. All the audio signals are fed into the audio encoder for feature extraction.
Temporal Interaction Module. To ensure temporal consistency while preserving the interactive dynamics of concurrent gestures, we propose a Temporal Interaction Module (TIM) to model the temporal association representation between each current speaker movements and conversational counterparts, As shown in Figure 3] we utilize the features extracted from mixed conversational audios to indicate the interaction information. Here, the dimensions of features in our TIM are all normalized as I xD, For notation simplicity, we take single-branc x,, for elaboration.
In particular, we first incorporate the current speaker audio embedding fc. as the query Q match the key feature K' and value feature V belonging to motion embedding fx.. via cross-attention
meshanism (Vaswani et al. 2017):
Q fc. W, &. W, fx. W.
Here, W denotes the projection matrix. Along with this operation, we obtain the updated current speaker motion embedding C. Similarly, we acquire the interactive motion embedding fx_ Cmis incorporated with mixed conversational speeches. Then we calculate the temporal correlation matrix M E R X between the updated current gesture embedding and interactive embedding, Here, the temporal correlation matrix represents the temporal variants between the current gesture sequences and interactive ones. Then, we exploit motion encoder to acquire a learnable weight parameter o as the temporal-intedd interaation dependency. Once we obtain the weight parameter, the current speaker motion embedding is boosted as follows:
C. + (1
fx,, C..i : = sigmoid(Enc(Mi)
where @ is Hadamard product, Enc denotes the motion encoder. The motion embedding of the conversation partner updated the same manner. In this fashion, the temporal interaction fidelity of generated gestures is well-preserved.
Mutual Attention Mechanism. To further enhance the interaction between two speakers, we construct bilateral cooperative branches that interact with each other to produce concurrent gestures. To be specific, we introduce the mutual attention layers that take the features of the counterpart the query Q in Multi-Hear Attention (MHA). respectively. We observe that exchanging the input order of the speaker's 's audio results in invariance effect of interactive body dynamics. In other words, the distribution of interaction data two speakers adheres to the same marginal distribution. Therefore, formulate the cooperating denoisers retaining shared weight update strategies. This encourages the gesture features after the TIM to be more temporal and interactive with partner ones, holistically.
3.4 OBJECTIVE FUNCTIONS
During the training phase, the denoisers of bilateral branches share the common network structure. Given the diffusion time step f, the current speaker audio {Ca, Ct}, the mixed conversation audio
Cmizi and the noised gestures { Xxa
{xx")
(t) xxb J. the denoisers are enforced to produce continuous human
gestures. The denoising process can be constrained by the simple objective:
Ceimple Exx,,, [| DDxx() C.. Cmizi |1? 1/x0 D(x,)) (1) Cb, Cmiiz t) |)]]]
(3)
where D is the denoiser, NNO. I) the added random Gaussian noise, (t)
(a.)) ((a,)) + (()) is
the gradually noise adding process at step t. (e) E (0, 1) is the constant Apee-parameter Moreover, we utilize the velocity loss Luel and foot contact loss ooot (Tevet et al. 2023) provide supervision on the smoothness and physical reasonableness, respectively. Finally, the overall objective is:
Liotal Asimple Csimpte + Luel + Lioat,
where Asimpte is trade-off weight coefficients.
In the inference, since the audio signals of the concurrent gestures generation as essential condition modality, the prediction of human postures formulated as fully conditioned denoising. This encourages framework strike a balance between high fidelity and diversity.

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
4.1 DATASETS AND EXPERIMENTAL SETTING
GES-Inter Dataset. Since the existing co-speech gesture datasets fail to provide interactive concur- rent body dynamics, we contribute new dataset named GES-Inter to evaluate our approach. The human postures of our GES-Inter are collected from 1,462 processed videos including talk shows and formal interviews. The extraction takes 8 NVIDIA RTX 4090 GPUs in one month, obtaining 20 million raw frames. After the complex data processing, we get more than 7 million validated instances. Finally, we acquire 27.390 motion clips that are split into training/ validation/ testing following criteria (Liu et al.
2022a 2024a) as 85% 7.5% 7.5%.
Implementation Details. We set the total generated sequence length N 90 with the FPS normalized as 15 in the experiments. Cmiz, Ca, and C₁ are represented as audio signal waves, initially. Then, these audio signals are converted into seeesppetss with an FFT window size of 1,024, and the hop length is 512. The dimension of input audio mel-spectrograms is 128 x 186. We follow the tradition of (Liu et al,, 2022b Qi et al. 2024bb c) to leverage the speech recognizer as the audio encoder. Each branch of our pipeline is implemented with 8 blocks within 8 heads of attention layers. The latent dimension D is set to 768.
In the training stage, we set Asimpte 15, empirically. The initial learning rate is set as 1 x 10-4 with an Adam W optimizer. Similar to Nichol & Dhariwal (202I), we set the diffusion time step as single NVIDIA H800 GPU with batch size of 128. The training takes a total of 100 epochs, accounting for 3 days. During inference, we adopt DDIM Song et al.) (220)) sampling strategy with 50 denoising timesteps to produce gestures. Our experiments only contain upper body joints without facial expressions and shape parameters. Our Co Gesture synthesizes upper body movements containing 46 joints (i.e. 16 body joints + 30 hand joints) of each speaker. Each joint is converted to a 6D rotation representation Zhou et al. for more stable modeling. The dimension of the generated motion sequence Ro 276, where 90 denotes frame number and 276 = 46 x 6 means upper body joints. The order of each joint follows the original convention of SMPL-X.
1,000 with the cosine noisy schedule. Our model is applied
Evaluation Metrics. To fully evaluate the realism and diversity of the generated co-speech gestures, we introduce various metrics:
FGD: Fréchet Gesture Distance (FGD) Yoon et al. (020)) is calculated as the distribution distance between the body movements of synthesized results and real ones via a pre-trained autoencoder.
BC: Beat Consistent Score (BC) / Beat Alignment Score (BA) Liu et al. (2022a 2024a) measures whether the generated motion dynamics are rhythmic consistent with the input speech audios. We report the average score of two speakers in our experiments.
Diversity: Similar to (Liu et al. 2022b Zhu et al. 2023: Liu et al.
2024a). the autoencoder of
FGD exploited to acquire feature embeddings the synthesized gestures. Here, the diversity score means the average distance of 500 randomly assembed pairs.
4.2 QUANTITATIVE RESULTS
Comparisons with SOTA Methods. To the best of our knowledge, we are the first to explore the coherent concurrent co-speech gesture generation with conversational human audio. To fully verify the superiority of our method, we implement various state-of-the-art (SOTA) counterparts from the perspective of dingle-person-bace
co-speech gesture generation (i.e., TalkSHOW (Yi et al. 2023), ProbTalk (Liu et al. 2024b). DiffSHEG (Chen et al.) 2024), EMAGE (Liu et al. (024a)) and text2motion generation (L.e, MDM (Tevet et al.] (023), InterX (Xu al. 2024) InterGin (Liang et al. (024b)). For fair comparisons, all the competitors are implemented by official source codes or pre-trained models released by authors. Specifically, in DiASHEE, we follow the convention of the original work to utilize the pre-trained HuBERT (Hsu et al. 22021) for audio feature extraction. In Tal,SHOW, we exploit the pre-trained Wav2vec (Baevski et al. (020) to encode the audio signals following the original setting. Apart from this, the remaining components for gesture generation in DiffSHEG and Sal are all trained from scratch on the newly collected GES-Inter dataset. For other methods, we modify their final output layer match the dimensions of our experimental
7

---

Published as a conference paper at ICLR 2025
Table 2: Comparison with the state-of-the-art counterparts on our newly collected GES-Inter dataset.
1 means the higher the better, and J indicates the lower the better. = means 95% confidence interval. The dotted line separates whether the methods are adopted from single-person co-speech generation or tex12motion counterparts.
DEt-Inter Dataset
Methods
FGD
BC
Diversity t
TalkSHOw (Yi al. 2023 eVPR
2.256
0.613
53.037=1.021
ProbTalk LLu et al. 2024b
CVPR
1.238
0.645
46,981 2:2173
DifSSHEG Chen et al.
CVRR
1.209
0.638
56.781 11.900
EMAGE Liu al 2024a
CVPR
1.884
0.637
60.917
MDM Tevet et al
2023
TOLR
1.696
0,654
65.529 +-2.218
InterX Xu ct
2024
CVPR
1.178
0.661
65.161
InterGen (Ltang et al.
2024b IJOV
1.012
0.670
69.455 11.00
Co Gesture (ours)
0.769
0.692
72.824 4:2.026
settings. Since the above tex12motion counterparts are designed without the audio incorporation setting, we adopt the same audio encoder as ours in the models.
As reported in Table 2 we adopt the FGD, BC, and diversity for well-rounded view of comparison. Our GeéGesture outperforms all the competitors by a large margin on the GES- Inter dataset. Remark- ably, our method even achieves more than 24% (i.e., (1.102 0. 0.769) /10//11111212 24%) improvement over the sub-optimal counterpart in FGD. We observe that both InterGen (Liang et al. (024b) and ours synthesize the authority gestures with much higher diversity than others. This caused by both of them employing the bilateral branches to generate concurrent gestures of two speakers. However, InterGen shows lower performance on FGD due to the lack of effective temporal interaction modeling. In terms of BC, method attains much better results than other counterparts. This aligns highly with our insight on the audio separation-basee bilateral diffusion backbone that encourages each branch to synthesize speech coherent gestures while preserving the vivid interaction of the results. Compared with single-person co-speech gesture based ones, our model still achieves the best performance. This can be attributed to our well-designed temporal interaction module.
Ablation Study. To further evaluate the effectiveness of our Gesture, we conduct extensive ablation studies of different components and experiment settings as variations.
Effects of the TIM and Mutual Attention: To verify the effectiveness of TIM and mutual attention mechanism, we conduct detailed experiments as reported in Table 3 The exclusion of the temporal interaction module (TIM) and the mutual attention mechanism lead to performance degradation in our full-version framework, respectively. Moreover, we conduct ablation by simply replacing the TIM with an MLP layer for feature fusion, the FGD and BC display the obvious worse impact as shown in the Table. The results verify that our TIM effectively enhances interactive coherency between two speakers. In particular, our temporal interaction module effectively models the tem- poral dependency between the gesture motions of the current speaker and the conversation partner. Therefore, implementation without it leads our framework to fail in produc- ing cooperative motions, thus signifi- cantly reducing the performance in all the metrics. Moreover, the exclusion of the mutual attention mechanism re- sults in FGD is obviously worse than the full version framework. This indi- cates that our mutual attention can ef- fectively handle complex interactions from the perspective of holistic fash- ion while balancing the specific move- ments of two speakers.
Table 3: Ablation study of TIM and mutual attention mecha- nism on our GES-Inter dataset.
RES-Inter Dataset
Methods
FGD J
BC
Diversity 1
w/ MLP
1.202
0.663
64.690=1.135 1.17
w/o TIM
1.297
0.676
67.953#1.280
w/o Mutual Attention
0.924
0.681
69.084 1.412
Co Gesture (full version)
0.769
0.692
72.824
#2.026
8

---

Published as a conference paper at ICLR 2025
Effects of the Bilateral Branches and Audio Mixed/ Separation: We also conduct the ab- lation study in different experiment settings. To demonstrate the effectiveness our bilateral construct the single branch-based diffusion pipeline that generates the gestures of two speakers in a holistic manner. As shown in Table [ by subtracting the bilat- eral branches from the full version pipeline, the indicator FGD displays much worse results (e.g., 0.769 - 1.669). This outcome verifies that our cooperative bilateral diffusion branches effectively handle the asymmetric motion of concurrent gestures of two speakers. This sup- ports our key technical insight on framework construction. Then, by subtracting the original mixed audio, the indicators FGD and BC present much worse performance. These results ver- ify the mixed audio signal displays effectively enhance the interaction between two speakers. Meanwhile, we further verify the ca- pability of our audio separation design where the model only takes the mixed conversational speech signal input. Based on the above-mente mone-meet fash- ion, the BC metric clearly attains much worse result than the separated one. Directly modeling mixed conver- sational speech to produce concurrent gestures impacts the interactive cor- relation between two speakers. This seriously affects the harmony of syn-
diffusion branches,
Table 4: Ablation study of bilateral branches and audio mixed/ separation on our GES-Inter dataset.
GES-Inter Dataset
Methods
FGD 4
BC
Diversity 1
w/o Bilateral Branches
1.669
0.640
64.542
w/o Mixed Audio w/o Audio Separation
1.227
0.656
64.899 1899*91.004
1.180
0.633
66. 65.15911180
Co Gestue (full version)
0.769
0.692
72.824 #2.026
thesized concurrent gestures and corresponding speech rhythm.
Effects of the Foot Contact Loss: Inspired by (Tevet et al. 2023 Liang et al., 2024b), we introduce foot contact loss to ensure the physical reasonableness of the generated gestures. Since we only model the upper body joints in experiments, we complete the lower body joints as T pose in forward kinematic function during calculate loss. We conduct the ablation study to verify the effectiveness of foot As illustrated in Table 5) the exclusion of the foot contact loss results in FGD and BC are obviously worse than the full version framework. This indicates that our foot contact loss displays a positive impact on the generated postures.
Table 5: Ablation study of foot contact loss on our GES-Inter dataset.
GES-Inter Dataset
Methods
FGD +
BC
Diversity t
w/o Foot Contact fot
1.082
0.675
68.448 68.448-1.08 11.082
Co Gesture (full version)
0.769
0.692
72.824 12.002
4.3 QUALITATIVE EVALUATION
Visualization. To fully demonstrate the superior performance of our method, we display the visualized keyframes generated by our Co²Gesture framework with other counterparts, as illustrated in Figure For better demonstration, the relative position coordinates of the two speakers are fixed in visualization. The lower body including the legs is fixed (e.g., seated) while visualizing due to the weak correlation with human speech. For example, it is quite challenging to model whether the two speakers are sitting or standing from only audio inputs. We showcase the two optimal methods from single-person gesture generation and text2motion, respectively. Our method shows coherent and interactive body movements against other ones. To be specific, observe that ProbTalk and DiffSHEG would synthesize the stiff results (e.g., blue rectangles of right speakers). Although the X generates the natural movements of the left speaker, it displays less interactive dynamics of the right speaker. In addition, the results synthesized by InterGien show reasonable interaction between two speakers. However, it may produce unnatural postures sometimes (as depicted in red circles). In contrast, our Co'Gesture can generate interaction coherent concurrent co-speech gestures. This highly aligns with our insight into the bilateral cooperative diffusion pipeline. For more visualization demo videos please refer to our anonymous webpage: https::/mmttie-ee. e.githuuiii//o./ io/Co3/ In our experiments, the length of the generated gesture sequence 90 frames with 15 FPS. Thus, all the demo videos in the user study have the same length of 6 seconds.
User Study. To further analyze the quality of concurrent gestures synthesized by ours against various competitors, we conduct user study by recruiting 15 volunteers. All the volunteers

---

Published as a conference paper at ICLR 2025
just told you that these things should be put Yeah, know know but...
ProbTalk
DifISHEG
InterX
InterGen
Co=Gesture (Ours)
Figure 4: Visualization of our generated concurrent 3D co-speech gestures against various state-of- the-art methods. The samples are from our newly collected GES-Inter dataset.
anonymously selected from various majors in school. For each method, we randomly se- lect two generated videos in the user study. Hence, each participant needs to watch 16 videos required to evaluate the generated results by all the
for seconds of each. The subjects counterparts in terms of naturalness, smoothness, and interaction coherency. The visualized videos randomly selected and ensure that each method has least samples. The sta- tistical results are shown in Figure 5 with the rating scale from 0-5 (the higher, the better). Our framework demonstrates the best perfor- mance compared with all the competitors. To be specific, our method achieves noticeable advan-
Naturalness
Smoothness
Interaction Coherency
45
tages from the perspective of smoothness and interaction coherency. This indicates the effec-
tiveness of our proposed bilateral denoising and temporal interaction module.
5 CONCLUSION
In this paper, we introduce a new task of co- herent concurrent co-speech gesture generation given conversational human speech. We first newly collected large-scale dataset contain- ing more than 7M concurrent co-speech gesture instances of two speakers, dubbed GES-
TolkSHOW
Interen
Figure 5: User study on gesture naturalness, mo- tion smoothness, and interaction coherency.
This high-quality dataset supports our task while significantly facilitating the research on 3D human motion modeling. Moreover, we propose novel framework named Co*Gesture that includes temporal inntraatiitt module to ensure the generated gestures preserve interactive coherence. Exten- sive experiments conducted on our GES Inter dataset show the superiority of our framework.
Limitation. Despite the huge effort we put into data preprocessing, the automatic pose extraction stream may influence dataset with some bad instances. Meanwhile, our framework only generates the upper body movements without expressive facial components. In the future, we will incorporate framework with tailor-designed facial expression modeling and investigate more stable data collection techniques to further improve the quality of our dataset. Besides, we will put more effort into designing specific interaction metrics for better concurrent gesture evaluation.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
The research was supported by Early Career Scheme (ECS- HKUST2220132)), Theme-based Research Scheme (T45- 205/21-N) from Hong Kong RGC, NSFC (No. 62206234), and Generative AI Research and Development Centre from InnoHK.
REFERENCES
Chaitanya Ahuja, Dong Won Lee, Yukiko I Nakano, and Louis-Philippe Morency. Style transfer for co-speech gesture animation: A multi-speakeer conditional-mixture approach. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16, pp. 248-265. Springer, 2020.
Tenglong Ao, Qingzhe Gao, Yuke Lou, Baoquan Chen, and Libin Liu. Rhythmic gesticulator: Rhythm-awand co-speech gesture synthesis with hierarchical neural embeddings. ACM Transac- lions on Graphics (TOG), 16): 2022.
Tenglong Ao, Zeyi Zhang, and Libin Liu. Gesturediffuclip: Gesture diffusion model with clip latents. ACM Transactions on Graphics (TOG), 42(4):1-18, 2023.
Alexci Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing
systems, 33:12449. 12449-124660, 2020.
Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. Whisperx: Time-accurate speech transcription of fong-form audio. INTERSPEECH 2023, 2023.
William Benson, Zachary Anderson, Evan Capelle, Maya Dunlap, Blake Dorris, Jenna Gorlewicz, Mitsuru Shimizu, and Jerry B Weinberg. Embodied expressive gestures in telerobots: tale two users. ACM Transactions on Human-Robee Interaction, 12(2):1-20, 2023.
Adnane Boukhayma, Rodrigo de Bem, and Philip HS Torr. 3d hand shape and pose from images in the wild. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 10843-10852, 2019.
Hervé Bredin, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, and Marie- Phillppe Gill. Pyannote. audio: neural building blocks for speaker diarization. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7124-712 IEEE, 2020.
Justine Cassell, Catherine Pelachaud, Norman Badler, Mark Steedman, Brett Achorn, Tripp Becket, Brett Douville, Scott Prevost, and Matthew Stone. Animated conversation: rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agents. In Proceedings of the 2 annual conference Computer graphics and interactive techniques, pp. 413 1994.
Junming Chen, Yunfei Liu, Jianan Wang, Ailing Zeng, Yu Li, and Qifeng Chen. Diffsheg: A diffusion-based approach for real-time speech-driven holistic 3d expression and gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Haein Cho, Inho Lee, Jingon Jang. Jae- Jae-Hyun Kim, Hanbee Lee, Sungjun Park, and Gunuk Wang. Real-time finger motion recognition using skin-conformable electronics. Nature Electronics, 6(8): 619-629, 2023.
Jan De Wit, Paul Vogt, and Emiel Krahmer. The design and observed effects of robot-performed manual gestures: A systematic review. ACM Transactions on Himan--Robe Interaction, 12(1): 1-62, 2023.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
11

---

Published as a conference paper at ICLR 2025
Maged Farouk. Studying human robot interaction and its characteristics. International Journal of Computations, Information and Manufacturing (ICIM), 2(1), 2022.
Yu Fu, Yan Hu, and Veronica Sundstedt. A systematic literature review of virtual, augmented, and mixed reality game applications in healthcare. ACM Transactions on Computing for Healthcare (HEALTH), 3(2):1-27. 2022.
Saeed Ghorbani, Ylva Ferstl, Daniel Holden, Nikolaus Troje, and Mare-André Carbonneau. Zeroeggs: Zero-shot example-based gesture generation from speech. In Computer Graphics Forum, volume 42, pp. 206-216. Wiley Online Library, 2023.
Lin Guo, Zongxing Lu, and Ligang Yao. Human- interaction sensing technology based on hand gesture recognition: A review. IEEE Transactions on Hunan-Maching Systems, 51(4): 300-309, 2021.
Ikhsanul Habibic, Weipeng Xu, Dushyant Mehta, Lingjie Liu, Hans-Peter Seidel, Gerard Pons-Moll. Mohamed Elgharib. and Christian Theobalt. Learning speech-driven 3d conversational gestures from video. In Proceedings of the 2/ ACM International Conference on Intelligent Virtual Agents, pp. 108, 2021.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:3451-3460, 2021.
Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language. Advances in Neural Information Processing Systems, 36:20067-2000 20079, 2023.
Taras Kucherenko, Rajmund Nagy, Youngwoo Yoon, Jieyeon Woo, Teodor Nikolov, Mihail Tsakov, and Gustav Eje Henter. The genea challenge 2023: A large-scale evaluation of gesture generation models in monadic and dyadic settings. Proceedings of the 25th International Conference on Multimodal Interaction, pp. 792-801, 2023.
Nhat Le, Tuong Do, Khoa Do, Hien Nguyen, Erman Tjiputra, Quang D Tran, and Anh Nguyen. Controllable group choreography using contrastive diffusion. ACM Transactions Graphics (TOG), 42(6):1-14, 2023.
Gilwoo Lee, Zhiwei Deng, Shugao Ma, Takaaki Shiratori, Siddhartha S Srinivasa, and Yaser Sheikh. Talking with hands 16.2 m: A large-scale dataset of synchronized body-finger motion and audio for conversational motion analysis and synthesis. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 763-772, 2019.
Sergey Levine, Philipp Krähenbühl, Sebastian Thrun, and Vladlen Koltun. Gesture controllers. In Acm siggraph 2010 papers, pp. 1-11. 2010.
Jiaman Li, Yihang Yin, Hang Chu, Yi Zhou, Tingwu Wang, Sanja Fidler, and Hao Li. Learning to generate diverse dance motions with transformer. arXiv preprint 2xXiv:2008.08/7/ 2020.
Jing Li, Di Kang, Wenjie Pei, Xuefei Zhe, Ying Zhang, Zhenyu He, and Linchao Bao. Audio2gestures: Generating diverse gestures from speech audio with conditional variational autoencoders. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV) pp. 11293- 11302, October 2021a.
Ruilong Li, Shan Yang. David Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 13401-13412, 2021b.
Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d scans. ACM Trans. Graph., (6666))194-11, 2017.
Yang Li, Xiaoxue Chen, Hao Zhao, Jiangtao Gong, Guyue Zhou, Federico Rossano, and Yixin Zhu. Understanding embodied reference with touch-line transformer. In ICL.R, 2023.
12

---

Published as a conference paper at ICLR 2025
Han Liang, Jiacheng Bao, Ruichi Zhang, Sihan Ren, Yuecheng Xu, Sibei Yang, Xin Chen, Jingyi Yu, and Lan Xu. Omg: Towards open- vocabulary motion generation via mixture of controllers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
482-493, 2024a.
Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu, Intergen: Diffusion-based multi- human motion generation under complex interactions. International Journal of Computer Vision, pp. 1-21, 2024b.
Jing Lin, Ailing Zeng. Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion-x: A large-scale 3d expressive whole-body human motion dataset. Advances in Neural Information Processing Systems, 36, 2024.
Chen Liu, Peike Patrick Li, Xingqun Qi. Hu Zhang. Lincheng Li, Dadong Wang, and Xin Yu. Audio- visual segmentation by exploring modal mutual semantics. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 7590-7598, 2023.
Haiyang Liu, Zihao Zhu, Naoya Iwamoto, Yichen Peng. Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng. Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis. In European Conference on Computer Vision, pp. 612-630. Springer, 2022a.
Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Xuefei Zhe, Naoya Iwamoto, Bo Zheng, and Michael J. Black. Emage: Towards unified holistic so-speech gesture generation via expressive masked audio gesture modeling. Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pp. 1144-1154, June 2024a.
Xian Liu, Qianyi Wu, Hang Zhou, Yinghao Xu, Rui Qian, Xinyi Lin, Xiaowei Zhou, Wayne Wu, Bo Dai, and Bolei Zhou. Learning hierarchical cross-modal association for co-speech gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10462-10472, 2022b.
Yifei Liu, Qiong Cao, Yandong Wen, Huaiguang Jiang, and Changxing Ding. Towards variable and coordinated holistic co-speech motion generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024b.
Matthew Loper. Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. Smpl: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2. pp. 551-866. 2023.
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech, volume 2017, pp. 498-502, 2017.
Muhammad Hamza Mughal, Rishabh Dabral, Ikhsanul Habibie, Lucia Donatelli, Mare Habermann, and Christian Theobalt. Convofusion: Multi-modal conversational diffusion for co-speech ges- ture synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1388-1398
June 2024a.
Muhammad Hamza Mughal, Rishabh Dabral, Ikhsanul Habibie, Lucia Donatelli, Mare Habermann, and Christian Theobalt. Convofusion: Multi-modal conversational diffusion for co-speech ges- ture synthesis. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1388-1399 2024b.
Evonne Ng, Javier Romero, Timur Bagautdinov, Shaojie Bai, Trevor Darrell, Angjoo Kanazawa, and Alexander Richard. From audio to photoreal embodiment: Synthesizing humans in conversations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1001-1010, June 2024.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusson probabilistic models. In International conference on machine learning, pp. 8162-8171. PMLR, 2021.
Joon Sung Park, Joseph O' Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pp. 1-22, 2023.
13

---

Published as a conference paper at ICLR 2025
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10975-10985, 2019.
Ziqiao Peng, Wentao Hu, Yue Shi, Xiangyu Zhu, Xiaomei Zhang, Hao Zhao, Jun He, Hongyan Liu, and Zhaoxin Fan. Synctalk: The devil is in the synchronization for talking head synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 666-676, 2024.
Isabella Poggi, Catherine Pelachaud, Fiorella de Rosis, Valeria Carofiglio, and Berardina De Carolis. Greta. a believable embodied conversational agent. Multimodal intelligent information presentation, pp. 3-25, 2005.
Xingqun Qi, Chen Liu, Muyi Sun, Lincheng Li, Changjie Fan, and Xin Yu. Diverse 3d hand gesture prediction from body dynamics by bilateral hand disentanglement. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4616-4626 2023.
Xingqun Qi, Chen Liu, Lincheng Li, Jie Hou, Haoran Xin, and Xin Yu. Emotiongesture: Audio- driven diverse emotional co-speech 3d gesture generation. IEEE Transactions on Multimedia, 2024a.
Xingqun Qi, Jiahao Pan, Peng Li, Ruibin Yuan, Xiaowei Chi, Mengfei Li, Wenhan Luo, Wei Xue, Shanghang Zhang, Qifeng Liu, and Yike Guo. Weakly-superrised emotion transition learning for diverse 3d co-speech gesture generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10424-10434, June 2024b.
Xingqun Qi, Hengyuan Zhang, Yatian Wang. Jiahao Pan, Chen Liu, Peng Li, Xiaowei Chi, Mengfei Li, Qixun Zhang, Wei Xue, et al. Cocogesture: Toward coherent co-speech 3d gesture generation in the wild. arXiv preprint urXiv:2405. 16874, 2024c.
Jaya Prakash Sahoo, Suraj Prakash Sahoo, Samit Ari, and Sarat Kumar Patra. Hand gesture recogni- tion using densely connected deep residual network and channel attention module for mobile robot control. IEEE Transactions on Instrumentation and Measurement, 72:1-11. 2023.
Mchmet Sargin, Yucel Yemez, Engin Erzin, and Ahmet M Tekalp. Analysis of head gesture and prosody patterns for prosody-driven head-gesture animation. IEEE Transactions on Pattern Analysis and Machine Intelligecee ((38):13301133 2008
Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang. Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando: 3d dance generation by actor-critic gpt with choreographic memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11050-11059, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020.
Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In The Eleventh International Conference on Learning Represen- tations, 2023. URL http:tttt://opereeeeeee
net/ forum SJd
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Liang Xu, Xintao Lv, Yichao Yan, Xin Jin, Shuwen Wu, Congsheng Xu, Yifan Liu, Yizhou Zhou, Fengyun Rao, Xingdong Sheng, et al. Inter-x: Towards versatile human- humann interaction analysis. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 11260-22271, 2024.
Sicheng Yang, Zhiyong Wu, Minglei Li, Zhensong Zhang, Lei Hao, Weihong Bao, Ming Cheng, and Long Xiao. Diffusestyleggsturre: stylized audio-driven co-speech gesture generation with diffusion models. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pp. 1860-5868,.05 2023.
14

---

Published as a conference paper at ICLR 2025
Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yandong Wen, Timo Bolkart, Dacheng Tao, and Michael J Black. Generating holistic 3d human motion from speech. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 469-480, 2023.
Youngwoo Yoon, Bok Cha, Joo-Haeng Lee, Minsu Jang, Jaeyeon Lee, Jachong Kim, and Geehyuk Lee. Speech gesture generation from the trimodal context of text, audio, and speaker identity. ACM Transactions on Graphies (TOG), 39(6):1-16, 2020.
Ailing Zeng, Lei Yang, Xuan Ju, Jiefeng Li, Jianyi Wang, and Qiang Xu. Smoothnet: A plug-and-pla network for refining human poses in videos. In European Conference Computer Vision, pp. 625-642. Springer, 2022.
Hongwen Zhang, Yating Tian, Yuxiang Zhang, Mengcheng Li, Liang An, Zhenan Sun, and Yebin Liu. Pymaf-x: Towards well-aligned full-body model regression from monocular images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023a.
Jianoong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
14730-14740, 2023b.
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208. 15001, 2022.
Zeyi Zhang, Tenglong Ao. Yuyao Zhang, Qingzhe Gao, Chuan Lin, Baoquan Chen, and Libin Liu. Semantic gesticulator: Semantics-aware co-speech gesture synthesis. ACM Transactions on Graphics (TOG), 43(4):1-17, 2024.
Weiyu Zhao, Liangxiao Hu, and Shengping Zhang. Diffugesture: Generating human gesture from two-person dialogue with diffusion models. In Companion Publication of the 25th International Conference Multimodal Interaction, pp. 179-185, 2023.
Yihao Zhi, Xiaodong Cun, Xuelin Chen, Xi Shen, Wen Guo, Shaoli Huang, and Shenghua Gao. Livelyspeaker: Towards semantic-aware co-speech gesture generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20807-20817. 2023.
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 5745-5753.
Lingting Zhu, Xian Liu, Xuanyu Liu, Rui Qian, Ziwei Liu, and Lequan Yu. Taming diffusion models for audio-driven op-seechh gesture generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10544-10553, 2023.
15

---

Published as a conference paper at ICLR 2025
Automatic Processing
Manual Processing
Step Conversatioall Vilco
Step Audio Extraction
Stop 7. Pise Exiination
Stop 20, Vilco Filtering
Thans
X
ucacsil voles
Step Spaach Recegattion
Stop E.l. Pose Ftttering
Step & Pose Filering
Teeet milk anter
serieee bellinggoo
X Srway
rody tie atear
X Kanunuri
6.55, reat 2lll.
Apperprancty stord
Seep 3. tue
Speaken
Eelabisty aodtoable
entar velleel change
Step 12, Audiokdppshee Aligument
Test
Spiide
Strp i. sadin Flbering
Step Smanchhing
neitt lin shock."
Veat
STEAAE oo"
Audio Extraction and Piltering
Moxad Alamect
Basic Video Processing
and Filtering
Figure 6: The overall workflow of our dataset construction. The videos are processed to obtain high-quality postures through advanced automatic technologies and professional expert proofreading.
A
APPENDIX
To showcase the superior quality of our GES-Inter dataset and the effectiveness of our proposed CoeGestuee we provide additional details on data collection and further visualization results below.
A.1 DATASET CONSTRUCTION
In this section, give detailed explanation of the data processing pipeline of our GES-Inter dataset. We summarize the acquisition, processing, and filtering of our GES-Inter dataset into two main procedures: automatic and manual processing steps, illustrated in Figure
A.1.1 AUTOMATIC PROCESSING STEPS
To build high-quality 3D co-speech gesture dataset with concurrent and interactive body dynamics, we collect a considerable number of videos. They are then processed using automated methods to extract both audio and motion information.
Basic Video Processing (Step 1, 2, 3): First, with related searching keywords, we collect more than 14K conversational videos along with their metadata (e.g., video length, frame resolution, audio sampling rate, etc. ). Those keywords include talk show, conversation, interview, etc. this step, we acquire raw video data totaling up to 1,095 hours. However, many of these videos do not meet requirements regarding category, quality, language, and other factors. Therefore, we filter them in step 2 to make sure: the speakers in the videos are real people rather than cartoon characters; ii) the videos meet an acceptable quality standard, featuring a resolution of at least 480p for clear visuals; and iii) only English conversations are included. In these preprocessing phases, due the large amount of raw video collected and labor consumption, we conduct initial filtering using automatic techniques without manually checking each video. Specifically, we leverage YOLOv8 for human detection, discarding clips that do not show realistic people (eg, cartoon characters). Meta information provided by downloaded videos directly filters the English conversational corpus. Following the initial filtering, proceed to step 3, where we use PySceneDetect cut the videos into short clips.
Audio Extraction and Filtering (Step 4, 5, 6): Audios and poses are two necessary attributes for our GES-Inter dataset. In Step 4, we extract audio from the video clips using FFmppe. In step 5, we initially employed the pyannote-auu audio technique for speaker diarization, configuring it for two speakers accommodate two-person dialogues. The pyannote-audin tool assigns each speech segment to the appropriate speaker. Next, we utilize WhisperX Bain et al. (2023) for speech-to- to-text transcription. After transcription, we cluster the speakers based the generated timestamps to better
16

---

Published as a conference paper at ICLR 2025
organize the dialogue. With the extracted audio and speaker diarization, filter out clips in Step 6 that either have relatively low audio quality or no word detected. This filtering process improves the efficiency of the subsequent pose estimation.
Pose Estimation and Filtering (Step 7, 8, 9): As an acknowledged 3D human representation standard, SMPL--X Pavlakos et al. (2019) is used to represent whole-body poses in various related tasks Jiang et al. (2023); Zhang et al. (2022); Liu et al. (2024b)ai a): Chen et al. (2024); Liang et al. (2024b). Accordingly, we employ the outting-edge pose estimator PyMAF-X Zhang al. (2023a)
to extract high-quality 3D postures including body poses, subtle fingers, shapes, and expressions of the speakers. We then apply five criteria filter the clips based on the pose annotation: containing only two people, no speaker looking back, no missing joint of the upper body, appropriately-sizer speakers, and relatively noticeable gesture without change of the speaker positions. However, upon examining the visualized motions, we still observe that some temporal jittering within the movements inevitable. To this end, we exploit SmoothNet motion refinement in Step 9, The jittery effects are mostly caused by the blurring of speakers moving quickly in consecutive video frames. Due to the strict keyword selection in raw video crawling, our dataset rarely contains two speakers standing walking around. If there several clips including the aforementioned postures, we will filter them out to ensure our dataset maintains unified posture representation.
al. (2022) for temporal smoothing and jittery
In particular, our manual review indicates that SmoothNet effectively generates cleaner and more reliable motion sequences while maintaining diverse range of postures. However, due to the frequent extreme variations in camera angles, speaker poses, and lighting in talk show videos, some inaccuracies pose estimations from PyMAF-X are unavoidable. Thus, inspired by Pavlakos et al. (2019), we translate the joint of arm postures as Euler angles with x, and order. Then, if the wrist poses exceed 150 degrees on any axis, if the pose changes by more than 25 degrees between adjacent frames (at 15 fps), we discard these abnormal postures over a span of 90 frames.
A.1.2 MANUAL PROCESSING STEPS
Building on the initial postures and audio obtained automatically, we introduce manual processing in this section to further refine the annotation.
We observe that several undesired clips have passed through the initial filtering, including non-conversational scenarios and dual-person shots with external voices. To ensure that all videos meet our standards, we recruit two groups of inspectors to meticulously review and eliminate any that do not comply with the specified criteria. The results of each group are sampled and inspected to guarantee authority.
Basic Video Filtering (Step 10):
We conduct manual review of the processed clips at a consistent ratio of 5:1, selecting one clip from each group of five while adhering to the order of scenecut. This approach is valid, as there may exist overlap among adjacent clips. All clips are organized into five groups, and each group is assigned to an inspector for thorough evaluation. The inspectors assess the visualizations using SMPL-X parameters determine whether the motion appears smooth, jittery, or unnatural. If any sequences are identified as jittery or unnatural. we discard the entire group of five clips from which the sample was taken.
Pose Filtering (Step 11):
Additionally, we eliminate instances where the speakers experience significant occlusion of their bodies during the interaction. This meticulous evaluation process greatly enhances the quality of our dataset.
We obtained speech separation results with an accuracy of 95%. Here we defined correct instances as those audio clips with accurate speech segmentation, correct text recognition, and accurate alignment. During our audio preprocessing, the audio is initially segmented by pyannote-audio technique to achieve 92% accuracy. Then, the accuracy of text recognized by WhisperX is 96%.
Audio&Speakker Alignment (Step 12):
Once we obtain the separated audios, to ensure the identity consistency between the separated audio and body dynamics, we conduct audio-speake alignment in this step. To be specific. professional human inspectors
recruited manually execute this operation. Inspectors first check every video clip with its diarization to ensure the sentence-level speaker identities are correct and consistent within the clip. Then, inspectors assign the specific spatial position, i.e., left or right, to the speaker identities
17

---

Interaction Coherency Published as a conference paper at ICLR 2025
Table 6: Statistical results in User Study. + denotes standard deviation.
Comparison Methods
Naturalness
Smoothness
TalkSHOw
2
2.4+0.6
|:011
ProbTalk
2.540.3
2.2
2 2
DiftSHEG
3.5+0.5
1.8
2.520.6
EMAGE
4
2.8+0.4
2.3+0.5
MDM
3,5
4
3.5
InterX
3,8=0.4
420.5
4
InterGen
420.5
4.2
440.2
Ours
4,4
4.5
4.2
in the diarization. Using the alignment and the revised diarization with timestamps, inspectors separate each extracted audio into two distinct files and name them according to the corresponding speakers. To ensure the high fidelity of the alignment, the initially aligned audio-speaker pairs are double-checked by another group of inspectors. Meanwhile, the human inspectors would further check the rationality of segmentation and text recognition results from the perspective of human perception. In this step, we set two groups of inspectors for cross- poss-validation to ensure the final alignment rate is 98%. In this manner, our GES-I Inter dataset contains high-quality human postures with corresponding separated authority human speeches and multi-modality annotations. We provide examples of audio separation for better demonstration (refer to our webpage: https::/mattiie e.
A.2 MORE DETAILS ABOUT EXPERIMENTAL SETTING
Due to the complex and variable positions of the two speakers of in-the-wild videos, we set the relative positions of the two speakers to fixed values. In the experiments, we only model the upper body dynamics of the two speakers. In particular, the joint order follows the convention of SMPL-X. During experiments, we follow the convention of (Liu al. 2022a 2024a) to resample FPS as 15. In our dataset, we retain all the metadata (e.g.. video frames, poses, facial expressions) within the original FPS (i.e, 30) of talk show videos. We will release our full version data and pre-processing code, thereby researchers can obtain various FPS data according to their tasks.
A.3 MORE DETAILS ABOUT USER STUDY
In the user study, all participating students are asked to evaluate each video without any indication of which model generated it. For fair comparison in user study, the demo videos are randomly selected. We count the motion fractions length of two speakers upon all the 16 demo videos. We adopt elbow joints as indicators to determine whether the motion occurs. Empirically, when the pose changes by more than 5 degrees between adjacent frames, we nominate the speakers who are moving now. Therefore, among 16 demo videos with 6 seconds, the average motion fraction lengths of the two speakers are 4.3 and 3.1 seconds.
A higher score reflects better quality, with 5 signifying that the video fully meets the audience's expectations, while indicates that the video completely unacceptable. To ensure faimesss, each video is presented on a PowerPoint slide with a neutral background. Before participants see the generated results, we show several pseudo-annotated demos our dataset as reference. All participants are required to watch the video at least once before they rate it. We invite participants batches at different time periods within week. Once students have submitted their ratings anonymously, we collect them to calculate average score. After completing the statistics, randomly selected 60% of them to rate again two weeks later, and the results show that there is no obvious deviation.
We report the detailed mean and standard deviation for each method as shown in Table 6 Our method achieves 10% ((4.44.0)/4=10%) large marginal improvement over suboptimal InterGen in Naturalness. Meanwhile, our method displays a much lower standard deviation than InterX and InterGien. This indicates the much more stable performance of method against competitors.
18

---

Published as a conference paper at ICLR 2025
Table 7: Significance Analysis of User Study
Comparison Methods
Naturalness
Smoothness
Interaction Coherency
TalkSHOw
5.345
3.567
4.123
ProbTalk
3.789
5.001
3,456
DiftSHEG
4,789
2.654
5,299
EMAGE
3.214
5.120
4.789
MDM
3.789
4.567
2,987
InterX
3.001
3.456
2.148
InterGen
2.654
3,299
2.234
w/o Bilateral
w/o Audio
Separation
w/o TIM
w/o Mutual
Co'Gesture (Full)
Figure 7:
Visualization of our generated concurrent 3D co-speech gestures the ablation study.
Best view screen.
To further verify the effectiveness of our user study, we conduct significant analysis of the user study using t-test, focusing on three key aspects: Naturalness, Smoothness, and Interaction Coherency. The results verify our method surpasses all the counterparts with significant improvements, including sub-optimal InterGen. particular, for the comparisons between our model and the other models, formulate null hypotheses (HO) "our model does not outperform another method". In contrast, the alternative hypothesis (HI) posits that "our model significantly outperforms another method," with significance level (a) set 0.05. Here, perform series of t-tests to compare the rating scores of our model against each of the other competitors individually and calculate all the t-statistics shown in Table Since recruit 15 volunteers, our degree of freedom(df) for every analysis is 14. Then, we look up the t-table with two tails and find out all the p-values are less than 0.05 (a). Therefore. we reject the null hypotheses, indicating that our model significantly outperforms all the other methods in every aspect.
A.4 ADDITIONAL VISUALIZATION RESULTS
Here, we provide more visualization results of the ablation study in our experiments. As shown in Figure 7 the full version of our framework demonstrates the vivid and coherent interaction of body dynamics against other versions. We also display more visualized demo videos on our anonymous website:
19


Reviews:
[{"review_id": "gspcvYGWFD", "reviewer": "Reviewer_CYKi", "summary": "This paper introduces a novel task: generating concurrent co-speech gestures for two interacting characters based on conversational speech audio. The authors first present a new dataset, GES-Inter, which contains full-body postures of two interacting characters reconstructed from video recordings. They then propose a co-speech gesture generation framework, Co$^3$Gesture, built upon bilateral cooperative diffusion branches with an integrated Temporal Interaction Module. Experimental results on the GES-Inter dataset demonstrate that this framework outperforms several state-of-the-art methods.", "questions": "1. Could the authors specify which baseline methods utilized pre-trained models?\n\n2. Could the authors provide additional ablation studies incorporating simple fusion modules?\n\n3. Could the authors perform a significance analysis for the user study?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper introduces a novel task: generating concurrent co-speech gestures for interacting characters based on conversational speech audio.\n\n2. The proposed Co$^3$Gesture model, featuring the specially designed Temporal Interaction Module (TIM), appears capable of generating alternating co-speech gestures for two interacting characters.\n\n3. The authors have compiled a new dataset specifically for the concurrent co-speech gesture synthesis task.", "weaknesses": "1. The proposed Co$^3$Gesture model does not account for the spatial relationships between the two speakers. Specifically, Speaker A consistently appears on the left, and both speakers are always seated in chairs.\n\n2. The authors report using pre-trained models for some baseline methods. However, this approach is problematic, as it is unreasonable to expect pre-trained models to produce realistic interactive co-speech gestures if they have not been trained on the new dataset. Training these models on the newly collected dataset would likely yield more reliable results.\n\n3. In the ablation study, the authors present metrics only for models without TIM and mutual attention modules. The study would be more persuasive if the authors also included metrics using a simple fusion module, such as an MLP or a concatenation operator, rather than entirely omitting the fusion modules.\n\n4. Qualitative results for ablated models are not provided. Including these results would strengthen the evaluation.\n\n5. The user study and comparison videos show only a marginal improvement over InterGen.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B9O8VTuDk6", "reviewer": "Reviewer_2ueC", "summary": "The paper presents a novel dataset and method to tackle the problem of concurrent co-speech gesture generation for two persons in conversation. The dataset consists of about 70 hours of curated video footage consisting of various pairs of conversing people --- taken mostly from talk show videos in the public domain --- such that both their gestures are fully visible. The proposed method trains a diffusion network with a temporal interaction module (TIM) performing cross-attentions between the audio and motion features of the two persons. The output of the TIM serves as the conditioning signal for the denoising motion decoder to generate the 3D motions of the two persons from noise. The authors show the benefits of their proposed approach through quantitative and qualitative comparisons, ablation experiments, and a user study.", "questions": "Some typos, e.g.,\n\nLine 258: pheromones -> phonemes\n\nTable 3: Attetion -> Attention", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors have tackled the challenging problem of generating concurrent, two-person co-speech gestures, which is a natural expansion of the scope of current co-speech gesture generation methods, and meaningfully takes the field of human motion understanding forward.\n\n2. The proposed dataset is carefully curated and thoughtfully designed to contain a sufficient quantity of two-person gestures, making it a good candidate for a benchmark for two-person interactive motion generation problems.\n\n3. The proposed method is technically sound and provides a baseline on the proposed dataset.", "weaknesses": "1. Some additional dataset preprocessing details may be useful for completeness.\n\n    (a) Do the authors manually check the three preprocessing steps for each video (Lines 794-801)? How are quality and consistency ensured across the dataset?\n\n    (b) How are the occlusions determined in the videos? Also, if any person in the video has a non-sitting posture, e.g., standing or walking around, are those motions tracked and filtered out?\n\n2. Some details and motivations of the proposed approach are missing.\n\n    (a) Why do the authors use $C_{mix}$, and not a combination of the cleanly separated signals $C_a$ and $C_b$, to get the interactive motion embeddings (Line 266)? It would also be good to see any mathematical relationships, visual representations, or ablation experiments to understand how $C_a$, $C_b$, and $C_{mix}$ relate to each other (e.g., is $C_{mix} = C_a + C_b$)?\n\n    (b) During the generation, do the authors consider any global translation and orientation? In other words, do they consider any translation and orientation for the root joint, or is the root fixed in place? The visual results seem to suggest the latter, but it is not clear from the paper.\n\n    (c) Since the authors only consider upper body motions (Line 244), why are they using a foot contact loss (Eqn. 4)? Is it possible to quantify the benefits of the foot contact loss for this problem (e.g., through an ablation experiment)?\n\n3. The baseline methods the authors compare with (Sec. 4.2) are designed for single-person co-speech gestures. How do the authors adapt them for two-person co-speech gesture generation? Also, why are the authors not performing quantitative comparisons with more relevant two-person motion generation methods, such as InterGen (Liang et al. 2024b) or InterX (Xu et al. 2024)?\n\n4. Some user study details are also missing.\n\n    (a) How many generated videos does each participant watch?\n\n    (b) What is the mean and variance of the lengths of the videos they watch?\n\n    (c) What fractions of those video lengths contain the motions of person A, and what fraction contains the motions of person B? This might be calculated by assuming a person has motion in a particular frame if the minimum difference in their joint positions from a few previous frames is above some empirically determined thresholds.\n\n    (d) What is the standard deviation in the participant responses? Particularly, it seems that the mean values for InterX, InterGen, and the proposed method are quite close.\n\n5. Most of the visual results only show gestures of one person while the other person is sitting idle, and most of these results are only 3-4 seconds long. It is hard to appreciate the two-person motion generation performance from these results. The one example which is longer than 10 seconds and shows the gestures of two persons also exhibits jittery motion. Have the authors investigated the source of this jitter and explored any steps to reduce or remove it? Also, it would help to see some quantification of the balance of motions between the two persons (e.g., highlighting a person when they are gesticulating, showing the number of frames the person is gesticulating for) in the generated results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VYXS6hn38X", "reviewer": "Reviewer_ryqB", "summary": "This paper focuses on the task of interactive motion generation in two-person dialogue scenarios. It first introduces a large-scale dataset for two-person interactive motion, which includes body and finger movements as well as facial expressions. An interactive motion generation model is proposed, along with a Temporal Interaction Module to ensure the temporal synchronization of gestures. \nBased on the proposed GES-Inter dataset, the authors benchmark the model against other state-of-the-art algorithms.", "questions": "**Separation of Speaker Audio** How effective is the speech separation? Could more examples be provided to illustrate the accuracy of speech segmentation, text recognition, and alignment?\n\n**Only Upper Body**\nThe paper does not clearly explain why only upper body motions are generated, is it due to insufficient quality in the lower body data within the dataset?\n\n**Body Shape and root**\nWhy do the body shape and root position appear to be fixed?\n\n**Unclear Representation** \nThe dimensions and composition of inputs like audio c_a ​and x_a are not clearly specified. Why x_a  only include upper body movements, and which specific joints are used?  Why do all generated results have the same shape? Does x_a include facial expressions and shape parameters?\nSpecifying the exact dimensions and components of each input, including which joints are used for upper body movements and whether facial expressions and shape parameters are included, would be helpful.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper extracts 3D data from videos, collecting a large-scale dataset. Compared to laboratory-based data collection, capturing natural interactions yields more authentic interactive motions.\n\nThe proposed method employs Bilateral Cooperative Diffusion and the Temporal Interaction Module, which shown to be effective in experiments.\n\nThe dataset separates audio for the two speakers, recognizes the text corresponding to the audio, and invites participants to review the data, which is commendable.", "weaknesses": "**Dataset Quality**\nCapturing accurate two-person interactive motions from video is inherently challenging, which raises concerns about the dataset's quality. From the GES-Inter dataset examples shown in the videos [3], there are noticeable issues, such as hand self-penetration problems for the yellow character. Additionally, the dataset's frame rate is only 15 FPS, which is lower than that of mainstream motion datasets, limiting the compatibility of this dataset with other human motion datasets.\n\nBased on the dataset and generated examples provided, all individuals share the same shape parameters, showing no variation in body shape, and their root positions remain fixed. As a foundational dataset project, it is crucial for the authors to clarify any pre-processing or post-processing steps applied to the data and the motivations behind these decisions.\n\nTo help the authors further demonstrate the quality of GES-Interdataset and improve its compatibility with other datasets, I offer the following suggestions: **(1)** Provide more comprehensive example videos showcasing the dataset.\n**(2)** Consider offering solutions to improve the dataset's frame rate, such as FPS enhancement tools or post-processing scripts.\n**(3)** Further explain any pre or post processing applied to the data and clarify why there is no observable variation in human shape or root position in the dataset and generated examples.\n\n**Comparison**\nThe dataset may lacks a comparison with previous [2]. The authors should consider adding a table to quantitatively compare the GES-Inter dataset with [2], highlighting differences in aspects such as size, capture equipment, diversity, and quality metrics. This comparison would provide a clearer understanding of the dataset's advantages and limitations relative to previous work.\n\n**Interaction Metric**\nThe correspondence between the interactive motions and the given speech in two-person interactions appears weak. It may be helpful for the authors to identify and categorize the typical types of interactive motions and consider developing or adopting more suitable metrics for evaluating the quality of interactions. In particular, the current metrics, such as FGD, BC, and Diversity, are insufficient for this purpose, and incorporating more interaction-specific evaluation criteria would strengthen the assessment..\n\n\n**Audio process**\nIn the methodology section, audio input (C_mix) is simply fed into the Audio Encoder without special processing for each speaker’s audio, particularly in cases where overlapping speech occurs when both speakers talk simultaneously. The authors should clarify how their method handles overlapping speech. It may be helpful to conduct an ablation study or provide examples that demonstrate the model's performance specifically in scenarios with overlapping speech.\n\n**Ambiguous Figure**\nFigure 3 make ambiguous, the authors should revise the figure to accurately represent the inputs to the Audio Encoder as described in the text. Based on the paper's description, the input to the Audio Encoder should be C_mix={c_a,c_b}, rather than the isolated representation of C_mix,c_a and c_b  shown in the figure. To clarify, placing c_a and c_b  on the left side of the audio diagram and omitting the corresponding arrows to the Audio Encoder would be more accurate. \nAlternatively, if the figure is correct, you should clarify this in the text.\n\n**Reference**\n\n[1] Yi, Hongwei, et al. \"Generating holistic 3d human motion from speech.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[2] Lee, G., Deng, Z., Ma, S., Shiratori, T., Srinivasa, S. S., & Sheikh, Y. (2019). Talking with hands 16.2 m: A large-scale dataset of synchronized body-finger motion and audio for conversational motion analysis and synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 763-772).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "is5cAS7A89", "reviewer": "Reviewer_7SiD", "summary": "This paper explores learning conversational gesture generation from in-the-wild video data, different from previous works that focused on smaller, in-lab, or middle-scale motion-captured datasets. The authors make three key contributions:\n1. **Data Collection**: \n\n    The paper introduces a 70-hour conversational dataset with pseudo-labels, processed with temporal smoothing and filtering to ensure data quality.\n\n2. **Proposed Baseline Model**: \n\n    The authors propose a baseline model consisting of Temporal Interaction Network (TIN) and mutual attention mechanisms for conversational gesture generation.\n\n3. **Performance**: \n\n    The results demonstrate that the proposed method outperforms previous single-speaker or conversational speaker methods on the newly created GES-Inter dataset.", "questions": "My primary question (will influence I raise my score or not) is the same as my weakness. \n\n1.  **unclear sentence mutual attention module**: \n\n    Some explanation of the mutual attention module unclear, particularly in lines 280-282: \"We observe that exchanging the input order ... distribution.\" The notion of \"order\" here is confused for me. does that mean switch the order of \"question\" and \"answer\", such as, \"how are you, --> good. \" becomes \"good. --> how are you\"? This will influence my understanding of the design motivation of this module.\n\nSuggestions not influence my scoring:\n\n1. **Raw Data Release**: \n\n    I suggest the authors consider releasing the raw video data after the scene cuts. This would allow users to explore additional tasks, such as video generation, using the raw data.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Overall, I am quite positive about this paper. To the best of my knowledge, it is the first attempt to generate conversational gestures using in-the-wild video data. The key strengths include:\n\n1. **Data Cleanup**: \n\n    The data processing is thorough, as mentioned in the appendix. The original collection consisted of over 1,000 hours of video, but after a 12-step filtering process, only 70 hours remained. This shows a well-designed filtering strategy, including speaker occurrence rules, to ensure data quality.\n   \n2. **Model Simplicity and Reproducibility**: \n\n    The model design is novel yet simple for a baseline approach. Instead of making the model overly complex, the authors build on strightforward diffusion models with audio inputs from two speakers. The plug-in TIN and mutual attention modules make this work easy to reproduce and extend in future studies.\n\n3. **TIN Design**: \n\n    The Temporal Interaction Network captures selective features refined from either single-speaker voices or mixed voices from both speakers, aligning with the physical nature of conversational gestures.\n\n4. **Experimental Results**: \n\n    The experiments compare recent baselines for both single-speaker and conversational gesture generation. The model achieves new SOTA results on the GES-Inter dataset.", "weaknesses": "While the paper is strong overall, I have one unclear point:\n1.  **unclear sentence mutual attention module**: \n\n    Some explanation of the mutual attention module unclear, particularly in lines 280-282: \"We observe that exchanging the input order ... distribution.\" The notion of \"order\" here is confused for me. does that mean switch the order of \"question\" and \"answer\", such as, \"how are you, --> good. \" becomes \"good. --> how are you\"? This will influence my understanding of the design motivation of this module.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)