Kati8/ICLR_2023_2025 huggingface dataset:
authors: Xingqun Qi
authors: Yatian Wang
authors: Hengyuan Zhang
authors: Jiahao Pan
authors: Wei Xue
authors: Shanghang Zhang
authors: Wenhan Luo
authors: Qifeng Liu
authors: Yike Guo
reviews: 
Comments: This paper explores learning conversational gesture generation from in-the-wild video data, different from previous works that focused on smaller, in-lab, or middle-scale motion-captured datasets. The authors make three key contributions:
1. **Data Collection**: 

    The paper introduces a 70-hour conversational dataset with pseudo-labels, processed with temporal smoothing and filtering to ensure data quality.

2. **Proposed Baseline Model**: 

    The authors propose a baseline model consisting of Temporal Interaction Network (TIN) and mutual attention mechanisms for conversational gesture generation.

3. **Performance**: 

    The results demonstrate that the proposed method outperforms previous single-speaker or conversational speaker methods on the newly created GES-Inter dataset.
Strengths: Overall, I am quite positive about this paper. To the best of my knowledge, it is the first attempt to generate conversational gestures using in-the-wild video data. The key strengths include:

1. **Data Cleanup**: 

    The data processing is thorough, as mentioned in the appendix. The original collection consisted of over 1,000 hours of video, but after a 12-step filtering process, only 70 hours remained. This shows a well-designed filtering strategy, including speaker occurrence rules, to ensure data quality.
   
2. **Model Simplicity and Reproducibility**: 

    The model design is novel yet simple for a baseline approach. Instead of making the model overly complex, the authors build on strightforward diffusion models with audio inputs from two speakers. The plug-in TIN and mutual attention modules make this work easy to reproduce and extend in future studies.

3. **TIN Design**: 

    The Temporal Interaction Network captures selective features refined from either single-speaker voices or mixed voices from both speakers, aligning with the physical nature of conversational gestures.

4. **Experimental Results**: 

    The experiments compare recent baselines for both single-speaker and conversational gesture generation. The model achieves new SOTA results on the GES-Inter dataset.
Weaknesses: While the paper is strong overall, I have one unclear point:
1.  **unclear sentence mutual attention module**: 

    Some explanation of the mutual attention module unclear, particularly in lines 280-282: "We observe that exchanging the input order ... distribution." The notion of "order" here is confused for me. does that mean switch the order of "question" and "answer", such as, "how are you, --> good. " becomes "good. --> how are you"? This will influence my understanding of the design motivation of this module.
Recommendation: 10
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4
reviews: 
Comments: This paper introduces a novel task: generating concurrent co-speech gestures for two interacting characters based on conversational speech audio. The authors first present a new dataset, GES-Inter, which contains full-body postures of two interacting characters reconstructed from video recordings. They then propose a co-speech gesture generation framework, Co$^3$Gesture, built upon bilateral cooperative diffusion branches with an integrated Temporal Interaction Module. Experimental results on the GES-Inter dataset demonstrate that this framework outperforms several state-of-the-art methods.
Strengths: 1. This paper introduces a novel task: generating concurrent co-speech gestures for interacting characters based on conversational speech audio.

2. The proposed Co$^3$Gesture model, featuring the specially designed Temporal Interaction Module (TIM), appears capable of generating alternating co-speech gestures for two interacting characters.

3. The authors have compiled a new dataset specifically for the concurrent co-speech gesture synthesis task.
Weaknesses: 1. The proposed Co$^3$Gesture model does not account for the spatial relationships between the two speakers. Specifically, Speaker A consistently appears on the left, and both speakers are always seated in chairs.

2. The authors report using pre-trained models for some baseline methods. However, this approach is problematic, as it is unreasonable to expect pre-trained models to produce realistic interactive co-speech gestures if they have not been trained on the new dataset. Training these models on the newly collected dataset would likely yield more reliable results.

3. In the ablation study, the authors present metrics only for models without TIM and mutual attention modules. The study would be more persuasive if the authors also included metrics using a simple fusion module, such as an MLP or a concatenation operator, rather than entirely omitting the fusion modules.

4. Qualitative results for ablated models are not provided. Including these results would strengthen the evaluation.

5. The user study and comparison videos show only a marginal improvement over InterGen.
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3