Understanding Co-speech Gestures in-the-wild
Authors: name: Sindhu B Hegde, name: K R Prajwal, name: Taein Kwon, name: Andrew Zisserman

Co-speech gestures play a vital role in non-verbal communication. In this
paper, we introduce a new framework for co-speech gesture understanding in the
wild. Specifically, we propose three new tasks and benchmarks to evaluate a
model's capability to comprehend gesture-text-speech associations: (i)
gesture-based retrieval, (ii) gestured word spotting, and (iii) active speaker
detection using gestures. We present a new approach that learns a tri-modal
speech-text-video-gesture representation to solve these tasks. By leveraging a
combination of global phrase contrastive loss and local gesture-word coupling
loss, we demonstrate that a strong gesture representation can be learned in a
weakly supervised manner from videos in the wild. Our learned representations
outperform previous methods, including large vision-language models (VLMs),
across all three tasks. Further analysis reveals that speech and text
modalities capture distinct gesture-related signals, underscoring the
advantages of learning a shared tri-modal embedding space. The dataset, model,
and code are available at: https://www.robots.ox.ac.uk/~vgg/research/jegal