DiffSign: A Different Approach to Continuous Sign Language Recognition with Diffusion Model

Authors: Xi Geng, Yunan Li, Zhuoqi Ma, Qiguang Miao
Primary area: generative models
Keywords: Continuous sign language recognition, Video-to-text generation, Weakly supervised, Contrastive learning

Abstract:
Cross-modal alignment is a general way for continuous sign language recognition (CSLR) tasks. However, Due to the weakly supervised nature of CSLR, manual alignment often fails to map sign frames to glosses accurately. 
In this paper, we propose a diffusion-based framework, achieving CSLR in a new view based on cross-modal generation, leveraging the inherent semantic consistency between sign videos and glosses. To address the issue of ambiguous boundaries in sign videos, we have also developed a contrastive learning-based feature enhancement strategy, which serves as a more sophisticated alternative to the simple attention mechanisms commonly used in text-to-image generation tasks. 
Extensive experiments on three public sign language recognition datasets demonstrate the effectiveness of generation way in CSLR and it can achieve better performance than state-of-the-art methods.
The code of our method will be available upon acceptance.

Decision: reject

Reviews and discussion:

Metareview: