Photo-Realistic Sign Language Production
Creators: name: Saunders, Benjamin
Publisher: name: University of Surrey
Year: 2024
Subjects: Sign Language Production
Photo-Realistic Sign Language Video Generation
Continunous Sequence-to-sequence Generation

description: Most computational sign language research focuses on the recognition and translation
description: of sign languages to spoken languages. Although useful, this technology is more
description: applicable to a hearing person understanding a Deaf person, and often not that helpful
description: for the Deaf community. The opposite task of Sign Language Production (SLP), the
description: translation of spoken language sentences into sign language videos, is far more relevant
description: to the Deaf and could significantly increase the availability of sign language content.<br>
description: <br>
description: Traditional SLP research focused on avatar-based techniques that generated cartoon and
description: robotic sign outputs, with a reliance on simple phrase lookup and expensive MoCap
description: technology. Recently, there has been an increase in deep learning approaches to SLP.
description: However, these works often focus on the production of isolated signs without realistic
description: transitions and a skeleton pose representation, resulting in robotic and non-realistic
description: animations that are poorly received by the Deaf.<br>
description: <br>
description: In this thesis, we improve the capability of SLP technology, focusing on the production
description: of photo-realistic continuous sign language videos direct from spoken language text.
description: We first present a baseline approach using concatenated isolated signs, and propose a
description: novel ’back translation’ evaluation metric used for the rest of the thesis.<br>
description: <br>
description: In our first contribution chapter, we present the first continuous SLP model to translate
description: from spoken language sentences to continuous sign language sequences in an end-to-end
description: manner. We introduce a Progressive Transformer architecture that uses an alternative
description: formulation of transformer decoding for continuous sequences. We propose both
description: adversarial training and Mixture Density Network (MDN) modelling to tackle
description: underarticulated outputs, and show improved model performance through both quantitative
description: back translation results and qualitative Deaf user studies.<br>
description: <br>
description: Building on the feedback to our continuous SLP approach, we next attempt to combine
description: the benefits of both continuous and isolated production. In our second contribution, we
description: separate the SLP task into two distinct but jointly-trained sub-tasks of translation and
description: animation, with an intermediary gloss supervision. We propose a Mixture of Motion
description: Primitives (MOMP) architecture that learns to combine specialised skeleton motion
description: primitives to produce novel sequences and reduces the adverse effect of ’regression to
description: the mean’.<br>
description: <br>
description: Although considerable progress towards continuous SLP was made in earlier
description: contributions, Deaf feedback shows the produced sequences still under-articulate hand
description: motion compared to baseline isolated methods. Our third contribution proposes a learnt
description: co-articulation between isolated signs to better reflect continuous signing whilst still
description: maintaining the inherent understandable nature seen in dictionary signs. We propose a
description: novel Frame Selection Network to learn the optimal subset of frames that best maps to
description: a continuous signing sequence. We conduct extensive deaf user evaluation to show that
description: this approach improves the natural signing motion of concatenated isolated sequences
description: and is overwhelmingly preferred to both previous contributions and baseline models.<br>
description: <br>
description: Previous contributions use an output skeleton pose representation, which has been
description: shown to be a major factor in the lack of comprehension. In our final contribution,
description: we introduce SIGNGAN, the first SLP model to produce photo-realistic sign language
description: videos to a level understandable by native Deaf signers. We use skeleton pose sequences
description: to condition a video-to-video synthesis model with a novel keypoint-based loss to
description: improve hand synthesis quality and a style conditioning to generate novel human
description: appearances. Finally, we conduct a Deaf user evaluation to show that SIGNGAN
description: photo-realistic outputs are more understandable than skeleton pose sequences.<br>
description: <br>
description: Given the contributions above, this thesis proposes an end-to-end pipeline to produce
description: photo-realistic sign language sequences from spoken language sentences. As we have
description: shown a potential pipeline to produce large-scale unconstrained translation, we suggest
description: future work focuses on the currently under-developed Text to Gloss translation step
description: alongside an automated collection of the large-scale datasets this task requires.<br>