Automatic Sign Language Production
Creators: name: STOLL, STEPHANIE MARLIS
Publisher: name: University of Surrey
Year: 2022

description: Sign Languages are the dominant form of communication used by the Deaf and Hard of
description: Hearing (HoH). Whilst there are technologies to translate between the hearing and the Deaf
description: and HoH, none of them provide the means for clear and easy communication. This is due
description: to the asynchronous multi-channel nature of sign languages, which makes it a more complex
description: translation problem, than translating between spoken languages. A common misconception is
description: that a sign language is just a sign for word replacement and that written language is sufficient.
description: However, a Deaf or HoH who was raised and educated in sign language might not be able
description: to read the spoken language of their country well enough. Similarly, an avatar, driven using
description: parametrised signs often produces incoherent and hard to understand signings. If it is driven
description: using motion capture data, the avatar produces acceptable signings, but this approach is costly
description: and not scalable. The research in this thesis addresses this problem by proposing data-driven,
description: deep-learning-based Sign Language Production (SLP).
description: In our first contribution chapter we introduce a novel approach to automatic SLP using recent
description: developments in Neural Machine Translation (NMT), Generative Adversarial Networks (GANs),
description: and motion generation. This preliminary system is capable of producing sign videos from spoken
description: language sentences. Contrary to former approaches that are dependent on heavily annotated data,
description: this approach requires minimal gloss and skeletal level annotations for training. We achieve
description: this by breaking down the task into dedicated sub-processes. We first translate spoken language
description: sentences into sign pose sequences by combining an NMT network with a Motion Graph (MG).
description: The resulting pose information is then used to condition a generative model that produces photo
description: realistic sign language video sequences. This is the first approach to continuous sign video
description: generation that does not use an avatar. We evaluate the translation abilities of our approach
description: on the PHOENIX14T Sign Language Translation dataset and set a baseline for text-to-gloss
description: translation. We further demonstrate the video generation capabilities of our approach for both
description: multi-signer and high-definition settings qualitatively and quantitatively using broadcast quality
description: assessment metrics.
description: In our second contribution chapter we focus on incorporating non-manuals (e.g. facial expressions, body and head pose) and increasing the resolution of produced utterances. We introduce a gloss2pose network architecture that is capable of generating human pose sequences conditioned
description: on glosses. Combined with a generative adversarial pose2video network, we are able to produce
description: natural-looking, high definition sign language video. For sign pose sequence generation, we
description: outperform our previous contribution by a factor of 18, with a Mean Square Error of 1.0673 in
description: pixels. For video generation we report superior results on three broadcast quality assessment
description: metrics. To evaluate our full gloss-to-video pipeline we introduce two novel error metrics,
description: to assess the perceptual quality and sign representativeness of generated videos. We present
description: promising results, significantly outperforming the then state-of-the-art in both metrics.
description: Our previous two contributions relied on gloss information. To make automatic SLP truly
description: scalable this reliance needs to be eliminated. Hence, the final contribution chapter introduces the
description: first method to automatically generate dense 3D sign sequences from text only. The approach
description: requires simple 2D annotations for training, which can be automatically extracted from video.
description: Rather than incorporating high-definition or motion capture data, we propose back-translation
description: as a powerful paradigm for supervision: By first addressing the arguably simpler problem
description: of translating 2D pose sequences to text, we can leverage this to drive a transformer-based
description: architecture to translate text to 2D poses. These are then used to drive a 3D mesh generator.
description: Our mesh generator P ose2Mesh us