Neural Sign Language Recognition and Translation
Creators: name: CAMGÖZ, NECATI CIHAN
Publisher: name: University of Surrey
Year: 2020

description: Sign languages have been studied by computer vision researchers for the last three
description: decades. One of the end goals of vision-based sign language research is to build systems
description: that can understand and translate sign languages to spoken/written languages or vice
description: versa, to create a more natural medium of communication between the hearing and the
description: Deaf. However, most research to date has mainly focused on isolated sign recognition
description: and spotting, neglecting the underlying rich grammatical and linguistic structures of sign
description: language that differ from spoken language. More recently, Continuous Sign Language
description: Recognition (CSLR) has become feasible with the availability of large benchmark
description: datasets, such as the RWTH-PHOENIX-Weather-2014 Dataset (PHOENIX14), and the
description: development of algorithms that can learn from weak annotations. Although, CSLR
description: is able to recognize sign gloss sequences, further progress is required to produce
description: meaningful spoken/written language interpretations of continuous sign language videos.
description: In this thesis, we introduce the Sign Language Translation (SLT) problem and lay
description: groundwork for future research on this topic. The objective of SLT is to generate
description: spoken/written language translations from continuous sign language videos, taking into
description: account the different word orders and grammar. We evaluate our approaches on the
description: RWTH-PHOENIX-Weather-2014T (PHOENIX14T) dataset, the first and the currently
description: only publicly available Continuous SLT dataset aimed at vision based sign language
description: research. It provides spoken language translations and gloss level annotations for
description: German Sign Language videos of weather broadcasts. We lay down several evaluation
description: protocols to underpin future research in this newly established field.
description: In the first contribution chapter of this thesis, we formalize SLT in the framework
description: of Neural Machine Translation (NMT) and propose the first SLT approach, Neural
description: Sign Language Translation. We combine Convolutional Neural Networks (CNNs) and
description: attention-based encoder-decoder models, which allows us to jointly learn the spatial
description: representations, the underlying language model, and the mapping between sign and
description: spoken language. We investigate different configurations of the proposed network
description: with both end-to-end and pretrained settings (using expert gloss annotations). In
description: our experiments, recognizing glosses and then translating them to spoken languages
description: (Sign2Gloss2Text) drastically outperforms an end-to-end direct translation approach
description: (Sign2Text). Sign2Gloss2Text utilizes a state-of-the-art CSLR model to predict gloss
description: sequences from sign language videos and then solves SLT as text-to-text translation
description: problem. This suggests that using gloss level intermediate representations, essentially
description: dividing the process into two stages, is necessary to train accurate SLT models.
description: Glosses are incomplete text-based representations of continuous multi-channel visual
description: signals, that are sign languages. Thus, the best performing two step configuration of
description: Neural Sign Language Translation has an inherent information bottleneck limiting
description: translation. To address this issue, in the second contribution chapter of this thesis we
description: formulate SLT as a multi-task learning problem. We introduce a novel transformer
description: based architecture, Sign Language Transformers, that jointly learn CSLR and SLT while
description: being trainable in an end-to-end manner. This is achieved by using a Connectionist
description: Temporal Classification (CTC) loss to bind the recognition and translation problems
description: into a single unified architecture. This joint approach does not require any ground-truth
description: timing information, simultaneously solving two co-dependant sequence-to-sequence
description: learning problems and leads to significant performance gains. We report state-of-the-art
description: CSLR and SLT results achieved by our Sign Language Transformers. Our translation
description: networks outperform both sign video to spoken language and gloss to spoken language
description: translation models, in some cases more than doubling the performance of Neural Sign
description: Language Translation (Sign2Text configuration - 9.58 vs. 21.80 BLEU-4 Score).
description: Models we introduce in both first and second contribution chapters heavily rely on
description: gloss information, either in the form of direct supervision or for pretraining. To realize
description: large scale sign language translation, that is on par with their spoken/written language
description: counterparts, we require more parallel datasets. However, annotating sign glosses is a
description: laborious task and acquiring such annotations for large datasets is infeasible. To address
description: this issue, in our last contribution chapter we propose modelling SLT based on sign
description: articulators instead of glosses. Contrary to previous research, which mainly focused on
description: manual features, we incorporate both both manual and non-manual features of the sign.
description: We utilize hand shape, mouthings and upper body pose representations to model sign in
description: a holistic manner.
description: We propose a novel transformer based architecture, called Multi-Channel Transformers,
description: aimed at sequence-to-sequence learning problems where the source information is
description: embedded over several channels. This approach allows the networks to model both the
description: inter and the intra relationship between asynchronous source channels. We also intro-
description: duce a channel anchoring loss to help our models preserve channel specific information
description: while also regulating training against overfitting.
description: We apply multi-channel transformers to the task of SLT and realize the first multi-
description: articulatory translation approach. Our experiments on PHOENIX14T demonstrate that
description: our approach achieves on par or better translation performance against several baselines,
description: overcoming the reliance on gloss information which underpin previous approaches.
description: Now we have broken the dependency upon gloss information, future work will be to
description: scale learning to larger datasets, such as broadcast footage, where gloss information is
description: not available.