Sign Language Translation with Sentence Embedding Supervision (2024)

Abstract: State-of-the-art sign language translation (SLT) systems facilitate the learning process through gloss annotations, either in an end2end manner or by involving an intermediate step. Unfortunately, gloss labelled sign language data is usually not available at scale and, when available, gloss annotations widely differ from dataset to dataset. We present a novel approach using sentence embeddings of the target sentences at training time that take the role of glosses. The new kind of supervision does not need any manual annotation but it is learned on raw textual data. As our approach easily facilitates multilinguality, we evaluate it on datasets covering German (PHOENIX-2014T) and American (How2Sign) sign languages and experiment with monoand multilingual sentence embeddings and translation systems. Our approach significantly outperforms other gloss-free approaches, setting the new state-of-the-art for data sets where glosses are not available and when no additional SLT datasets are used for pretraining, diminishing the gap between glossfree and gloss-dependent systems.

heading: 1 Introduction
text: Sign Language Translation (SLT) aims at generating text from sign language videos. There are several approaches to SLT reported in the literature, with sign2text and sign2gloss2text the most widely used. While sign2text directly translates video into text with or without the help of glosses (Camgöz et al., 2018), sign2gloss2text passes through an intermediate gloss step before translation into spoken language text (Ormel et al., 2010). That is, sign2gloss2text breaks down the problem into two independent sub-problems using glosses as a pivot language. A gloss is a textual label associated with a sign, and, although human signers do not in general use them, performance in automatic SLT has long been upper bounded by the gloss supervision and their use as an intermediate representation
(Camgöz et al., 2018). The advantage of translation without glosses is that collecting data is much easier. Even though translation results are better for approaches that use glosses as intermediate representation (Chen et al., 2022a,b), this comes at the cost of annotating all the video data with glosses which is a time consuming manual task. For many data sets glosses are simply not available. On the plus side, with gloss supervision-based SLT architectures, one can take full advantage of the maturity of text2text machine translation between glosses and spoken language text.
In this work, we present a novel approach sign2(sem+text), a model that gets rid of glosses and adds supervision through sentence embeddings, SEM, pretrained on raw text and finetuned for sign language. Our experimental results demonstrate the strength of the novel approach on both standard small datasets with gloss annotation and larger datasets without. In the latter case, we achieve state-of-the-art results for the American Sign Language (ASL) dataset How2Sign when no additional SLT datasets are used1 improving over Tarrés et al. (2023) by 4 BLEU points. For German Sign Language (DGS), our new approach achieves translation quality scores between the previous best glossfree system (Zhou et al., 2023) and the current state-of-the-art using glosses (Chen et al., 2022b) on the PHOENIX-2014T dataset. Our code and models are publicly available.2

heading: 2 Related Work
text: Camgöz et al. (2018) proposed three formalisations considering SLT as a seq2seq problem that converts a sequence of signs into a sequence of words: (i) sign2text, a model that encodes video frames using pretrained 2D CNNs as spatial features and then
1Uthus et al. (2023) and Rust et al. (2024) obtain better results by using the YouTube-ASL dataset for pretraining.
2https://github.com/yhamidullah/sem-slt
425
uses an RNN to generate the text; (ii) gloss2text, a model learning the translation between a sequence of textual glosses and fluent spoken language text; and (iii) a sign2gloss2text model that adds an extra intermediate gloss layer between the video and output text levels of a sign2text architecture to provide additional gloss supervision using a CTC loss.
In follow-up work, Camgöz et al. (2020) proposed an architecture for joint learning continuous sign language recognition (CSLR) and SLT which uses the same input as Camgöz et al. (2018); Zhou et al. (2021), that is, pretrained visual features, but a transformer (Vaswani et al., 2017) for text generation. Camgöz et al. (2020) conjectured that gloss2text results with ground-truth glosses provide an upper bound for SLT. Supporting this assumption, their translation quality on PHOENIX2014T as measured by BLEU (Papineni et al., 2002) achieved 24.5 on gloss2text and 21.8 on sign2text.
Yin and Read (2020) used a different visual representation with a multi-cue network (Zhou et al., 2020) to encode videos. Cues included face, hands and pose besides the full frame. With a BLEU score of 24.0 they impoved over sign2gloss2text Camgöz et al. (2020) and concluded that their visual representation was better than the spatial frame embeddings used by the Camgöz et al. (2020).
Chen et al. (2022a,b) used both pretraining of a network based on S3Ds (Xie et al., 2018) on action recognition for CSLR (sign2gloss) and pretraining of a textual transformer (gloss2text) with mBART-25 (Liu et al., 2020). Both types of pretraining are progressively adapted to the domain of the task by adding data closer to the domain. An additional mapping network between the vision and language parts allows Chen et al. (2022a) to build an end2end sign2text model relying on inter-
nal gloss supervision. To the best of our knowledge, Chen et al. (2022b) is the current state of the art for both sign2text (BLEU=28.95) and sign2gloss2text (BLEU=26.71), all on the PHOENIX-2014T data set.
Over the last few years, several gloss-free models have emerged (Li et al., 2020; Zhao et al., 2022; Yin et al., 2023). Zhou et al. (2023) obtains the current state-of-the-art in this category by utilising visual-language pretraining following CLIP (Radford et al., 2021). On the datasets (Camgöz et al. (2018); Zhou et al. (2021)) where the two approaches can be compared, translation quality diminishes by up to 7 BLEU points when the glosses are not used (Yin et al., 2023; Zhou et al., 2023). Tarrés et al. (2023) uses the How2Sign dataset (Duarte et al. (2021)) (where no gloss information is available) with I3D (Carreira and Zisserman (2017)) features for video representations and a Transformer. Uthus et al. (2023) introduces a new dataset, YouTube-ASL, 10 times larger than the previous one (Duarte et al. (2021)), and uses 2D pose estimation and pretraining to improve on Tarrés et al. (2023) best results on How2Sign (BLEU=8.09 vs BLEU=12.4). Simultaneously with our work, Rust et al. (2024) pretrains a self-supervised and privacy-aware visual model on YouTube-ASL to achieve the new state-of-theart performance on How2Sign (BLEU=15.5).

heading: 3 SEM-based Architectures
text: In our work we build two systems that revolve around textual sentence embeddings, SEM, as depicted in Figure 1. The figure presents two independent modules sign2sem and sem2text (left plots) that we later combine in sign2sem2text and sign2(sem+text) in an end2end setting (right plot).
• sign2sem Module This module predicts an intermediate SEM vector. Given a set of frames (video) features, sign2sem produces a vector representing the sentence signed in the video using a transformer encoder.
Pretraining the visual feature sentence embedding model on text. We follow Reimers and Gurevych (2019) and train a Siamese network with twin subnetworks 1 and 2. We compute the loss as the minimum squared error (MSE):
Le = 1
N
N∑
i=1
(cos(S1,i, S2,i)− cos(E1,i, E2,i))2
where N is the batch size, and S and E contain the target text SEM vectors and the predicted output SEM vectors respectively. In our experiments, the target SEM vector is given by sBERT (Reimers and Gurevych, 2019) here and in our models below.
• sem2text Module This module is responsible for the text reconstruction from sentence embeddings SEM. It produces the text translation of the video features encoded in a given SEM vector. The core sem2text model is a transformer model; we compare encoder–decoder and only decoder systems for the task:
- Encoder–decoder (SLTr): this version uses the sign language transformer (SLTr) architecture as in Camgöz et al. (2020). We use a transformer base with a linear projection from the SEM vector input instead of the usual word embedding layer.
- Decoder only with pretrained mBART: this version uses a pre-trained mBART-25 decoder and a linear layer to project the SEM vectors into the mBART model dimensions.
Pretraining We train both transformers (SLTr from scratch and the already pretrained mBART25) with Wikipedia data and then finetune them on the SL datasets. We compute the translation output loss as the cross-entropy:
Lo = CE(T,O) = − 1
N
N∑
i=1
M∑
j=1
(Tij · log(Oij))
where N is the batch size, M the vocabulary size, T is the target text and O is the output text.
After pretraining each component (sign2sem and sem2text), we combine them together for end2end training. We explore two approaches: an approach that only uses the output loss Lo, sign2sem2text, and an approach that integrates an additional supervision loss Le, sign2(sem+text).
• sign2sem2text is a simple pipeline combination of sign2sem and sem2text where the output SEM of the first module is used as input by the second module to obtain the final text prediction. The two pretrained modules (with both variants SLTr and mBART) are put together and trained in an end2end manner without any intermediate supervision. This formalisation is the sentence embedding equivalent to the sign2gloss2text approach.
• sign2(sem+text) performs translation using the same components as sign2sem2text. However, it uses the sign2sem SEM output as additional intermediate supervision using MSE loss computed against the target text SEM in a multitask learning approach. Both, Le (sentence embedding) and Lo (output text), are used jointly to train the model.
For SLTr, we take the SEM before the tanh and pooling (see Figure 1 (left–middle)), and project it into the SLTr model dimension. The supervision is applied after the SLTr encoder. For mBART, the supervision happens right before the mBART.
Our architectures can be trained both monolingually and mutilingually simply by using multilingual embeddings and merging multilingual training data.

heading: 4 Experimental Settings
text: We use two diverse (language and domain) datasets for our experiments:
RWTH-PHOENIX-2014T (Camgöz et al., 2018) 11 hours of weather forecast videos from 9 signers. Signers use German Sign Language and both transcriptions and glosses are available.
How2Sign (Duarte et al., 2021) 80 hours of instructional videos with speech and transcriptions and their corresponding American Sign Language videos (glosses unavailable) from 11 signers.
Detailed statistics for each dataset are provided in Appendix A. We preprocess the textual part of the datasets in a way that allows us to compare to the results obtained by Camgöz et al. (2018). We tokenise and lowercase the input for both training and evaluation. We apply BPE (Sennrich et al., 2016) with a vocabulary size of 1500 for Phoenix2014T and 5000 for How2Sign. When pretraining sem2text SLTr, we use a shared (en–de) vocabulary size of 32000. In cases where we use pretrained models, we keep the tokenisation of the model.
For video files, we extract frames using ffmpeg. We normalise the images, and resize them to 224x224. In this step, we initially obtain frame features from a pretrained model (Tan and Le, 2019), which does not contain gloss information. We then apply pooling to remove the spatial dimensions, followed by batch normalisation with ReLU, following the approach outlined by Camgöz et al. (2020). This generic approach facilitates the combination of datasets in the multilingual setting.
We use two multilingual pretrained models that cover both German and English, sBERT (Reimers and Gurevych, 2019)3 for sentence embeddings and mBART (Liu et al., 2020)4 as a language model. For further pretraining we use 26 million sentences per language from the English and German Wikipedia dumps extracted with Wikitailor (España-Bonet et al., 2023).
Following Müller et al. (2022) and Müller et al. (2023), we evaluate the models using three common automatic metrics in machine translation: BLEU (Papineni et al., 2002), chrF (Popović, 2015) and BLEURT (Sellam et al., 2020). Specifics can be found in Appendix C. In all cases, we estimate 95% confidence intervals (CI) via bootstrap resampling (Koehn, 2004) with 1000 samples.

heading: 5 Results and Discussion
text: Table 1 presents the results for our models and variants. Two major trends are observed: (i) massive pretraining of the sem2text module (mBART vs SLTr) significantly improves the results, confirming the observations by Chen et al. (2022a)
3We use all-MiniLM-L12-v2 model with 384 dimensions. 4We use mBART-25 1024 dimensions.
and (ii) the multitask approach sign2(sem+text) is better than the pipeline approach sign2sem2text. These findings hold for all three evaluation metrics at 95% confidence level.
Potentially beneficial effects of multilinguality are less evident. Monolingual and multilingual approaches are not distinguishable within the 95% CIs, possibly due to large differences in the domain of the datasets preventing effective transfer between languages.
Our best system, sign2(sem+text) with the pretrained text decoder, achieves state-of-the-art results on How2Sign when no additional SLT dataset is used for pretraining, improving from 8 to 12 BLEU points over Tarrés et al. (2023). For PHOENIX-2014T, we surpass all previous glossfree approaches (24 vs 21 BLEU), but we are still below the best approach that uses glosses (Chen et al., 2022b) (24 vs 29 BLEU).
Reconstruction quality: sem2text. In our approach, sentence embeddings take the role of manually produced glosses in previous work. Our sem2text translation module defines the upper-bound results for the full system as gloss2text did in previous work. Our best sign2(sem+text) models with mBART produce a reconstruction score of BLEU 38.0±2.4/23.3±1.0, chrF 57.5±1.9/43.9±0.9 and BLEURT 0.588±0.019/0.571±0.008 for PHOENIX-2014T/How2Sign (see Table 2). Where the comparison with glosses is available (PHOENIX), we improve over gloss2text by up to 10 BLEU points. We hypothesise that a sentence is better represented by its embedding than by a string of glosses and this explains why the translation
quality for sem2text is higher than for gloss2text. If these components (sem2text and gloss2text) are the upper-bound to the end2end sign2text translation, SEM-based systems are potentially at an advantage. These results, together with the fact that SEM models can be applied to raw data without annotations, highlight the promising future prospects of, especially, sign2(sem+text).
SEM-based vs gloss-based SLT. For comparison purposes, we integrate SEM supervision in a state-of-the-art gloss-based SLT system, Signjoey (Camgöz et al., 2020), by replacing their gloss supervision by SEM supervision. We perform no pretraining and train the two systems under the same conditions. We observe that convergence with SEM is faster and requires less than half of the iterations to finish (5k vs 12k) using the same setting and resources. The detailed training evolution
is shown in Appendix D.
Translation quality vs output length. Figure 2 shows the token length distribution of PHOENIX2014T and How2Sign along with the average BLEURT score on each interval. The equivalent plots for chrF and BLEU are in Figures 4 and 5 in Appendix E respectively. In the PHOENIX test set, almost 90% of the sentences contain 20 tokens or less, while the number decreases to 60% for How2Sign. The 10-20 token range is the one with the best scores. While the drop in performance in translation quality for long sentences is smaller in How2Sign, the difference in the distribution affects the global quality.

heading: 6 Conclusions
text: We present a new approach to sign language translation using automatically computed sentence embeddings instead of manual gloss labels as intermediate representation with (sign2(sem+text)) and without (sign2sem2text) SEM supervision. We outperform the state-of-the-art of gloss-free SLT when no additional SLT datasets are used for pretraining, closing the gap to gloss-based SLT.
According to the upper-bound set by sem2text translation quality, there is still room for improvement for the end2end SEM-based SLT models. In this work, we limited ourselves to existing visual feature extractors, in the future we plan to train a SEM-based visual feature extractor on SL datasets in order to get closer to our sem2text upper-bound and match gloss-based performance.
Limitations
Our SL datasets cover American English and German. Sentence embeddings for these languages are good quality as lots of textual data is available
for pre-training. It remains to be studied how the quality of the embeddings affects the final translation quality. This is important for low-resourced languages, i.e. languages with limited amounts of monolingual text data but, to the best of our knowledge, no public sign language data set exists for them.

heading: Acknowledgements
text: This work has been funded by by BMBF (German Federal Ministry of Education and Research) via the project SocialWear (grant no. 01IW20002).

heading: A Datasets Statistics
text: Table 3 summarises the statistics for the corpora used in the experiments.
B Infrastructure and Network Hyperparameters
We implement our SLT framework using PyTorch, and libraries from sBERT (Reimers and Gurevych, 2019) and Huggingface (Wolf et al., 2019). Our code is publicly available at Github.5
Tables 4, 5 and 6 show the hyperparameters and training times for the sign2sem and sem2text with SLTr and mBART transformers respectively. We run our experiments using 8 A100-80GB GPUs. For sign2sem2text and sign2(sem+text), each experiment runs for 72 hours and the configurations are inherited from the standalone modules sign2sem and sem2text.

heading: C Automatic Evaluation
text: Following Müller et al. (2022) and Müller et al. (2023), we evaluate the models using three common automatic metrics in machine translation: BLEU (Papineni et al., 2002), chrF (Popović, 2015) and BLEURT (Sellam et al., 2020). Notice that even though other semantic metrics based on embeddings might correlate better with human judge-
5https://github.com/yhamidullah/sem-slt
ments (Kocmi et al., 2021; Freitag et al., 2022), they cannot be used for sign language translation because the source is video and not text. We use sacreBLEU (Post, 2018) for BLEU6 and chrF7 and the python library for BLEURT.8
Previous work starting with Camgöz et al. (2018) does mainly report only BLEU scores, but they do not specify the BLEU variant used or the signature in sacreBLEU. Therefore, comparisons among systems might not be strictly fair.

heading: D Gloss-based vs SEM-based Systems’ Training Performance
text: Figure 3 shows the training evolution for a simple SLT system with no additional supervision (top), additional gloss supervision (middle) and SEM supervision (bottom) implemented in the Signjoey framework (Camgöz et al., 2020) and trained on PHOENIX-2014T. We use the best hypeparameters in Camgöz et al. (2020) and add our SEM supervision as a replacement of their recognition loss.
The three plots in Figure 3 include a red line at
6BLEU|nrefs:1|bs:1000|seed:16|case: mixed|eff:no|tok:13a|smooth:exp|version: 2.4.0
7chrF2|nrefs:1|bs:1000|seed:16|case: mixed|eff:yes|nc:6|nw:0|space:no|version: 2.4.0
8BLEURT v0.0.2 using checkpoint BLEURT-20.
translation quality BLEU=20 for reference. The first thing to notice is that both supervision methods reach the red line, but the one lacking any additional supervision lays behind. Second, we observe that the system with the additional SEM supervision reaches BLEU=20 earlier than the system with glosses: the gloss system needs 12k to finish and only 5k iterations are needed in the case of SEM. In both cases, we use early stopping with BLEU patience 7. Finally, notice that the gloss and SEM systems achieve the same translation quality but one does not need any data annotation with SEM.

heading: E Ablation Study on Sentence Length
text: Following the analysis of Section 5, we include the translation quality scores BLEU and chrF per sentence length.

heading: Future Work
text: In this work, we limited ourselves to existing visual feature extractors, in the future we plan to train a SEM-based visual feature extractor on SL datasets in order to get closer to our sem2text upper-bound and match gloss-based performance. Limitations
Our SL datasets cover American English and German. Sentence embeddings for these languages are good quality as lots of textual data is available
for pre-training. It remains to be studied how the quality of the embeddings affects the final translation quality. This is important for low-resourced languages, i. e. languages with limited amounts of monolingual text data but, to the best of our knowledge, no public sign language data set exists for them