Towards Privacy-Aware Sign Language Translation at Scale (2024)

Abstract: A major impediment to the advancement of sign language translation (SLT) is data scarcity. Much of the sign language data currently available on the web cannot be used for training supervised models due to the lack of aligned captions. Furthermore, scaling SLT using large-scale web-scraped datasets bears privacy risks due to the presence of biometric information, which the responsible development of SLT technologies should account for. In this work, we propose a twostage framework for privacy-aware SLT at scale that addresses both of these issues. We introduce SSVP-SLT, which leverages selfsupervised video pretraining on anonymized and unannotated videos, followed by supervised SLT finetuning on a curated parallel dataset. SSVP-SLT achieves state-of-the-art finetuned and zero-shot gloss-free SLT performance on the How2Sign dataset, outperforming the strongest respective baselines by over 3 BLEU-4. Based on controlled experiments, we further discuss the advantages and limitations of self-supervised pretraining and anonymization via facial obfuscation for SLT.

heading: 1 Introduction
text: Used by millions worldwide, sign languages play a crucial role in facilitating communication for many d/Deaf and hard-of-hearing individuals. Visual in nature, these languages make use of the co-articulated features of hands (i.e., finger positioning, shape, movement, palm orientation, etc.), body postures, gaze, mouth gestures, mouthings and facial expressions to convey meaning (Stokoe, 1980). Globally, there are more than 300 sign languages, each with their own grammar and vocabulary.2 American Sign Language (ASL) alone
1Code and data are available at https://github.com/ facebookresearch/ssvp slt.
2 https://www.un.org/en/observances/
sign-languages-day
is estimated to have more than half a million native users, ranking it among the most commonly used languages in the United States (Mitchell and Young, 2022).
Despite the prevalence of sign languages, they are still under-served by translation technology. Besides under-investment (Yin et al., 2021) and the inherent difficulty of SLT,3 another key explanation for this imbalance is the lack of sufficiently large, clean, and labeled parallel corpora. Current state-of-the-art SLT systems require detailed and time aligned annotations (Zhou et al., 2023; Uthus et al., 2023), which is not scalable, as annotating sign language data is a labour intensive task and can only be done by proficient signers.
We argue that a promising solution to SLT’s data scarcity is to utilize publicly available unannotated sign language data.4 In other domains of computer vision and NLP, a common practice is to pretrain on large-scale unannotated web datasets and later finetune on curated, task-specific datasets (Devlin et al., 2019; Radford et al., 2018, 2019; Raffel et al., 2020; He et al., 2022). This practice is largely unexplored in the SLT domain and comes with additional challenges. In particular, moving
3Results of the WMT 2023 SLT task evince this difficulty; the best system only achieved ∼1 BLEU (Müller et al., 2023).
4For example, Uthus et al. (2023) filtered their YoutubeASL dataset from ∼88K to 11K videos based largely on the availability and quality of English captions.
8624
to large-scale sign language processing makes it increasingly difficult to control the composition of the training data. Because sign language videos typically feature faces and upper bodies and thus are biometrically identifying, such research may exacerbate privacy risks. Hence, developing sign language technologies responsibly requires us to account for these risks and explore techniques to protect privacy.
In this work, we study the effectiveness of selfsupervised video pretraining for SLT, under consideration of the aforementioned privacy risks. We first propose a generic, scalable and privacyaware two-stage framework for SLT, summarized in Table 1. We introduce SSVP-SLT (SelfSupervised Video Pretraining for Sign Language Translation), an implementation of this framework consisting of two or optionally three stages: pretraining a continuous sign language encoder via masked autoencoding (MAE; He et al., 2022) on anonymized video, then optionally bridging the modality gap via CLIP-style video-text pretraining (Radford et al., 2021), and finally training an SLT system via supervised finetuning using extracted features from the pretrained model. Our best performing models achieve 15.5 BLEU finetuned and 7.1 BLEU zero-shot on the How2Sign dataset (Duarte et al., 2021), surpassing SOTA in both settings by over 3 BLEU while using data anonymized via facial obfuscation. We also introduce a new ASL-to-English SLT benchmark dataset, DailyMoth-70h, consisting of ∼70h of continuous signing in native ASL. We then evaluate the downstream performance impact and discuss the benefits and limitations of facial blurring to achieve anonymization. Through controlled ablation studies of SSVP-SLT, we identify what factors contribute to a strong pretraining and finetuning recipe. We conclude by discussing opportunities and challenges of self-supervised pretraining for sign language processing.

heading: 2 Background and Related Work
text: Gloss-free SLT Glosses are a way of representing individual signs into a written form. Being monotonically aligned to signs, they can be a useful medium between sign and spoken languages. Most SLT approaches to date rely on them (Chen et al., 2022a,b; Zhang et al., 2023). The task of predicting glosses from continuous signing is typically performed via gloss supervision either
jointly or in a cascaded approach with supervised SLT finetuning (Camgöz et al., 2018, 2020).
However, glosses are also considered an incomplete and inaccurate representation of sign language (Camgöz et al., 2020; Müller et al., 2023). Furthermore gloss annotation is a labour intensive task. Due to these constraints, there is a growing body of research on gloss-free SLT. Most such approaches incorporate techniques aimed at reducing the modality gap, such as training the visual encoder via sign spotting (Tarrés et al., 2023; Shi et al., 2022),5 adding inductive bias in the attention mechanism (Yin et al., 2023), using conceptual anchor words (Lin et al., 2023), or performing visual-language pretraining (Zhou et al., 2023). Uthus et al. (2023) also benefit from a pretrained text model (T5; Raffel et al., 2020). Similar to Zhou et al. (2023), we also leverage language supervision to reduce the modality gap, albeit in conjunction with self-supervised video pretraining.
Sign Language Video Anonymization Sign language videos typically feature signers’ faces, which convey essential linguistic information. However, in virtual domains, particularly in spaces involving the discussion of sensitive topics, exposing one’s face (and identity) may lead to various forms of personal risks. Such exposures could even lead to harm associated with professional or insurance discrimination. For these reasons, the d/Deaf and hard-of-hearing community has long expressed interest in anonymization and privacy protection techniques for sign language content (Lee et al., 2021), and sign language video anonymization has, in recent years, become an active area of research (Isard, 2020; Xia et al., 2024).
For general-domain images and videos, simple anonymization techniques such as facial obfuscation via overlaying or blurring are widely used and accepted (Frome et al., 2009; Yang et al., 2022). In the sign language domain, such techniques may be inadequate due to the loss of information in the facial region (Lee et al., 2021). More specifically, in signed language, non-manual features such as mouthing, eyebrow and head movements are used extensively to enrich grammar. Certain signs with similar manual features are only disambiguated through mouth morphemes. Moreover,
5While Tarrés et al. (2023) do not explicitly mention the use of sign spotting, they rely on features extracted from an I3D model (Carreira and Zisserman, 2017) by Duarte et al. (2022), who used an iterative sign spotting technique.
for SLT while the pretrained SignHiera is kept frozen (❄). The input video in the second stage can be unblurred.
facial expressions are often used to indicate emphasis, negation, a question, etc. (Baker-Shenk, 1985; Valli and Lucas, 2000; Neidle et al., 2000).
Recent work has focused on anonymizing sign language content via avatars (Tze et al., 2022; Lee et al., 2021) and transferred or synthetic human appearances (Saunders et al., 2021; Xia et al., 2022, 2024). While promising, these approaches are nascent and we are unaware of studies that determine to what extent models can learn to recover or disambiguate obfuscated linguistic information from context. That being said, human studies suggest that signers struggle to comprehend content anonymized in such a way (Lee et al., 2021).
Lacking an obvious alternative, in this work we return to the relatively straightforward technique of facial blurring. Despite its limitations, we demonstrate that blurring can raise privacy protection with little performance degradation.6 This, we argue, can facilitate large-scale video anonymization when applied to publicly available sign language data.

heading: Masked Autoencoding for Video and Beyond
text: Following its success as a self-supervised learning paradigm in the image domain (He et al., 2022), MAE has been widely applied in other areas, including audio (Huang et al., 2022), language (Rust et al., 2023), and video (Feichtenhofer et al., 2022; Tong et al., 2022; Wang et al., 2023). Considering that MAEs have been shown to be capable of acquiring both language and basic video understanding from pixels alone, it is conceivable that high-quality sign language representations can be learned directly from RGB video data via MAE, given enough data. Recently, Sandoval-Castaneda et al. (2023) explored MAE among other self-supervised learning techniques in the context of isolated sign language recogni-
6In Appendix A we discuss issues with pose landmarks, often promoted as a privacy-preserving alternative to video.
tion (ISLR) and found it to be highly useful. MAE has, however, not yet been successfully applied to SLT. In SLT, videos are much longer, and learning high-quality representations requires models to capture long-range spatiotemporal dependencies. Our usage of MAE, or self-supervised pretraining in general, therefore stands in contrast to recent SLT methods, gloss-based and gloss-free methods alike, which instead fully rely on supervised training that requires annotated captions (Zhou et al., 2023; Tarrés et al., 2023; Lin et al., 2023; Uthus et al., 2023).

heading: 3 Generic Framework
text: We first outline a generic, scalable and privacyaware two-stage transfer learning framework for SLT (See Table 1).
In Stage I, we train a model, with the goal to learn high-quality continuous sign language representations, via self-supervised learning. The data used at this stage is always anonymized. We make no assumptions on how the data may be anonymized, i.e. face blurring as discussed in §2, or more sophisticated methods such as using synthetic appearances.
In Stage II, we finetune the model from the first stage in a supervised manner using a smaller and hand-curated parallel dataset. Ideally, the finetuning dataset, being manageable in size, can be unanonymized after gaining explicit consent from signers in the data to minimize information loss.

heading: 4 Method
text: The base implementation of our framework is designed as a two-step approach, termed SSVP-SLT. We provide a high-level overview in Fig. 1.
Self-Supervised Video Pretraining (MAE) We first aimed to pretrain a capable sign language encoder on video data alone—no gloss, pseudogloss, or spoken-language text annotations—
allowing us to leverage large amounts of unannotated sign language video in the future, alleviating the data scarcity issue in SLT training.
Our sign video encoder, SignHiera, builds on Hiera (Ryali et al., 2023), a vision transformer that combines a hierarchical architecture, shown to be crucial for learning phonetically meaningful sign representations (Sandoval-Castaneda et al., 2023) with masked autoencoding (MAE), a widely used self-supervised learning paradigm (He et al., 2022). Its hierarchical architecture also makes Hiera more efficient to train than other MAE-based video transformers such as VideoMAE (Tong et al., 2022; Wang et al., 2023) or MAE-ST (Feichtenhofer et al., 2022).
Hiera embeds a video into a sequence of spatiotemporal tokens. A large percentage of tokens is randomly masked, while the rest is passed through a hierarchical transformer stack with several pooling operations. The decoder receives fused multiscale features extracted before each pooling operation and processes them via a lightweight transformer stack. A final linear projection yields pixel logits. The loss is computed as the normalized mean squared error between the original and predicted pixel values of the masked tokens.
SignHiera is initialized from the original HieraBase-16×224 checkpoint pretrained on Kinetics400 (Kay et al., 2017). In order to capture longerrange spatio-temporal dependencies in signed utterances, we increased the clip length from 16 to 128 video frames, leading to an 8× sequence length, and accordingly resized and reinitialized the position embeddings. We further added attention masking to accommodate shorter videos with temporal padding and added a third Q-pooling operation after the last encoder stage to save GPU memory. We trained with a masking ratio of 0.9.
Supervised SLT Finetuning The translation network is an encoder-decoder transformer model (Vaswani et al., 2017). Our default configuration uses a pretrained T5-v1.1 (Raffel et al., 2020; Shazeer, 2020), following Uthus et al. (2023). We also experimented with BART (Lewis et al., 2020) and Tarrés et al. (2023)’s setup, training a ∼10M parameter transformer from scratch.
The only difference from a text transformer is that our translation network takes in video features extracted from the pretrained SignHiera. We used SignHiera’s final intermediate representations, which are of size RB× T 2 ×H 32 ×W 32 ×D, where
B is the batch size, T=128, H=W=224 is the input video size, and D=768 is the feature size. We mean-pooled over the spatial dimensions to obtain feature vectors of size RB× T 2 ×D. Videos shorter than 128 frames were padded for encoding, and the padding was then removed from the extracted features. For longer videos, we used a sliding window with stride T
2 and concatenated the result-
ing features. A linear projection Wproj ∈ RD×D ′ mapped the extracted features to a sequence of size RB×S×D ′
, with S being the sequence length of the extracted features and D′ the transformer’s hidden size. This sequence was processed by the transformer as usual (Vaswani et al., 2017).
Adding Language-supervised Pretraining We also experimented with extending SSVP-SLT with a language-supervised pretraining (LSP) step to bridge the modality gap between the input videos and text translations. Bridging this gap may improve gloss-free SLT performance, as discussed in §2. In what follows, we refer to SSVP-SLT with an additional LSP stage as SSVP-SLT-LSP. The LSP stage fits in between the self-supervised MAE pretraining and the supervised SLT finetuning stage. Since language supervision requires annotations, the LSP stage should be considered a part of stage II in our generic framework.
Our LSP approach is illustrated in Figure 2. We first initialized a CLIP model (Radford et al., 2021) with our MAE-pretrained SignHiera encoder as the vision tower and a pretrained T5v1.1 encoder as the text tower. We then jointly trained the CLIP model via contrastive video-text pretraining and the SignHiera model via MAE. The goal is to help the SignHiera encoder, which is involved in both tasks, learn strong continuous sign representations grounded in the target
modality (text). The videos were masked with a 90% ratio even for computing the contrastive loss, which is similar to FLIP (Li et al., 2023), and enabled end-to-end training by drastically reducing the memory footprint. The two losses (MAE and contrastive) were balanced via GradNorm (Chen et al., 2018), which helped stabilize training, compared to using fixed loss weights.7

heading: 5 Experimental Setup

heading: 5.1 Datasets
text: Youtube-ASL (YT) We used Youtube-ASL (Uthus et al., 2023), the largest available ASL training dataset with ∼1000h of in-the-wild video and over 2500 signers, both during pretraining and supervised finetuning.8 We used a version in which all signers’ faces are blurred for privacy.
How2Sign (H2S) We also used How2Sign (Duarte et al., 2021), an ASL dataset with ∼80h of video and nine different signers in a green screen studio setting, for pretraining, finetuning, and downstream evaluation of our SLT models. Again, we used a version with blurred faces only.
DailyMoth-70h To isolate the impact of face blurring during pretraining and finetuning on SLT performance, we relied on a new dataset, which we name DailyMoth-70h. This dataset contains over 70h of video of a single signer from the ASL news page TheDailyMoth and was obtained through a license agreement with TheDailyMoth’s host.9 We used both unblurred and blurred dataset versions and report dataset statistics in Appendix B.

heading: 5.2 Training and Evaluation Protocols
text: We briefly describe our training and evaluation protocols. The full configurations for pretraining and SLT training are listed in Appendix C.1.
Face blurring We used an internal face blurring software and ensured its reliability on the YT and H2S datasets via a combination of automatic and manual verification techniques. Example frames sampled from two blurred videos from
7In contrast to Zhou et al. (2023), we did not jointly train the text decoder as doing so did not improve performance in preliminary experiments and led to training instabilities.
8For the lack of a readily available larger, unannotated dataset, Youtube-ASL fits both dimensions of our framework: the large, publicly-available, unannotated dataset and the curated, parallel dataset.
9 https://www.dailymoth.com/
the DailyMoth-70h data are shown in Appendix Figure 5.
MAE pretraining We largely follow the Hiera pretraining recipe from Ryali et al. (2023). In our default configuration, we trained for 800 effective epochs (accounting for repeated sampling as in Feichtenhofer et al. (2022)) with the AdamW optimizer (Kingma and Ba, 2015; Loshchilov and Hutter, 2019). The learning rate was set to 8e−4 with linear warmup over 120 epochs and cosine decay to 1e−5. The batch size was 2 × 128, with 2 being the repeated sampling factor. Similar to Zhou et al. (2023), we employed video data augmentation via random cropping, horizontal flipping, and RandAug (Cubuk et al., 2020). We used a 128 × 2 temporal sampling strategy, i.e., sampling 128 frames with a stride of 2, which fully accommodates ∼85–95% of videos in the data.
SLT finetuning When finetuning only on How2Sign or DailyMoth-70h, we closely followed the setup of Tarrés et al. (2023), training a ∼10M parameters transformer from scratch; see Appendix C.1 for more details. For How2Sign, we reused their lowercase unigram tokenizer (vocab size 7K). For DailyMoth-70h, we trained a cased tokenizer (unigram, 7K vocab size), which we found to work better due to the large proportion of named entities in the data.
When finetuning on Youtube-ASL, as we needed a model with more capacity we relied on a pretrained T5-v1.1 with default tokenizer, following Uthus et al. (2023). We trained for up to 100 epochs with early stopping, batch size 512 and AdamW with peak rate 5e−4, linear warmup over two epochs and cosine decay to 1e−7. We used dropout of 0.3 and 0.2 label smoothing in the cross-entropy loss. We did not use video data augmentation unless specified otherwise.
Language-supervised pretraining We performed 200 epochs of LSP with a batch size of 512 on top of 600 epochs of MAE-only pretraining. We did not use repeated sampling, which is incompatible with the contrastive loss. We again used AdamW, warming up to a learning rate of 1e−4 over ten epochs, followed by cosine decay to 1e−6. The GradNorm optimizer has a one epoch warmup, a peak learning rate of 1e−2, and decays to 1e−4. Data augmentation and temporal sampling are the same as for MAE pretraining.

heading: METHOD BLUR FT DATA BLEU-1 BLEU-2 BLEU-3 BLEU ROUGE-L BLEURT
text: Evaluation We used beam search with 5 beams and no length penality. We evaluated every epoch when finetuning on How2Sign or Dailymoth-70h and every 500 steps for Youtube-ASL. We kept the checkpoint with the highest validation BLEU4 and evaluated it on the respective test set.
Notation Below, we use superscript and subscript to indicate the pretraining dataset and number of epochs, respectively. For instance, SSVPSLT YT+H2S800 refers to 800 epochs of MAE pretraining on Youtube-ASL and How2Sign. For SSVPSLT-LSP, 600→200 denotes 600 epochs of MAE pretraining followed by 200 epochs of LSP.

heading: 5.3 Baselines
text: Lin et al. (2023) propose to bridge the visual and text modalities via contrastive anchoring of encoded visual features to embeddings of conceptual words in the target sequence. Tarrés et al. (2023) is the SOTA on How2Sign without additional SLT data, training a 6+3 layer transformer from scratch on features from an I3D model (Carreira and Zisserman, 2017). The I3D model was first pretrained on Kinetics (Carreira and Zisserman, 2017) and BOBSL (Albanie et al., 2021), and finetuned on How2Sign for sign language recognition using annotations generated via iterative sign spotting (Duarte et al., 2022). Uthus et al. (2023)
is the current SOTA on How2Sign, and finetunes a pretrained T5-v1.1-Base model for SLT directly on pose landmarks extracted from YouTube-ASL and How2Sign videos.

heading: 5.4 Evaluation Measures
text: We report BLEU via SacreBLEU (Papineni et al., 2002; Post, 2018).10 We also report ROUGE-L (Lin, 2004) and BLEURT (Sellam et al., 2020) from the BLEURT-20 checkpoint, which has been shown to correlate well with human judgments.

heading: 6 Results and Discussion
text: Comparison against the state-of-the-art We present our main results in Table 2. Our best models significantly improve over the previous 12.4 BLEU state-of-the-art by Uthus et al. (2023). SSVP-SLT achieves 14.7 and 14.6 BLEU when pretraining on YT and YT+H2S respectively. Our best overall model, utilizing SSVP-SLT-LSP, achieves 15.5 BLEU, a 3.1 point improvement over the baseline. When pretraining and finetuning on YT only, we also observe a 3.1 BLEU improvement (4.0 vs. 7.1) over the previous SOTA in the zero-shot setting. These results demonstrate
10 nrefs:1|case:mixed|
eff:no|tok:13a|smooth:exp|version:2.3.1
the overall effectiveness of SSVP-SLT and, more broadly, self-supervised pretraining for SLT.
Pretraining on YT+H2S performs almost the same as training on YT only, with the YT-only models even sometimes performing best. Given the distributional gap between the datasets (in-thewild YT vs. studio H2S video) and the fact that the YT+H2S models consumed more data, this finding is somewhat surprising. While this may be due in part to randomness in the training dynamics, it could also mean that sufficient finetuning can compensate for not accessing the H2S data at pretraining, presumably because the pretraining set is sufficiently large and diverse. This encouraging result suggests that we can pretrain on large data independently of knowing what our finetuning and inference dataset will be—a crucial requirement for practical SLT applications.
We find that YT data is beneficial both for pretraining and finetuning, which emphasizes the importance of training on large and diverse data and suggests that we can expect further gains from scaling to large public unannotated video.
Finally, we find that bridging the modality gap via language-supervised pretraining yields a 1.2 BLEU improvement over its MAE-only counterpart. Given enough annotated data, the technique can be employed independently of self-supervised pretraining at little extra cost.

heading: BLUR BLEU ROUGE-L BLEURTPRETRAIN SLT
text: What’s the effect of blurring? We isolate the impact of facial obfuscation via blurring on SLT performance by training SSVP-SLT models on DailyMoth-70h with and without blurring during pretraining and SLT training. We report the results in Table 3. As expected, performance is best when not blurring (28.8 BLEU) and worst when blurring at finetuning time (28.1 and 27.5 BLEU). Crucially, some performance can be recovered after pretraining on blurred data when performing SLT on unblurred data (28.2 BLEU), validating
our proposed framework (see Table 1).11 This means we can pretrain in a privacy-aware manner without sacrificing too much performance.
How long should you pretrain? The sign language MAE task is intricate. The model first needs to abstract away surface-level information such as background and signer appearance. It then needs to implicitly acquire an understanding of ASL, capturing long-range dependencies in the video. It is therefore worth investigating basic scaling properties. In Figure 3, we show downstream SLT performance over the course of pretraining. Similar to Ryali et al. (2023) and He et al. (2022), we observe consistent downstream improvements as pretraining progresses, suggesting that the models are not overfitting to the training data even after extensive pretraining. These results underscore the task’s effectiveness and indicate that further scaling would likely yield additional gains.

heading: H2S

heading: YT + H2S
text: 11We hypothesize that even more performance could be recovered if the SignHiera video encoder was unfrozen during SLT training, allowing adaptation to the facial information.
Is encoding longer clips necessary? Increasing clip length is costly due to the poor scaling of global attention, raising the question of whether encoding longer clips is needed. Our results indicate that the answer is yes. Table 4 compares the performance of the original 16-frame Hiera with our 128-frame SignHiera. While 16-frame Hiera achieves non-trivial performance after 800 epochs, it is substantially outperformed by 800 epoch SignHiera (7.0 vs 5.0 BLEU for H2S and 14.3 vs 10.4 BLEU for YT+H2S). This may be partially explained by the fact that SignHiera sees up to 8× as much data every step. However, we also see that 800-epoch 16-frame Hiera lags far behind even a 100-epoch SignHiera (which has seen roughly the same number of tokens) when training on a large dataset (12.5 vs 10.4 BLEU in the YT+H2S setting). When training on less data (H2S), 16-frame Hiera is still worse than 400-epoch SignHiera (5.39 vs 5.0 BLEU, see Figure 3). Overall, this suggests that certain information cannot easily be acquired from shorter clips.

heading: MODEL PARAM PT BLEU ROUGE-L BLEURT
text: How to choose the text model? We investigate how the architecture and initialization of the text transformer affects performance. Table 5 compares pretrained and randomly initialized BART (Lewis et al., 2020), the English monolingual counterpart to mBART (Liu et al., 2020), which has previously been successfully adapted for German and Chinese sign languages (De Coster and Dambre, 2022; Chen et al., 2022a; Zhou et al., 2023), and T5-v1.1 as used by Uthus et al. (2023).
Overall, T5 outperforms BART, possibly due to larger capacity, but the gap is small. While it is worse to finetune a randomly initialized T5 model compared to the pretrained one (corroborating findings by Uthus et al. (2023)), for BART we find the opposite result. We conclude that whether text pretraining is helpful needs to be evaluated on a case-by-case basis. It may be worth investigating in the future whether an additional pretraining
or finetuning step may lead to better adaptation of the text model to sign language.

heading: AUG BLEU ROUGE-L BLEURT
text: Should we augment data at finetuning? Augmentation such as flipping, cropping, and RandAug may improve generalization, but it comes at a high storage cost at finetuning time, as the video features are extracted offline. Is it worth the cost? We compared using 60 epochs of augmented videos (a 60-fold increase in storage) with not using any augmentation. The results in Table 6 show that using augmentation yields a reasonable 0.4 BLEU gain, suggesting that augmention can be useful when storage is not a major concern.

heading: INITIALIZATION MAE CLIP BLEU ROUGE-L BLEURT
text: Are both pretraining objectives necessary? In §6, we saw that language-supervised video-text pretraining is highly effective when combined with MAE. Are both necessary? We compared pretraining for 200 epochs with either and both objectives, initializing from a 600-epoch SSVPSLT checkpoint, and also performed 200 epochs of CLIP-only pretraining from the original pretrained Hiera. The results in Table 7 show that removing either objective results in a performance drop. The drop is larger when removing MAE, indicating its continued importance after 600 epochs of MAEonly training. Initializing from the original Hiera results in very poor performance (2.1 BLEU), suggesting that language-supervised pretraining alone is not useful in our setting. Considering that language supervision has previously been shown to be effective in isolation (Zhou et al., 2023), this may be due to the FLIP-style masking and the fact
that we do not jointly pretrain T5. We also emphasize that language-supervised pretraining falls under stage II of our framework as it requires annotations; it can, therefore, only serve as an addition to self-supervised pretraining, but not a replacement.

heading: 7 Conclusion
text: Through controlled experiments, we studied the effectiveness of self-supervised pretraining for SLT while considering privacy risks. We introduce SSVP-SLT, a novel, scalable, and privacy-aware SLT method that leverages masked autoencoding on anonymized video. It achieves state-of-theart ASL-to-English translation performance on the How2Sign benchmark, outperforming the best previous models in the finetuned and zero-shot setting by over 3 BLEU.
Our results demonstrate the promise of selfsupervised learning to alleviate the data scarcity issue and further scale up sign language processing in the future. We found that video anonymization, even via simple techniques such as face blurring, has relatively little negative impact on downstream performance, further proving that we can build more proficient systems without neglecting important privacy concerns. We hope that this work, alongside the code and data we release, will spur future developments that benefit the d/Deaf and hard of hearing communities.

heading: Limitations
text: Compute Requirements Currently, SSVP-SLT requires access to substantial compute to train at the scale of Youtube-ASL (600K videos). This is primarily due to the high dimensionality of video data, exacerbated by the long clip length and information density in sign language content, which creates a data-loading bottleneck and increases the memory footprint, especially in combination with a transformer architecture. Our longest pretraining run in full precision (fp32) took approximately two weeks on 64 A100 GPUs. We believe that it will be important to drive down this cost in the future and make large-scale video pretraining more accessible. While many simple interventions such as mixed precision, gradient accumulation, and gradient checkpointing could drastically reduce the memory footprint, they usually come at the cost of training time or stability. In general, we note that this limitation is not unique to our approach but often not apparent due to training being
conducted on nearly 100× smaller datasets such as RWTH-Phoenix-Weather 2014T (7K videos; Camgöz et al., 2018).
Anonymization We rely on face blurring for video anonymization, which is known to incur a loss of linguistic information (see §2). In the future, it will be worth investigating more sophisticated methods, such as using synthetic appearances. Also, largely due to a lack of linguistic tools for continuous signing, we did not investigate what effects anonymization may have on the translations qualitatively. For instance, it would be instructive to know whether the model successfully disambiguates certain phonemes with similar manual features through context in the absence of facial information.
Languages Due to the availability of sufficiently large datasets for our pretraining experiments, we only experiment with American Sign Language and English, the two highest-resource signed and spoken languages. We aim to diversify this language selection in the future.

heading: Ethics Statement
text: Regarding performance, our models may contain demographic biases and underperform for certain race, gender, and age groups. For instance, even though the YouTube-ASL dataset (a dataset we used for pretraining and supervised finetuning) features over 2500 signers, the authors did not provide demographic details of these signers. Similarly, our DailyMoth-70h dataset includes only one signer (white, male, and early middle-aged). As such, it is unclear how our models perform for underrepresented users, who, aside from having diverse identities, may introduce different accents or regional variations of ASL that our models do not adequately capture. We call for future research on SLT to be more explicit about documenting demographic biases in their datasets and models.
Lastly, we emphasize that anonymization inherently does not offer any formal privacy guarantees in contrast to frameworks such as differential privacy (Dwork, 2006), which fundamentally comes at a (often substantial) cost in utility (Geng et al., 2020). As such, while our work (and the use of facial obfuscation in general) represents an important first step towards comprehensively protecting the privacy of signers, it should not be relied on in sensitive or high-stakes applications.

heading: Acknowledgements
text: We thank members of the Seamless Communication team at FAIR for helpful feedback and discussions throughout this project. We also thank Florian Metze and Amanda Duarte for their prompt assistance with our questions about the How2Sign dataset.

heading: A Pose landmarks vs RGB Video
text: Pose landmarks (e.g., from MediaPipe Holistic) are cited as a privacy-preserving alternative to RGB video for SLT (Moryossef et al., 2021; Uthus et al., 2023). While they may indeed offer benefits in terms of efficiency and generalization, we argue that they do not offer meaningful privacy protection either. For instance, using a sufficiently large number of facial landmarks that are estimated accurately results in what is essentially a scan of the facial geometry, a biometric identifier according to legislation like the Biometric Information Privacy Act (BIPA).12 Despite abstracting away shallow information about a person’s appearance, pose information could, therefore, be (mis)used in similar ways as de-anonymized video. Analogous to facial obfuscation in video, one could reduce the number of facial landmarks or add noise to them to hinder re-identification, but doing so also results in (arguably even more) loss of information.

heading: B DailyMoth-70h Dataset
text: We introduce DailyMoth-70h, a dataset containing over 70h of video with aligned English captions of a single native ASL signer (white, male, and early middle-aged) from the ASL news page TheDailyMoth.13 We obtained the data via a license agreement with the host of TheDailyMoth.
Download and License The fully selfcontained dataset is available under a CC BY-NC license at https://github.com/ facebookresearch/ssvp slt.
Statistics We provide detailed dataset statistics in Table 8 and Figure 4.
Purpose The dataset is fully self-contained and can therefore serve as a new benchmark for studying the task of single-signer translation (e.g., for building personalized systems). Furthermore, sign language translation involves overcoming several challenges such as generalizing over signers, their appearances and their signing styles as well as mapping spatio-temporal signed utterances to spoken words. DailyMoth-70h can be used to disentangle some of these challenges by eliminating the signer and style variances and allow researchers to
12 https://www.ilga.gov/legislation/ilcs/ilcs3.
asp?ActID=3004&ChapterID=57 13 https://www.dailymoth.com/
ablate their models more focused on the sign-tospoken language mapping.
Preprocessing We first segmented the raw videos into clips of ∼5.6 seconds on average based on their aligned English captions. Each entry in the SubRip subtitle (SRT) file, which maps video timestamps to captions, was chosen to be a distinct datapoint. Accordingly, example clips are often sentence fragments rather than full sentences.
Afterwards, the segmented video clips were automatically cropped such that the signer is approximately in the center of the frame and resized to 224 × 224 pixels. The preprocessed clips were saved in their native framerates, which are either 24 or 30 fps.
Next, many videos had burned-in captions which, if not removed, would reduce the translation task to a simple OCR task. We, therefore, used an off-the-shelf text detection model to identify burned-in captions in the videos, and blurred the captions conservatively. Although the blurring may be imperfect due to errors made by the text detector, this intervention should nevertheless solve the concern of models shortcutting the SLT task for the most part.
Finally, we split the data into training, validation, and test splits. The proportions (85% / 6% / 9%) were chosen to approximately match How2Sign. The validation and test examples were randomly sampled from the subset of videos posted after January 2022, which avoids data leakage from Youtube-ASL or OpenASL (Shi et al., 2022), both of which have cut-off dates before/during January 2022, into the DailyMoth-70h evaluation splits. The training examples were randomly sampled from the full range of dates (January 2021 to April 2023).

heading: C Reproducibility

heading: C.1 Model and Training Configurations
text: We report our pretraining configurations for SSVP-SLT in Table 9 and SSVP-SLT-LSP in Table 10. Our finetuning configurations are listed in Table 11 for Youtube-ASL (+ How2Sign) and Table 12 for How2Sign-only and DailyMoth-70h.

heading: C.2 Code
text: Our implementation uses Python 3.10 and PyTorch 2.0.1 (Paszke et al., 2019) compiled with CUDA 11.7. The code is available under a CC BY-NC license at https://github.com/ facebookresearch/ssvp slt.

heading: C.3 Hardware & Runtime
text: We ran our experiments on NVIDIA A100 80GB and V100 32GB GPUs. On Youtube-ASL (+ How2Sign), pretraining took ∼20 minutes (SSVPSLT) / 30 minutes (SSVP-SLT-LSP) per effective epoch on 64 A100 or 128 V100 GPUs. On How2sign or DailyMoth-70h, an effective epoch of SSVP-SLT pretraining took ∼3 minutes. Finetuning and evaluating T5 and BART on YoutubeASL (+ How2Sign) took ∼8 and 4 minutes, respectively, per epoch on 32 V100 GPUs. Training T5 was slower due to training in full precision as opposed to fp16 and using a smaller batch size with gradient accumulation. Finetuning and evaluating with Tarrés et al. (2023)’s setup on How2Sign or DailyMoth-70h took ∼1 minute per epoch on a single V100 GPU.

heading: D Qualitative Examples
text: In Table 13, we provide qualitative examples of our best-performing model (15.5 BLEU
on How2Sign), compared against the bestperforming models from Tarrés et al. (2023), Uthus et al. (2023), as well as the reference translations. The examples were picked from the How2Sign test split by Tarrés et al. (2023). Examples (3)–(5) are, anecdotally, more challenging than the average test example. We find that our model is mostly on-topic and matches the syntactic structure, although it can still struggle with repetitions and the mixing-up of signs. Our model’s failure patterns are more similar to Uthus et al. (2023)’s models—possibly a result of finetuning the same base model (T5-v1.1) on the same datasets (Youtube-ASL and How2Sign). For instance, in example (3), both models mistakenly predict the verb “feed” (and mispredict everything that comes after) but flawlessly match the syntactic structure of the reference translation. Overall, both baselines appear to exhibit a higher occurrence of (complete) mis-translation, which aligns with our quantitative results.

heading: Future Work
text: We call for future research on SLT to be more explicit about documenting demographic biases in their datasets and models. Lastly, we emphasize that anonymization inherently does not offer any formal privacy guarantees in contrast to frameworks such as differential privacy (Dwork, 2006), which fundamentally comes at a (often substantial) cost in utility (Geng et al. , 2020). As such, while our work (and the use of facial obfuscation in general) represents an important first step towards comprehensively protecting the privacy of signers, it should not be relied on in sensitive or high-stakes applications