SlowGuess/NeurIPS_2023_OCR huggingface dataset:
David Uthus, Garrett Tanzer, Manfred Georg
{duthus,gtanzer,mgeorg}@google.com
Machine learning for sign languages is bottlenecked by data. In this paper, we present YouTube-ASL, a large-scale, open-domain corpus of American Sign Language (ASL) videos and accompanying English captions drawn from YouTube. With  $\sim 1000$  hours of videos and  $>2500$  unique signers, YouTube-ASL is  $\sim 3x$  as large and has  $\sim 10x$  as many unique signers as the largest prior ASL dataset. We train baseline models for ASL to English translation on YouTube-ASL and evaluate them on How2Sign, where we achieve a new finetuned state of the art of 12.39 BLEU and, for the first time, report zero-shot results.
The primary bottleneck for machine learning research on sign languages is data. As minority languages used by historically marginalized Deaf/Hard of Hearing communities, sign languages lack the plentiful online resources that have facilitated modern machine learning advances [4, 44, 12]. This is compounded by the fact that sign languages have no standardized written form: mining the videos that do exist is more difficult than retrieval for spoken language text. For translation specifically, there is the added problem of finding spoken language captions that are aligned to corresponding sign language content, rather than a voiceover with its own timing. The result is that datasets tend to be constructed by recording new footage in a studio or curating videos from a small number of manually selected content creators, which limits variety.
In order to address these challenges, we present YouTube-ASL, a large-scale, open-domain corpus of American Sign Language (ASL) videos and accompanying English captions, primarily intended for ASL to English machine translation. We mined these videos from YouTube using a two-step process: first, we used automatic content-based annotations to identify potentially relevant captioned videos; and second, we used skilled human annotators to filter out videos with poor quality or misaligned captions. The result is a dataset with 984 hours of high-quality captioned video featuring  $>2500$  unique signers, which is  $\sim 3x$  as large as the largest prior ASL dataset [39] and has  $\sim 10x$  as many unique signers as any sign language dataset to date.
We train simple baseline models for sentence-level ASL to English translation on YouTube-ASL by embedding MediaPipe Holistic landmarks [26, 16] into the T5 language model [33]. Because YouTube videos may be removed over time and therefore cannot form a stable test set—and for comparison to prior work—we evaluate on a standard benchmark, How2Sign [13]. Borrowing from trends in mainstream machine learning [32, 15, 10], we provide not just finetuned but also zero-shot results to test out-of-domain generalization. We achieve a new finetuned state of the art of 12.39 BLEU (vs. the prior SOTA of 8.03 [40]), and for the first time report a zero-shot score, 3.95 BLEU.
<sup>1</sup>Here, “zero-shot” refers to evaluation on an independently constructed benchmark without any kind of domain adaptation. This is different from the use of “zero-shot” in machine translation for transfer to unseen language pairs, or “zero-shot” in prompting for prompts without in-context examples.
Table 1: Summary statistics for different sign language translation datasets. See Section 3.3 for details on how these statistics were derived for YouTube-ASL.
Name	Language	Vocab.	# Hours	# Signers	Source
RWTH-PHOENIX-2014T [5]	DGS	3K	11	9	TV
BOBSL [2]	BSL	77K	1447	39	TV
SWISSTXT [7]	DSGS	-	88	-	TV
VRT-RAW [7]	VGT	-	100	-	TV
CSL-Daily [47]	CSL	2K	23	10	Lab
KETI [22]	KVK	419	28	14	Lab
Public DGS Corpus [18]	DGS	-	50	-	Lab
SP-10 [42]	various	17K	14	79	Web
AfriSign [17]	various	20K	152	-	Web
How2Sign [13]	ASL	16K	79	11	Lab
OpenASL [39]	ASL	33K	288	220	Web
YouTube-ASL (ours)	ASL	60K	984	>2519	Web
We publicly release the YouTube-ASL video IDs. We hope that YouTube-ASL will be useful for general sign language pretraining, as well as downstream tasks such as ASL to English translation and caption alignment—both in the near term to aid in the construction of larger sign language datasets, and eventually to improve accessibility for the Deaf/Hard of Hearing community.
In this section, we review prior sign language translation datasets and methods for translation from sign languages to spoken languages.
Table 1 shows statistics on different sign language translation datasets. There are three main sources for sign language data: ad hoc recorded footage, interpreted TV broadcasts, and online video sharing platforms.
In the first category are datasets that manually recruit signers and record them performing translations of desired phrases, either in a lab setting or with a camera on their personal device. These datasets tend to be small and feature few signers for logistical reasons, and may have exhaustive annotations because the small size of the dataset makes it feasible. This includes datasets such as CSL-Daily [47], with phrases related to daily life in Chinese Sign Language; KETI [22], with phrases related to emergency situations in Korean Sign Language; Public DGS Corpus [18], with elicited dialogues in German Sign Language; and How2Sign [13], with "How To" instructional monologues translated into American Sign Language.
In the second category are datasets that collate interpreted TV programs from a collaborating national broadcaster. These datasets tend to be larger than newly recorded ones, but often use a small number of non-native interpreters and lack fine-grained caption alignment (because the supervision comes from the spoken language audio track). This includes datasets such as RWTH-PHOENIX-2014 [5], with weather forecasts interpreted into German Sign Language; SWISSTXT [7], with news/weather programs interpreted into Swiss German Sign Language; VRT [7], with news programs interpreted into Flemish Sign Language; and BOBSL [2], with BBC programs in many domains interpreted into British Sign Language. At 1447 hours, BOBSL is the largest sign language translation dataset to date (including the present work), but has only 39 signers and speech-aligned subtitles, vs. YouTube-ASL's  $>2519$  signers and sign-aligned captions—though the two datasets are complementary because they are for different languages.
In the third category are datasets that curate content from online video sharing platforms. In prior sign language translation datasets, this content is drawn from a small number of manually selected channels.
$^{2}$ https://github.com/google-research/google-research/tree/master/youtube_asl
This includes datasets such as SP-10 [42], with example sentences from an online multilingual sign dictionary; AfriSign [17], with translated Bible passages hosted on the Jehovah's Witnesses website; and OpenASL [39], with videos from three YouTube channels: DailyMoth, Sign1News, and the National Association of the Deaf. OpenASL is the largest prior ASL dataset and closest work to YouTube-ASL: the key difference is that YouTube-ASL is constructed with open-ended mining from automatic tags, rather than manual channel curation. OpenASL is largely a subset of YouTube-ASL, which—by utilizing the long tail of channels—is  $\sim 3x$  as large and has  $\sim 10x$  as many unique signers.
There are several datasets for easier tasks than translation, like isolated sign recognition and finger-spelling recognition, that mine from the web by ambiguous means. MS-ASL [21], WLASL [24], ChicagoFSWild [37]/ChicagoFSWild+ [38], CISLR [19], and Indian-SL [35] are word-level datasets mined from YouTube, sign language-targeted sites like ASLU and ASL-LEX, or other unnamed video sharing platforms. These works do not specify how they retrieved their videos, so it is possible that they used a similar automatic tagging approach to YouTube-ASL, albeit on a more limited scale.
2.2 End-to-End Sign Language Translation
Originally, sign language translation approaches operated on glosses, linguistic annotations that represent individual signs, or cascaded translation through glosses as an intermediate step, like speech to text translation often cascades through speech recognition. More recently, due to a variety of deficiencies in glosses and lack of widespread gloss data, the field has shifted to end-to-end modeling with encoder-decoder Transformers, starting with Camgoz et al. [5].
The two main classes of approaches are those that take learned video embeddings as input [6, 40, 28] (via video encoders, primarily I3D [9], pretrained on tasks such as isolated sign recognition), and those that take estimated pose landmarks as input [28] (such as MediaPipe [26] or OpenPose [8]). Some works achieve modest gains given constant data with architectural tweaks like treating different cues in the input video (hands, face) differently [46, 43]. It is unclear to what extent these techniques are necessary or beneficial on larger datasets. Other works seek to benefit from transfer from spoken language or other sign language data [11, 45, 17]. All of these works train and evaluate on splits derived from the same underlying continuous sign language corpus (different datasets across papers), and sometimes multiple such datasets independently in the same paper. In contrast, we train on YouTube-ASL using an uncomplicated approach and evaluate on How2Sign, reporting both finetuned and zero-shot results to get a more robust understanding of our model's state-of-the-art performance.
3 The YouTube-ASL Corpus
YouTube-ASL is a corpus of American Sign Language (ASL) videos with accompanying English captions drawn from YouTube. Video sharing platforms like YouTube are appealing sources of sign language data because they host swaths of diverse content that are more broadly representative of real world conditions than studio footage is. Of course, much of this data is irrelevant or low-quality, so it is imperative to develop cost-effective ways to sift through it.
We used a two-step pipeline to construct the corpus: first, retrieval using automatic content-based annotations, and second, filtering by skilled human annotators at a per-video level. This automatic retrieval step represents a departure from prior continuous sign language corpora and brings us closer to mining approaches from mainstream machine learning.
3.1 Automatically Retrieving Candidate Videos
As described previously in Abu-El-Haija et al. [1], the YouTube video annotation system associates machine-generated tags with each video in the form of Knowledge Graph entities, which are based on the video's metadata, context, and content signals. We retrieved listed public videos tagged as being related to sign language generally or American Sign Language specifically, as of January 2022.<sup>3</sup> This automatic tagging step, while having higher recall than prior works, was flawed in that it was not aware of sign language in the video content itself—to be expected due to the limited nature of current sign language processing. This means that, for example, videos in sign language that do not explicitly mention sign language in the content or context were unlikely to be discovered. This failure
Some video IDs may have been removed from this set over time due to video deletions.
mode was most salient for press conferences with simultaneous interpreters, which tend not to have well-aligned captions anyway.
Given these retrieved videos, we drilled down on those with user-generated captions—i.e., captions that were manually uploaded rather than automatically derived from speech—because speech-derived captions are not tightly aligned with signed content. As a heuristic filtering step, we automatically removed videos with duration  $< 10$  seconds or  $>5$  hours, width  $< 480$  pixels or height  $>360$  pixels, and frame rate  $< 15$  fps or  $>60$  fps. We arrived at these values through an iterative mining and auditing process, so that we could reduce annotator labor spent on irrelevant videos without excluding too many relevant ones. From inspection, the heuristic excluded a negligible amount of desirable videos. The one class of useful videos one might expect this to exclude, short isolated sign videos as used by MS-ASL [21] and WLASL [24], tends to have the label in the video title or description rather than captions, so removing videos under 10 seconds does not have a substantial impact. Videos that were over 5 hours were often either live interpreted broadcasts (which did not have aligned captions) or not sign language (e.g., corrupted videos or mostly static content for hours on end).
Finally, we used off-the-shelf person detection tools to exclude videos where none of the captions corresponded to spans with exactly one person present in the video. We limit the scope of our efforts to signing monologues due to the challenges of modeling conversations between multiple signers.
The result was a list of 88,002 candidate videos that might contain ASL with high-quality captions.
3.2 Identifying High-Quality Videos with Skilled Human Annotators
While some smaller datasets like How2Sign [13] use annotators to manually align all captions, this becomes prohibitively expensive for larger datasets. For this reason, OpenASL [39] and BOBSL [2] use annotators to correct only their validation and test sets. We take a coarser-grained approach to annotations but apply it to our entire list of 88,002 candidates: we use humans to identify videos that are roughly suitable and include them in our corpus without modification.
To do so, we hired 3 native ASL signers with English proficiency to serve as annotators. The annotators used a bespoke internal tool that would display a given YouTube video and present label options. In order to save time, the annotators were able to mark that their labels held for an entire channel of videos rather than each video individually. Therefore it is possible that certain videos in the corpus are channel outliers and do not meet quality standards, but generally large channels have consistent quality. Each video was labelled by only one annotator unless they brought it up for wider discussion.
Through an iterative process involving written instructions, virtual meetings (through an ASL interpreter or signing project members), and escalations by email for edge cases, we aligned on standards for when to accept a video into the corpus. Some of the reasons for exclusion include: the video's captions do not exclusively correspond to signing; the video is in a sign language other than ASL; the video's captions do not correctly translate the ASL; and the captions are poorly aligned<sup>4</sup>. Notably, in order to increase the size of the corpus, we chose to include videos across all skill levels and signing styles, as long as they were comprehensible to an ASL user and correctly captioned. This variety is beneficial for sign language recognition tasks, where models should be able to understand all signers, but may limit the corpus's usefulness for generation tasks, where consistency and controllability are important.
The result was a list of 11,093 videos whose captions are generally well-aligned English translations of signed ASL content.
3.3 Corpus Statistics
The final, human-filtered YouTube-ASL corpus consists of 11,093 ASL videos with 984 total hours of footage. This is  $\sim 3\mathrm{x}$  the size of OpenASL [39], the largest prior ASL dataset, but smaller than BOBSL [2], a British Sign Language dataset. See Table 1 for a comparison between the high-level
For caption alignment, we are targeting captions that start and stop in close proximity to the signing, and the timings appear to have been annotated based on the video rather than a voice-over. Within a sentence, which may consist of multiple captions, there may be but is not necessarily an alignment because the syntax of ASL is different from English.
Table 2: Statistics on the distribution of captions and videos in the YouTube-ASL corpus.
Number of captions	610,193
Caption length (Average / 90th percentile, in characters)	48.9 / 88.0
Caption length (Average / 90th percentile, in words)	8.8 / 16.0
Caption duration (Average / 90th percentile, in seconds)	4.8 / 8.76
Video duration (Average / 90th percentile, in seconds)	318.95 / 675.80
Figure 1: Distribution of caption and video durations. For the video duration graph, we omit 27 videos whose duration exceeds 3600 seconds (between 3610 and 9017 seconds).
attributes of YouTube-ASL and prior sign language translation datasets, including total number of hours.
These videos are paired with 610,193 English captions, with a total duration of 813 hours. See Table 2 for statistics on the distribution of captions, as well as Figure 1 for visualizations. The average caption length (8.8 words) and duration (4.8 seconds) are relatively short, which reflects that sentences may be split across multiple captions. We computed vocabulary size by counting the number of distinct strings between whitespace or punctuation across all captions. It is important to keep in mind that in addition to the signing itself, these videos' captions vary in style, literalness of translation (whether the content was originally produced in ASL and translated, or translated into ASL from these captions), spelling/grammar correctness, and more. This degree of variability is difficult to quantify in comparisons between datasets.
We use the number of unique channels, 2519, as an approximate lower bound for the number of unique signers in the dataset: some channels may feature many signers, and some signers may appear across multiple channels. Note that with this method, OpenASL [39] would be estimated to have 3 signers, while its authors reached a count of 220 signers using more fine-grained methods. Even this likely underestimate is  $\sim 10x$  the count of any individual sign language dataset to date.
Figure 2 shows the distribution of videos per channel, for channels with at least 20 videos. There are a few channels with many videos—in particular, the two largest channels are the same news channels featured in OpenASL—and then a long tail of channels with fewer videos. This means that the bulk of new footage present in YouTube-ASL but not OpenASL comes from relatively small channels, which helps variety. See Figure 3 for a sense of the distribution of (machine-annotated) topics across videos: they seem more diverse than prior datasets from video sharing platforms but still shaped by typical YouTube use cases, compared to BOBSL's more topic-balanced BBC programming.
In order to demonstrate the potential of YouTube-ASL, we consider a simple method for sentence-level machine translation from ASL to English built using off-the-shelf components. We use a deliberately barebones approach to avoid introducing inductive bias that helps in more limited settings but becomes harmful with scale.
Figure 2: Distribution of videos per channel for channels with at least 20 videos.
Figure 3: A selection of high-level topics, with the number of YouTube-ASL videos automatically tagged as related to them. Note that a single video can be tagged with more than one topic.
4.1 Preprocessing
For our target English outputs, we use the raw captions from YouTube-ASL. Each training example is clipped to the boundaries of a single caption. We filter out captions with length  $>300$  characters or duration  $<200\mathrm{ms}$  or  $>60\mathrm{s}$ , which tend to be malformed, and any captions corresponding to video spans where exactly one person is not present. We do not lowercase the captions or apply any other kind of text normalization.
For our sign language inputs, we use MediaPipe Holistic landmarks [26, 16], rather than raw video. Sign language models that use pose-based inputs have a history of underperforming those that operate on learned video embeddings [21, 27]; it is unclear to what extent this is due to the information bottleneck in the (imperfectly predicted) pose representation, vs. availability of higher quality pretrained video encoders than pretrained pose encoders. Pose inputs offer some benefits like computational efficiency and privacy.
MediaPipe Holistic is a lightweight model that predicts 532 3D landmarks (in x-, y-, and z- image-space coordinates) for the hands, pose, and face of a single human in video footage. For sign language understanding tasks, many of these landmarks are redundant (high-detail face mesh) or unnecessary (lower body), and add undesirable complexity. We discard all but 85 of these points, selected a priori using domain knowledge about sign languages:
- For each hand, we use all 21 landmark points.
- For the pose, we use 6 landmark points, for the shoulders, elbows and hips. This discards the lower body and pose landmarks redundant with the hand and face modules.
- For the face, we use 37 landmark points, from the eyes, eyebrows, lips, and face outline.
5These are indices 11, 12, 13, 14, 23, 24.
These are indices 0, 4, 13, 14, 17, 33, 37, 39, 46, 52, 55, 61, 64, 81, 82, 93, 133, 151, 152, 159, 172, 178, 181, 263, 269, 276, 282, 285, 291, 294, 311, 323, 362, 386, 397, 468, 473.
Figure 4: Overview of our model pipeline. Starting from an ASL video clip, we use MediaPipe Holistic to compute 3D landmarks for the face, hands, and body of the subject. We then discard irrelevant landmarks and normalize the remainder. These are concatenated and embedded by a linear projection layer into T5.1.1-Base, which then decodes the English translation. The blue components (Linear projection and T5) are the trainable parameters.
We normalize the landmarks by scaling them to fit in a unit bounding box across the duration of the clip. We represent landmarks that are not present in a frame with a large negative value. MediaPipe also predicts visibility (self-occlusion) of landmarks within the frame, which we ignore. To reduce sequence length, we discard every second frame. The final preprocessed input is therefore a half-frame rate sequence of 255-dimensional landmark vectors. Note that this half frame rate may vary from 7.5 to 30fps depending on the original video's frame rate, though most end up at 12 to 15 fps.
4.2 Model
Our model is a slightly modified version of T5 [33], which is an encoder-decoder Transformer [41] that has been trained on web-crawled English text. Rather than embed text tokens using a vocabulary of learned embeddings, we embed each 255-dimensional landmark frame into the encoder using a learned linear projection layer. Otherwise, our architecture is identical to T5.1.1-Base.
We set the encoder context window to 256 tokens (frames) and the decoder context window to 128 tokens, which accommodate the training examples after halving the input frame rate and encoding the target text with T5's SentencePiece vocabulary [23].
We choose not to provide train, validation, and test splits for YouTube-ASL. Because YouTube videos may be deleted over time, the validation and test splits could not serve as a stable benchmark. We instead evaluate on How2Sign [13], a studio-recorded dataset released under CC BY-NC 4.0 consisting of "How To" instructional narratives translated from English into ASL. This also allows us to integrate trends towards more robust evaluation from speech and text modeling [32, 15, 10], where models trained on large web corpora are evaluated both zero-shot and finetuned on independently constructed benchmarks.
Practices for constructing test sets in prior sign language dataset works are mixed. For example, OpenASL [39] and AfriSign [17] construct their test sets by randomly splitting at the sentence level; SP-10 [42] does the same but with multiway translations identified as a single sentence. How2Sign [13] samples document-level narratives rather than individual sentences, but most signers are shared between the train and test sets, and some narratives are present in both the train and test sets, translated by different signers. BOBSL [2] invests substantial effort into creating signer-independent, topic-balanced splits; this is perhaps why its translation baseline scores only 1.00 BLEU despite the dataset's size. Zero-shot evaluation lets us sidestep these issues and get a better sense of the model's quality for real use.