YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpus
Authors: David Uthus
Garrett Tanzer
Manfred Georg
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Sign language translation
dataset
TL;DR: Introducing a new datasets composed of YouTube videos for American Sign Language to English translation.

Abstract: Machine learning for sign languages is bottlenecked by data. In this paper, we present YouTube-ASL, a large-scale, open-domain corpus of American Sign Language (ASL) videos and accompanying English captions drawn from YouTube. With ~1000 hours of videos and >2500 unique signers, YouTube-ASL is ~3x as large and has ~10x as many unique signers as the largest prior ASL dataset. We train baseline models for ASL to English translation on YouTube-ASL and evaluate them on How2Sign, where we achieve a new fine-tuned state of the art of 12.397 BLEU and, for the first time, nontrivial zero-shot results.

YouTube-ASS: A Large-Scale, Open-Domain American Sign
Parallel Corpus
David Uthus, Garrett Tanzer, Manfred Georg
Google (duthus, (duthus,gtanzzer,uuu.
Abstract
Machine learning for sign languages is bottlenecked by data. In this paper, we present YoU, large-scale, open-domain corpus of American Sign Lan- guage (ASL) videos and accompanying English captions drawn from YouTube. With - 1000 hours of videos and >2500 unique signers, YouTube-ASe is -3x large and has -10x as many unique signers as the largest prior ASL dataset. We train baseline models for ASL to English translation on YouTube-AAS ASL and evaluate them on How2Sign, where we achieve a new finetuned state of the art of 12.39 BLEU and, for the first time, report zero-shot results.
1 Introduction
The primary bottleneck for machine learning research on sign languages is data, As minority languages used by historically marginalized Deaf/Hard of Hearing communities, sign languages lack the plentiful online resources that have facilitated modern machine learning advances [4, 44, 12]. This is compounded by the fact that sign languages have standardized written form: mining the videos that do exist is more difficult than retrieval for spoken language text. For translation specifically, there the added problem of finding spoken language captions that aligned corresponding sign language content, rather than a voiceover with its own timing. The result is that datasets tend to be constructed by recording new footage in studio or curating videos from small number of manually selected content creators, which limits variety.
In order to address these challenges, we present YouTube-ASL. large-scale, open-domain corpus of American Sign Language (ASL) videos and accompanying English captions, primarily intended for ASL English machine translation. We mined these videos from YouTube using two-step process: first, we used automatic content-base based annotations to identify potentially relevant captioned videos; and second, used skilled human annotators filter out videos with poor quality misaligned captions. The result is a dataset with 984 hours of high-quality captioned video featuring >2500 unique signers, which is ~3x large the largest prior ASL dataset [39] and has ~10x as many unique signers as any sign language dataset to date.
We train simple baseline models for sentence- level ASL to English translation on YouTube-ASL by embedding MediaPipe Holistic landmarks [26, 16] into the T5 language model [33]. Because YouTube vidcos may be removed over time and therefore cannot form a stable test set-and for comparison to prior work-we evaluate on a standard benchmark, How2Sign [13]. Borrowing from trends in mainstream machine learning [32, 15, 10]. we provide not just finetuned but also zero-shot! results to test out-of-dommin generalization. We achieve a new finetuned state of the art of 12.39 BLBU (vs. the prior SOTA of 8.03 [40]), and for the first time report zero-shot score, 3.95 BLEU.
Here, "zero-shot" refers to evaluation on an independently constructed benchmark without any kind of domain adaptation. This is different from the use of "zero-shot" in machine translation for transfer to unseen language pairs, or "zero-shot" in prompting for prompts without in-context examples.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Table I: Summary statistics for different sign language translation datasets. See Section 3.3 for
details on how these statistics were derived for YouTube-ASL.
Name
Language
Vocab.
Hours
Signers
Source
RWTH--PHOENIX-21447 [5]
DGS
3K
11
9
TV
BOBSL [2]
BSL
77K
1447
39
TV
SWISSTXT [7]
DSGS
88
TV
VRT-RAW [7]
VGT
100
TV
CSL-Daily [47]
CSL
2K
23
10
Lab
KETI [22]
KVK
419
28
14
Lab
Public DGS Corpus [18]
DGS
50
Lab
SP-10 [42]
various
17K
14
79
Web
AfriSign [17]
various
20K
152
Web
How2Sign [13]
ASL
16K
79
11
Lab
OpenASL [39]
ASL
33K
288
220
Web
YouTube-Ar AALTTbbe-SSLLC (ours)
ASL
60K
984
>2519
Web
We publicly release the YouTube-ASI video IDs. 2 We hope that YouTube-ASL. will be useful for general sign language pretraining, as well as downstream tasks such as ASI. English translation and caption alignment- in the near term to aid in the construction of larger sign language datasets, and eventually to improve accessibility for the Deaf/Hard of Hearing community.
2
Related Work
In this section, we review prior sign language translation datasets and methods for translation from sign languages spoken languages.
2.1 Sign Language Translation Datasets
Table I shows statistics on different sign language translation datasets. There are three main sources for sign language data: ad hoc recorded footage, interpreted TV broadcasts, and online video sharing platforms.
In the first category are datasets that manually recruit signers and record them performing translations of desired phrases, either in a lab setting or with camera on their personal device. These datasets tend to be small and feature few signers for logistical reasons, and may have exhaustive annotations because the small size of the dataset makes it feasible. This includes datasets such as CSL-Daily [47], with phrases related daily life in Chinese Sign Language; KETI [22], with phrases related to emergency situations in Korean Sign Language; Public DGS Corpus [18], with elicited dialogues in German Sign Language; and How2Sign [13], with "How To" instructional monologues translated into American Sign Language.
In the second category datasets that collate interpreted TV programs from collaborating national broadcaster. These datasets tend to be larger than newly recorded ones, but often use small number of non-native interpreters and lack fine-gra fine-grained caption alignment (because the supervision comes from the spoken language audio track). This includes datasets such as TWTHH-HHEEIXX-- 2014 [5], with weather forecasts interpreted into German Sign Language: SWISSTXT [7], with news/weather programs interpreted into Swiss German Sign Language; VRT [7]. with news programs interpreted into Flemish Sign Language; and BOBSL [2], with BBC programs many domains interpreted into British Sign Language. At 1447 hours, BOBSL is the largest sign language translation dataset to date (including the present work), but has only 39 signers and speech-aligned subtitles, vs. YouTube-ASL'ss 2519 signers and sign-aligned captions though the two datasets are complementary because they are for different languages.
In the third category are datasets that curate content from online video sharing platforms. In prior sign language translation datasets, this content is drawn from small number of manually selected channels.
Phttps://githun wtt.looooooooooooeeeeo com/google-ressaarch hwtpr//rrrrrrrrre///////////ttue

---

This includes datasets such as SP-10 [42], with example sentences from an online multilingual sign dictionary; AfriSign [17], with translated Bible passages hosted on the Jehovah's 's Witnesses website; channels: DailyMoth, Sign /News, and the National Association of the Deaf. OpenASI. the largest prior ASL dataset and closest work to YouTube-ASL.: the key difference is that YouTTubeAASL, is constructed with open-ended mining from automatic tags, rather than manual channel curation. OpenASL is largely a subset of YouTube-ASLL which-by utilizing the long tail of channels is -3x as large and has - 10x many unique signers.
and OpenASI. [39], with videos from three
There are several datasets for easier tasks than translation, like isolated sign recognition and finger- spelling recognition, that mine from the web by ambiguous means. MS MSS-ALL [21], WLASL [24], ChicagoFSWide S7//ChicagoFSWild+ [38]. CISLR [19], and Indian-SL [35] are word- datasets mined from YouTube, sign language-targeter sites like ASLU and ASL-LEX, or other unnamed video sharing platforms. These works do not specify how they retrieved their videos, so is possible that they used similar automatic tagging approach to YouTube ASI,, albeit on a more limited scale.
2.2 End-to-End Sign Language Translation
Originally. sign language translation approaches operated on glosses, linguistic annotations that represent individual signs, cascaded translation through glosses an intermediate step, like speech to text translation often cascades through speech recognition. More recently, due to a variety of deficiencies in glosses and lack of widespread gloss data, the field has shifted to end-to-end modeling with encoder-decoder Transformers, starting with Camgöz et al. [5].
The two main classes of approaches are those that take learned video embeddings as input [6, 40, 28] (via video encoders, primarily I3D [9], pretrained on tasks such as isolated sign recognition), and those that take estimated pose landmarks as input [28] (such as MediaPipe [26] or OpenPose [8]). Some works achieve modest gains given constant data with architectural tweaks like treating different cues in the input video (hands, face) differently [46, 43]. It is unclear to what extent these techniques are necessary or beneficial on larger datasets. Other works seek to benefit from transfer from spoken language other sign language data [11, 45, 17]. All of these works train and evaluate on splits derived from the same underlying continuous sign language corpus (different datasets across papers), and sometimes multiple such datasets independently in the same paper. In contrast, we train YouTube-ASL using an uncomplicated approach and evaluate on How2Sign, reporting both finetuned and zero-shot results to get more robust understanding of our model's state-of-the-an performance.
3 The YouTube-ASL Corpus
YouTube-ASL is a corpus of American Sign Language (ASL) videos with accompanying English captions drawn from YouTube. Video sharing platforms like YouTube are appealing sources of sign language data because they host swaths of diverse content that more broadly representative of real world conditions than studio footage is. Of course, much of this data is irrelevant or low-quality, so it imperative to develop cost-effective ways to sift through it.
We used a two-step pipeline to construct the corpus: first, retrieval using automatic content- based annotations, and second, filtering by skilled human annotators a per-video level. This automatic retrieval step represents a departure from prior continuous sign language corpora and brings us closer to mining approaches from mainstream machine learning.
3.1 Automatically Retrieving Candidate Videos
As described previously in Abu-EI-Haija et al. [1], the YouTube video annotation system associates machine- settin--eenrrted tags with each video in the form of Knowledge Graph entities, which are based on the video's 's metadata, context, and content signals. We retrieved listed public videos tagged as being related to sign language generally or American Sign Language specifically, This automatic tagging step, while having higher recall than prior works, was flawed in that it was not aware of sign language in the video content itself-to be expected due to the limited nature of current sign language processing. This means that, for example, videos in sign language that do not explicitly mention sign language in the content or context were unlikely to be discovered. This failure
of January 2022.
Sooe video IDs may have been removed from this set over time due to video deletions.
3

---

mode was most salient for press conferences with simultaneous interpreters, which tend not to have well-aligned captions anyway.
Given these retrieved videos, we drilled down on those with user-generated captions-i.e. captions that were manually uploaded rather than automatically derived from bpeach-because speech-derived captions are not tightly aligned with signed content. As heuristic filtering step, we automatically removed videos with duration <10 seconds or >5 hours, width <480 pixels or height >360 pixels, and frame rate <15sps or >60fps. We arrived at these values through an iterative mining and auditing process, so that we could reduce annotator labor spent on irrelevant videos without excluding too many relevant ones. From inspection, the heuristic excluded negligible amount of desirable videos. The one class of useful videos one might expect this to exclude, short isolated sign videos as used by MS-ASL [21] and WLASL [24], tends to have the label in the video title or description rather than captions, so removing videos under 10 seconds does not have substantial impact. Videos that were over 5 hours were often either live interpreted broadcasts (which did not have aligned captions) or not sign language (e.g., corrupted videos or mostly static content for hours on end).
Finally, we used off-the-shelf person detection tools to exclude videos where none of the captions corresponded spans with exactly one person present in the video. We limit the scope of our efforts to signing monologues due to the challenges of modeling conversations between multiple signers.
The result list of 88,002 candidate videos that might contain ASL with high-quality captions.
3.2 Identifying High-Quality Videos with Skilled Human Annotators
While some smaller datasets like How2Sign [13] use annotators to manually align all captions, this becomes prohibitively expensive for larger datasets. For this reason, OpenASI. [39] and BOBSI. [2] use annotators to correct only their validation and test sets. We take coarser-grained approach to annotations but apply it to our entire list of 88,002 candidates: we use humans to identify videos that are roughly suitable and include them in our corpus without modification.
To do so, we hired 3 native ASL signers with English proficiency to serve as annotators. The annotators used a bespoke internal tool that would display a given YouTube video and present label options. In order to save time, the annotators were able to mark that their labels held for an entire channel of videos rather than each video individually. Therefore it is possible that certain videos in the corpus are channel outliers and do not meet quality standards, but generally large channels have consistent quality. Each video was labelled by only one annotator unless they brought it up for wider discussion.
Through an iterative process involving written instructions, virtual meetings (through an ASL inter- preter or signing project members), and escalations by email for edge cases, we aligned on standards for when accept video into the corpus. Some of the reasons for exclusion include: the video's captions do not exclusively correspond to signing; the video is in a sign language other than ASL; the video's 'ss captions do not correctly translate the ASL; and the captions are poorly aligned 4. Notably, in order to increase the size of the corpus, we chose to include videos across all skill levels and signing styles, as long as they were comprehensible to an ASL user and correctly captioned. This variety is beneficial for sign language recognition tasks, where models should be able to understand all signers, but may limit the corpus's usefulness for generation tasks, where consistency and controllability are important.
The result was list of 11,093 videos whose captions are generally well-aligned English translations of signed ASL content.
3.3 Corpus Statistics
The final, human-filtered YouTube-ASL corpus consists of 11,093 ASL videos with 984 total hours of footage. This is -3x the size of OpenASL [39], the largest prior ASL dataset, but smaller than BOBSL [2]. British Sign Language dataset. See Table for comparison between the high-level
*For caption alignment, we are targeting captions that start and stop in close proximity to the signing, and the timings appear to have been annotated based on the video rather than a voice-over. Within a sentence, which may consist of multiple captions, there may be but is not necessarily an alignment because the syntax of ASL is different from English.

---

Table 2: Statistics on the distribution of captions and videos in the YouTube-ASL corpus.
Number of captions
610,193
Caption length (Average / 90th percentile, in characters)
48.9 88.0
Caption length (Average / 90th percentile, in words)
8.8 / 16.0
Caption duration (Average 90th percentile, in seconds)
4.8 8.76
Video duration (Average 90th percentile, seconds)
318.95 675.80
10a
10
Count
Count
10a
10
100
10
0
20
40
60
1,000
2,000
3,000
Caption duration (seconds)
Video duration (seconds)
Distribution of caption and video durations. For the video duration graph, omit 27 videos whose duration exceeds 3600 seconds (between 3610 and 9017 seconds).
Figure 1:
attributes of YouTube-ASL. and prior sign language translation datasets, including total number of hours.
These videos are paired with 610, 193 English captions, with a total duration of 813 hours. See Table for statisties on the distribution of captions, well as Figure 1 for visualizations. The average caption length (8.8 words) and duration (4.8 seconds) are relatively short, which reflects that sentences may be split across multiple captions. We computed vocabulary size by counting the number of distinct strings between whitespace or punctuation across all captions. It is important to keep mind that in addition the signing itself, these videos' captions vary in style, literalness of translation (whether the content was originally produced in ASL and translated, or translated into ASL from these captions), spelling/grammar correctness, and more. This degree of variability is difficult to quantify in comparisons between datasets.
We use the number of unique channels, 2519, unique signers in the dataset: some channels may feature many signers, and some signers may appear across multiple channels. Note that with this method, OpenASL. [39] would be estimated to have signers, while its authors reached count of 220 signers using more fine- grained methods. Even this likely underestimate -10x the count of any individual sign language dataset date.
approximate lower bound for the number of
Figure shows the distribution of videos per channel, for channels with at least 20 videos. There are few channels with many videos- in particular, the two largest channels the same channels featured in OpenASL- and then a long tail of channels with fewer videos. This means that the bulk of new footage present in YouTube-AST. but not OpenASL comes from relatively small channels, which helps variety. See Figure 3 for a sense of the distribution of (machine-annotated topics across videos: they seem more diverse than prior datasets from video sharing platforms but still shaped by typical YouTuue use cases, compared to BOBSL's more topic-balanced BBC programming.
Baseline Approach
In order to demonstrate the potential of YouTube-ASL, we consider a simple method for sentence- level machine translation from ASL to English built using off-the-shelf components. We use deliberately barebones approach to avoid introducing inductive bias that helps in more limited settings but becomes harmful with scale.
5

---

103
Video count
10a
Channel
Figure 2: Distribution of videos per channel for channels with least 20 videos.
Education
2,557
Culture and Lifestyle
1,956
Health
1,670
Politics
1,332
News
1,251
Religion
1,196
University
1,173
Deaf Services
834
Arts and Music
574
Employment and Jobs 258
Entertainment 1449
Science
Storytelling 70
Mathematics 69
Figure 3: A selection of high-level topics, with the number of YouTube-ASL videos automatically tagged as related to them. Note that a single video can be tagged with more than one topic.
4.1 Preprocessing
the raw captions from YouTube-ASL. Each training example clipped to the boundaries single caption. We filter out captions with length >300 characters or duration <200ms or >60s, which tend to be malformed, and any captions corresponding to video spans where exactly one person is not present. We do not lowercase the captions or apply any other kind of text normalization.
For our target English outputs,
For our sign language inputs, we use MediaPipe Holistic landmarks [26, 16], rather than raw video. Sign language models that use pose-based inputs have a history of underperforming those that operate on learned video embeddings [21, 27]; it is unclear to what extent this due the information bottleneck in the (imperfectly predicted) pose representation, vs. availability of higher quality pretrained video encoders than pretrained pose encoders. Pose inputs offer some benefits like computational efficiency and privacy.
MediaPipe Holistic a lightweight model that predicts 532 3D landmarks (in x-, y-, and z- image- space coordinates) for the hands, pose, and face of a single human in video footage. For sign language understanding tasks, many of these landmarks are redundant (high-detail face mesh) or unnecessary (lower body), and add undesirable complexity. We discard all but 85 of these points, selected priori using domain knowledge about sign languages:
For each hand, we use all 21 landmark points.
For the pose, we use 6 landmark points, for the shoulders, elbows and hips. This discards the lower body and pose landmarks redundant with the hand and face modules.
For the face, we use 37 landmark points, from the eyes, eyebrows, lips, and face outline."
*These are indices 12, 13,
24.
'These are indices 0, 4, 13, 14, 17, 33, 37, 39, 46, 52, 55, 61, 64. 81, 82, 93, 133, 151, 152, 159, 172, 178. 181, 263, 269, 276, 282, 285, 291, 294, 311, 323, 362, 386, 397, 468, 473.

---

Landmark reduction and normalization
Video frames
!il MediaPipe
Linear projection
Landmarks
T5.1. 15.1.1-Bass
Output
Figure 4: Overview of model pipeline. Starting from ASL video clip, we use MediaPipe Holistic to compute 3D landmarks for the face, hands, and body of the subject. We then discard irrelevant landmarks and normalize the remainder. These concatenated and embedded by linear projection layer into T5. I--ss,, which then decodes the English translation. The blue components (Linear projection and T5) the trainable parameters.
We normalize the landmarks by scaling them fit in a unit bounding box across the duration of the clip. We represent landmarks that are not present in frame with large negative value. MediaPipe also predicts visibility (self-occlusion) of landmarks within the frame, which we ignore. To reduce sequence length, we discard every second frame. The final preprocessed input is therefore a half- frame rate sequence of -55-dimensional landmark vectors. Note that this half frame rate may vary from 7.5 to 30fps depending on the original video's 's frame rate, though most end up at 12 to 15 fps.
4.2 Model
Our model is a slightly modified version of T5 [33], which is an encoder-decoder Transformer [41] that has been trained on web-crawled English text. Rather than embed text tokens using a vocabulary of learned embeddings, we embed each 255-dimensional landmark frame into the encoder using learned linear projection layer. Otherwise, our architecture is identical to L-.a...
We set the encoder context window 256 tokens (frames) and the decoder context window 128 tokens, which accommodate the training examples after halving the input frame rate and encoding the target text with T5's SentencePiece vocabulary [23].
5 Experiments
We choose not to provide train, validation, and test splits for YouTube-ASL. Because YouTube videos may be deleted over time, the validation and test splits could not serve as a stable benchmark. We instead evaluate on How2Sign [13], studio-recorded dataset released under CC BY-NC 4.0 consisting of "How To" instructional narratives translated from English into ASL. This also allows us to integrate trends towards more robust evaluation from speech and text modeling [32, 15, 10], where models trained on large web corpora are evaluated both zero-shot and finetuned on independently constructed benchmarks.
Practices for constructing test sets in prior sign language dataset works are mixed. For example, OpenASL [39] and AfriSign [17] construct their test sets by randomly splitting the sentence level; SP-10 [42] does the same but with multiway translations identified as a single sentence. How2Sign [13] samples document-levar narratives rather than individual sentences, but most signers are shared between the train and test sets, and some narratives are present in both the train and test sets, translated by different signers. BOBSL [2] invests substantial effort into creating signer-independent, topic-balanced splits; this is perhaps why its translation baseline scores only 1.00 BLEU despite the dataset's size. Zero-shot evaluation lets us sidestep these issues and get a better sense of the model's quality for real use.
7

---

Table Metrics for ASL to English translation on How2Sign. Our models either train from scratch or finetune pretrained T5 checkpoint, and are trained on How2Sign (H2S) only, YouTube-ASL (YT-ASI.) only, mixture of H2S and YT-ASI, or YT-ASI. and then finetuned on H2S.
Approach
Training Schedule
BLEU-1
BLEU-2
BLEU-3
BLEU
BLEURT
Alvarez et al. [3]
H2S
17.40
7.69
3.97
2.21
GloFE-VN [25]
H2S
14.94
7.27
3.93
2.24
31.65
Tarrés et al.[40]
H2S
34.01
19.30
12.18
8.03
H2S
13.92
4.69
1.82
0.86
30.65
Ours
YT-ASL
14.53
5.47
2.61
1.41
29.55
(no pretraining)
YT-ASL H2S
28.60
14.56
8.68
5.60
37.72
YT-ASL H2S
28.38
15.41
9.55
6.26
39.40
H2S
14.96
5.11
2.26
1.22
29.98
Ours
YT-ASL
20.93
10.35
6.14
3.95
34.98
(pretrained)
YT-ASL + H2S
36.35
23.00
16.13
11.89
44.78
YT-ASL H2S
37.82
24.13
16.92
12.39
46.63
5.1 Setup
We ablate across four different training schedules:
H2S. We train only How2Sign, YouTube-ASL, for like-for-like comparison with prior methods.
YT-ASL: We train only on YouTube-ASL, and evaluate on How2Sign zero-shot.
YT-ASL H2S: We train a mixture of How2Sign and YouTube-ASL, mixed in propor-
tion to the size of the datasets.
YT-ASL- H2S: We train on YouTube-ASI then finetune on How2Sign.
We also ablate the effect of pretraining on English text by comparing models trained from scratch using the 1..1.11Bar architecture, vs. finetuned from the 15.1.1 pretrained checkpoint.
We train with batch size of 128 and learning rate of 0.001 with Adafactor [36]; other hyperparameters the T5X defaults. For models trained solely on How2Sign data, train for 20,000 steps. For models trained on YouTube-ASL (including with How2Sign mixed in), we train for 200,000 steps. When finetuning on How2Sign after training YouTube-ASL, finetune for an additional 5,000 steps. Each 1,000 steps takes approximately 0.25 TPUv4-hours.
Following prior work, present BLEU [29] and BLEURT [34] scores. BLEU scores computed using SacreBI BEEU [30] version 2, with all default options. BLEURT scores are computed using checkpoint BLEURT-20 [31, 14]. We decode using beam search with beam width of 5.
5.2 Results
See Table 3 for metrics comparing our models to prior works on How2Sign [3, 25, 40]. The best results come from training on YouTube-ASL from pretrained checkpoint, then finetuning on How2Sign, which achieves 12.39 BLEU vs. the state the art of 8.03 BLEU [40]. The base model achieves 3.95 BLEU zero-shot, which is nontrivial but substantially worse than the finetuned score. Factors that could contribute to this gap include train/test Icakage of signers and narratives, How2Sign's narrow domain, and the extra ~00% training data it represents.
Results substantially worse when training from scratch, which suggests that T5's English pretrain- ing gives the model better initialization, as De Coster et al. [11] found for frozen pretrained language models. Results are absymal when trained without YouTube-ASL. The most direct comparison of our approach to prior work is T5 trained from scratch on How2Sign only, which reaches just 0.86 BLEU, despite training on the same data as Tarrés al. [40]'s 8.03 BLEU. This might be explained by their use of pretrained video encoder and various decisions they made to optimize for small amounts of data (smaller network, more text normalization, careful hyperparameter sweep), whereas we used less tuned configuration that was intended for larger datasets.

---

Table Qualitative examples from our best finetuned and zero-shot models, on sentences sampled from How2Sign by Tarrés et al. [40]. See Table 5 in the Appendix for the complete set of examples.
Reference
And that's a great vital point technique for women's 's self defense. It's really a great point for women's self defense.
Tarrés et al.
Ours (zero-shot)
It's really great, especially for women who are facing barriers. It's really great for women's self defense.
Ours (finetuned)
Reference
In this clip I'm going to show you how to tape your cables down. In this clip I'm going to show you how to improve push ups. This video will show how to use the code online.
(2)
Tarrés et al. Ours (zero-shot)
Ours (finetuned)
In this clip we'r re going to show you how to cut a piece of clay.
Reference
In this segment we're going to talk about how to load your still for distillation of lavender essential oil.
(3) Tarrés et al.
Ok, in this clip, we're going talk about how to fold the ink for the lid of the oil.
Ours (zero-shot) Ours (finetuned)
This video will discuss how to submit a digital form for the survey. In this clip we're re going to talk about how to feed a set of baiting lizards for lava field oil.
See Table for qualitative examples of the translations produced by our best finetumed and zero-shot models, on sentences sampled from How2Sign by Tarrés et al. [40]. The translations capture elements of the reference translation but are clearly not yet of usable quality. The zero-shot predictions hew less closely to the references, but the errors usually make sense in light of the sign language input. For example, in (1), the sign used to mean "defense" also means "barrier".
6 Conclusion
In this paper, we presented YouTube-ASL, a new, publicly available parallel corpus for American Sign Language and English that is -3x the size and has -10x many unique signers the largest prior ASL dataset. Our key improvement over prior work is that we used automatic tagging followed by human filtering to increase mining recall without harming precision. We demonstrated the value of this data with a simple baseline built from off-the-shelf components (MediaPipe Holistic and T5) that achieves new finetuned state of the art in ASI. English translation How2Sign, 12.39 BLEU. We also reported a zero-shot score of 3.95 BLEU, a first for sign language translation. We hope that Toue-beAASL will be immediately useful for research on methods for sign language translation and caption alignment, as well as tools for automatic annotation/iilering of new sign language datasets. Because YouTube-ASL has so much signer variety, including across dialect and skill level, may be less useful for generation than recognition tasks.
While our baseline improves upon prior work, even the finetuned translations are subjectively low- quality and are not yet useful the real world. We hope that more refined modeling approaches will provide better results with the same data, but despite our and prior efforts, ASL is still low-resource language by modern standards [20]-let alone the many other sign languages of the world, most of which are even less resourced. Future work may look to address this by mining broader datasets with other kinds of supervision, and exploring multilingual transfer at larger scales. As model quality improves, future work should perform more comprehensive evaluations understand differences across domains, dialects, levels of fluency, signer appearance, and other such factors.
7 Ethical Considerations
Sign language datasets pose privacy challenges, because the signer's 's appearance (body, facial ex- pressions), which is personally identifying, is a vehicle for the language itself. Video anonymization techniques are not yet mature enough to be useful in this regard. Our corpus is composed of sign language content that uploaders made publicly visible on YouTube, and we release only video IDs so that changes to the underlying videos automatically reflected in the corpus. While the corpus cov-

---

ers broader variety of channels than prior works, this does not mean it is necessarily representative of the signing population or even if it were representative, that models trained on it would work equally well for everyone.
We train models reduced poses form of anonymization, but this is not suitable for all modeling approaches and may harm model quality. Until sign language translation models are closer usable quality, there is little risk of societal harm, except that individuals or organizations mistakenly rely on models that are inadequate. As we approach that point, sign language processing will adopt the risks of natural language processing in general, but with great potential improve accessibility for Deaf/Hard of Hearing people.
References
[1] Sami Abu-EI-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. YouTube-SM: A large-scale video classification benchmark, 2016. URL https: haapp://arxiivi /twxxv.xivvv////// wwwg/org/////eeeeeeooeee abbs/1699.... 00677.0677......
[2] Samuel Albanie, Gül Varol, Liliane Momeni, Hannah Bull, Triantafyllos Afouras, Himel Chowdhury, Neil Fox, Bencie Woll, Rob Cooper, Andrew McParland, and Andrew Zisserman. BBC-Oxford British Sign Language dataset, 2021. URL ptttps//aarivv.oo wwt..orr///b///i//eo
03635.
[3] Patricia Cabot Sign language translation dataset. 2022.
Álvarez,
Xavier Giró Nieto, and Laia Tarrés Benet. transformers for the How2Sign URL https: /
based on
http//mmatee.e// upc. www.wwp/c...eeee...o aadu//w//gg based-rransformers-booe wttp.///b/bbblicaaiioooooo
sign- mwnguage-transsinn
dataset.
[4] Danielle Bragg, Oscar Koller, Mary Bellard, Larwan Berke, Patrick Boudreault, Annelies Braffort, Naomi Caselli, Matt Huenerfauth, Hernisa Kacorri, Tessa Verhoef, Christian Vogler, and Meredith Ringel Morris. Sign language recognition, generation, and translation: An interdisciplinary perspective. In Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS 19, page 16-31, New York. NY, USA, 2019. Association for Computing Machinery. ISBN 9781450366766 doi: 10.1145/3308561 3353774. URL https::ppp.. / ///////oo org/ 100 10. 1145 3005//333056...1 3353774.
[5] Necati Cihan Camgöz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. Neural sign language translation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018. URL https: /dooi. org/ 1tng/10.11 100110//CVPR.201 00812.
[6] Necati Cihan Camgöz, Oscar Koller, Simon Hadfield, and Richard Bowden. Sign language transformers: Joint end-to-end sign language recognition and translation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. URL https: :tttps://doi. org/10. 1109/ /00/9///V220000///// 2020 01004.
[7] Necati Cihan Camgöz, Ben Saunders, Guillaume Rochette, Marco Giovanelli, Giacomo Inches, Robin Machtrab-Ribback. and Richard Bowden. Content4all open research sign language translation datasets. 2021. URL pettps:/aariie org/ abt.//0.s22110... 02351.
[8] Z. Cao, G. Hidalgo, Simon, S. Wei, and Sheikh. OpenPose: Realtime multi-person 2D pose estimation using part affinity fields. IEEE Transactions Pattern Analysis and Machine Intelligence, 43(01)) jan 2021. ISSN 1939-3539, doi: 10.1109/TPAM..2019 20992999 URL https: //oii ieeecompttersociety org/ /10. 1109/TPAMII AAAMI 2019 2929257.
[9] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. URL attps://áon http///iioorgg/..///// 10011109///00 CVPR. 2017.502.
[10] Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. FLEURS: Few-shot learning evaluation of universal representations of speech. In 2022 IEEE Spoken Language Technology Workshop (SITT), pages 798-805, 2023. URL https: http///oo.ooo///o//ooo org/ 10.
2023. 10023141.
10

---

[11] Mathieu De Coster, Karel D' D'Oosterlinck, Marija Pizurica, Paloma Rabaey, Severine Verlin- den, Mieke Van Herreweghe, and Joni Dambre. Frozen pretrained transformers for neural sign language translation. In Proceedings of the 1st International Workshop on Automatic Trunslation for Signed and Spoken Languages (AT4SSL.). pages 88-97, Virtual, August 2021. Association for Machine Translation in the Americas. URL ARR.https://calatttoo. 2021 wwl.mmuuuuit...mmm at4ss1.10.
org/
[12] Mathieu De Coster, Dimitar Shterionov, Micke Van Herreweghe, and Joni Dambre. Machine translation from signed spoken languages: State of the art and challenges. Universal Access in the Information Society, pages 1-27, 2023. URL https: //doi:////..../ 1trg/00.0077 $10209-023-00992 1.
[13] Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giro-i Nieto. How2Sigm: A large-scale multimodal dataset for continuous American Sign Language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2735-2744, June 2021. URL https: //doi.org/100 1et1/1109///// /CV/R/64P233...// 2021 00276.
[14] Research. rhoolleresearcchbleeut:: Bleurt is metric for natural language generation based on transfer learning. URL httth:///githutit com/ how.gooooooeeaaeeeeeeeeeeeeeeeee chneearcch/beerrr
[15] Naman Goyal, Cymthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Mare' Aurelio Ranzato, Francisco Guzman, and Angela Fan. The flores-101 evaluation benchmark for low-resource and multilingual machine translation, 2021.
[16] Ivan Grishchenko and Valentin Bazarevsky. Mediapipe holistic simultancous face, hand and pose prediction, on device, Dec 2020. URL pottps://ail bttt:::/aaiggoogeeeleee. googleblog. wttpee/.oo...////////// wwwcco20/ 12/ aodiapipe-holistie wtt.ssssmmllalteeeeeeeeeeeeeeeee face. html.
[17] Shester Gueuwou, Kate Takyi, Mathias Müller, Marco Stanley Nyarko, Richard Adade, and Rose-Mary Owusuaa Mensah Gyening. Afrisign: Machine translation for african sign languages. In 4th Workshop on African Natural Language Processing, 2023. URL https: htwp/rpeeeeeeeeeeeeeee eview. net/forum? -eeeiid=HHeleo GH33k3J22k.
[18] Thomas Hanke, Mare Schulder, Reiner Konrad, and Elena Jahn. Extending the Public DGS Corpus in size and depth. In Proceedings of the LREC2020 9th Workshop the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives, pages 75-82, Marseille, France, May 2020. European Language Resources Association (ELRA). ISBN 979-10-95546. 54-2. URL http:::/p//// aalaaaataaaattoccccccaaccaaaaaat
[19] Abhinav Joshi, Ashwani Bhat, Pradeep S, Priya Gole, Shashwat Gupta, Shreyansh Agarwal, and Ashutosh Modi. CISLR: Corpus for Indian Sign Language recognition. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10357- 10366, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties. doi: 10. 10011863/112222.mmmll/ main.707. URL https: http:///aaaaaatloa////// org/ 2022. caammp-mmii.700 707.
[20] Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the nip world, 2021. URL https://arxiv org/abs /2oss//20444 09095.
[21] Hamid Reza Vaezi Joze and Oscar Koller. MS-ASL: A large-scale data set and benchmark for understanding american sign language. CoRR, abs/1812.001053 2018. URL http: //arxiv. org/abs/1812. attaaa/////1..00 1015/13311.........
[22] Sang-Ki Ki Ko, Chang Jo Kim, Hyedong Jung, and Choongsang Cho. Neural sign language translation based on human keypoint estimation. Applied Sciences, 9(13), 2019. ISSN 2076- 3417. doi: 10.3390/app9132663
URL httpps://wwer :/wwww .nwpi com /ttp//200764377///13
[23] Taku Kudo and John Richardson. SentencePiece: simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference
11

---

on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66 71, Brussels. Belgium, November 2018. Association for Computational Linguistics. doi: 10.1 N..1863//1/D188 8-112... URL.
wtw.oigyyrrg//gD0000000 2012.
[24] Dongxu Li, Cristian Rodriguez, Xin Yu, and Hongdong Li. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. In The IEEE Winter Conference on Applications of Computer Vision, pages 1459-1469, 2020. URL https:
//doi.org/10, 1109/ /AAV45572.. 2020. 9093512.
[25] Kezhou Lin, Xiaohan Wang, Linchao Zhu, Ke Sun, Bang Zhang, and Yi Yang. Gloss-free end-to-end sign language translation, 2023. URL https: //axiiv.////i//////eeaeeeem... org/ abs/ 2205. 12876.
[26] Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang. Ming Guang Yong, Juhyun Lee, Wan-Teh Chang, Wei Hua, Manfred Georg, and Matthias Grundmann. MediaPipe: A framework for building perception pipelines, 2019. URL htwps: http::::/aaxxv.rrregg abs/ 1906.08122.
[27] Amit Moryossef, Ioannis Tsochantaridis, Joe Dinn, Necati Cihan Camgöz, Richard Bowden, Tao Jiang, Annette Rios, Mathias Muller, and Sarah Ebling. Evaluating the immediate applicability of pose estimation for sign language recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 1434-3440) June 2021. URL https: /twii org/ 10. L009CCVPW53009 2021 00382.
[28] Mathias Müller, Sarah Ebling, Eleftherios Avramidis, Alessia Battisti. Michèle Berger, Richard Bowden, Annelies Braffort, Necati Cihan Camgöz, Cristina aspaña-bonet Roman Grund- kiewicz, Zifan Jiang, Oscar Koller, Amit Moryossef, Regula Perrollaz, Sabine Reinhard, Annette Rios, Dimitar Shterionov, Sandra Sidler-miserez, and Katja Tissi. Findings of the first WMT shared task on sign language translation (WMT-SLT22). In Proceedings of the Seventh Conference on Machine Translation ( (WMT), pages 744-772 772, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. URL htttps://aclee aallccllanooooyyyyyoooooooooolll arg/2022 wmt-1.77
[29] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.1115/107303331733333 URL https:
//aclanthology org/ 20t:/P22-110 1040-
[30] Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Belgium, Brussels, October 2018. Association for Computational Linguistics. URL https: /wwww aclweb. org/ anthology/ 6319.
[31] Amy Pu, Hyung Won Chung, Ankur Parikh, Sebastian Gehrmann, and Thibault Sellam. Learn- ing compact metrics for MT. In Proceedings of the 202/ Conference Empirical Methods in Natural Language Processing, pages 751-762, Online and Punta Cana, Dominican Re- public, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2221. emnlp-main.58. URL https:tt//. http///aaaaaaaooac..aaaaa org/2021 emnlp-main. 58.
[32] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision, 2022. URL https: / //arxiv.org/abst abs 2212 04356.
[33] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Mach. Learn. Res., 21(1), jan 2020. ISSN 1532-4435. URL https://dl.
acm. org/doi// hts///d//bb///////////// 111111055555 08555/3455711 16. 3455856.
[34] Thibault Sellam, Dipanjan Das. and Ankur Parikh. BLEURT:
Learning robust metrics for text generation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881 7892, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/ 1/06533/v12002..ac 704..704. URL https: //aaaaaaaaaaatoyoccaaaaac URRLattps://adaatthllooyy org/2020. 2020 acl mainn.
704.
12

---

[35] Prem Selvaraj. Gokul Nc, Pratyush Kumar, and Mitesh Khapra. OpenHands: Making sign language recognition accessible with pose-based pretrained models across languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguis-
ties (Volume 1: Long Papers), pages 21 2114-2133, Dublin, Ireland, May 2022. Associa- tion for Computational Linguistics. doi: 10. 10.18665//112222aalloooo 150. URL https:
/ //aclanthology org att.//2222cc111000 1ong.llog..1 150.
[36] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596 4604. PMLR, 10-15 Jul 2018. URL
mlr www.ppressvvee
shazeer 18a. html.
[37] Bowen Shi, Aurora Martinez Del Rio, Jonathan Keane, Jonathan Michaux, Diane Brentari, Greg Shakhnarovich, and Karen Livescu. American Sign Language fingerspelling recognition in the wild. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 145-152, 2018. URL https: //ooi. org/oo////../... 10. 1109/ /.9//LTT.. 2018. 8639639.
[38] Bowen Shi. Aurora Martinez Del Rio, Jonathan Keane, Diane Brentari, Greg Shakhnarovich, and Karen Livescu. Fingerspelling recognition in the wild with iterative visual attention. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), October 2019. URL battps://doll htt:://oo.orrr////// 10. 1109/ ML.9/IICO ICCV 2019. 00550.
[39] Bowen Shi, Diane Brentari, Greg Shakhnarovich, and Karen Livescu. Open-domain sign language translation learned from online video, 2022. URL 12870.
[40] Laia Tarrés, Gerard I. Gállego, Amanda Duarte, Jordi Torres, and Xavier Giró Nieto. Sign language translation from instructional videos, 2023. URL https://aaxix: http://axxvvv.og/aaa/ wrw./bs/////aa 06371.
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, ed-
U. Von Luxburg, itors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: / //poccedd/s/ccos/ss....... neurips... wwwripp.cc/pppe iaappprfillss__ wtt.le//pappe//17///iiii Paper pdf.
[42] Aoxiong Yin, Zhou Zhao, Weike Jin, Meng Zhang, Xingshan Zeng, and Xiaofei He. MISLT: Towards multilingual sign language translation. In 2022 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5099-5109, 2022. URL. https//ts///do org/
10. 20009/////2288../0 2022. 00505.
[43] Kayo Yin and Jesse Read. Better sign language translation with STMC-transformer. In Proceedings of the 28th International Conference on Computational Linguistics. pages 5975- 5989, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. doi: 10. 11166////122220cooonngn--- m25.. URL httpss//cccaaattllllgg org/ 2020. coling main. www..ma..................
[44] Kayo Yin, Amit Moryossef, Julie Hochgesang, Yoav Goldberg, and Malihe Alikhani. Including signed languages in natural language processing, 2021. URL
abs/
[45] Biao Zhang, Mathias Müller, and Rico Sennrich. SLTUNET: A simple unified model for sign language translation. In The Eleventh International Conference on Learning Representations, 2023. URL https::/ //tp://////pereiieet net/ meet/forumiide Derrum?7d-EB4477p.55
[46] Hao Zhou, Wengang Zhou, Yun Zhou, and Houqiang Li. Spatial-temporal multi-cue network for continuous sign language recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 3009-11600,,,,,01,, Apr. 2020. doi: 10. (6669//aa..33400.... URL https:
/ //ojs. aaai. org/ index.idxxxxi wwpp///..pppp/eeeemmme wttpe..ppp//////aaiiiieeeeeooo WAA.article/vveeee...
13

---

[47] Hao Zhou, Wengang Zhou, Weizhen Qi. Junfu Pu, and Houqiang Li. Improving sign language translation with monolingual data by sign transsatnnsa... In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1316-1325, June 2021.
URI. htt.htttt:///o.... org/10 10. 1109/ CVPR46437 2021.00137
14

---

A
Appendix
This document contains supplementary material for the YouTube-ASL paper.
A.1
Full Qualitative Results
Table 5:
The complete set of qualitative examples from best finetuned and zero-shot models, on sentences sampled from How2Sign by Tarrés et al. [40].
Reference
And that's great vital point technique for women's 's self defense.
Tarrés et al.
It's really a great point for women's self defense.
Ours (zero-shot)
It's really great, especially for women who are facing barriers. It's really great for women's self defense.
Ours (finetuned)
Reference
In this clip I'm going to show you how to tape your cables down. In this clip I'm going show you how to improve push ups. This video will show how to use the code online.
(2) Tarrés et al.
Ours (zero-shot)
Ours (finetuned)
In this clip we going to show you how to cut a piece of clay.
Reference
In this segment we're going to talk about how load your still for distillation of lavender essential oil.
(3) Tarrés et al.
Ok, in this clip, we' we're going talk about how to fold the ink for the lid of the oil.
Ours (zero-shot) Ours (finetuned)
This video will discuss how to submit a digital form for the survey. this clip we're re going to talk about how to feed a set of baiting lizards for lava field oil.
Reference
You are dancing, and you going need the veil and you are going to just grab the veil as far as possible.
(4) Tarrés et al.
So, once you'l re belly dancing, once you' 've got to have the strap, you't 're going need grab the thumb, and try to avoid it. he's dancing lot. Now he needs hat and chain
Ours (zero-shot)
Ours (finetuned)
Their hopping and dancing is now, they're 're going to need their squat and squat and they re going to be able to move independently.
Reference
But if you have to setup new campfire, there's 's two ways to do it in a very low impact: one is with mound fire, which we should in the campfire segment earlier and the other way to setup low impact campfire is to have a fire pan, which is just steel pan like the top of trash can.
Tarrés et
And other thing I'm going to talk to you that's what it's going to do, it's kind of a quick, and then I don't want to take a spray skirt off, and then don't want it to take it to the top of it. But you have to set up a new campfire, you have to set up a campfire. You have to do it in a campfire, set up tentfire.
little bit more space, a space
(5)
Ours (zero-shot)
Ours (finetuned)
But if you have to set up new campfire, there are two ways to do low impact fire, one a cone fire, which we have to do in the tent earlier, and the other one is to set up a campfire in fire pan.
Reference
So, this is very important part of the process.
(6) Tarrés et al.
It's very important part of the process.
Ours (zero-shot)
Wash your hands. Alright, let's get started.
Ours (finetuned)
B Datasheets for Datasets
We provide documentation of the dataset based Datasheets for Datasets 7,
"annss//aniioorg/oo////1...00000
15

---

B.1 Motivation
For what purpose was the dataset created? The dataset was created primarily to serve as training data for ASL to English machine translation; prior datasets are smaller and have fewer unique signers. We used human annotators to identify high-quality ASL videos with well-aligned captions, but it was not feasible to manually correct or align any of the included captions. This is generally sufficient for translation, but slightly less ideal for tasks like ASI. to English caption alignment, where consistent alignment standards might be desired. The dataset is probably also less suitable for English to ASL translation, due to the signing variation across videos, though this may be addressed with methods for improved controllability.
Who ereated the dataset (e.g,, which team, research group) and on behalf of which entity (e.g., company, institution, organization)? This dataset was created by Dave Uthus, Garrett Tanzer and Manfred Georg for Google.
Who funded the creation of the dataset? Google.
B.2 Composition
What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Each instance is the video id of a YouTube video.
How many instances are there in total (of each type, if appropriate)? There are 11,093 video ids.
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from larger set? This contains a subset of videos available on YouTube of people signing in American Sign Language with English captions. This is not strictly representative of the larger set, as we have applied a combination of automatic and manual filtering techniques to find high quality videos with high-quality captions.
What data does each instance consist of? Each instance consists of single video id, which represents an ASL video with associated English captions.
Is there label or target associated with each instance? The English captions may be considered the target for each instance, but this depends on the task being attempted.
Is any information missing from individual instances? No.
Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? No.
Are there recommended data splits (e-g,, training, developmentvvliiattonn, testing)? No. YouTube datasets can change over time due to the nature of the platform (videos can made private or deleted), thus there are no recommended splits
Are there any errors, sources of noise, or redundancies in the dataset? Yes. During the annota- tion process, we allowed the annotators to mark whole channels with the same annotations, so there may be some videos which are not of the same quality as the rest of the channel. There may also be minor errors in the videos or captions that the annotators explicitly deemed acceptable.
Is the dataset self-contained, does it link to or otherwise rely on external resources (e.g.. websites, tweets, other datasets)? The dataset is not self-contained, as it consists of YouTuube video ids only. As such, there is no guarantee that the dataset will remain constant over time. The actual videos themselves fall under YouTube's Terms of Service https:/ http://www..bbbbeee.eee/o statitt template=terme
Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor- patient contidentiality, data that includes the content of indi- viduals' non-public communications)? No, the dataset consists of video ids for public videos only, and does not rehost any of the underlying data.
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxicty? The video ids comprising the dataset itself are random identifiers. The videos referenced by our video ids are hosted by YouTube and therefore subject to YouTube community guidelines. We annotators did not encounter any such content.
16

---

Does the dataset identify any subpopulations (e.g., by age, gender)? The dataset does not identify any subpopulations.
Is it possible to identify individuals (i.e., one or more natural persons), either directly or in- directly (i.e,, in combination with other data) from the dataset? The video ids comprising the dataset itself are random identifiers. The videos referenced by our video ids include people signing in ASL, which inherently includes the person's appearance. Our dataset does not provide any extra information about these people that was not already publicly available from their uploaded videos, and respects when videos are deleted/made private by virtue of using video ids.
Does the dataset contain data that might be considered sensitive in any way (e.g.. data that reveals race or ethnic origins, sexual orientations, religious belief',, political opinions union memberships, or locations; financial or health data; biometric or genetic data; forms of gov- ernment identification, such social security numbers; criminal history)? As with the previous question, the videos referenced by our video ids may contain information about many topics, the signer chose discuss that information in their publicly uploaded video. Our dataset does not provide any extra information and respects deletions.
B.3 Collection Process
How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), indirectly in- ferred/derived from other data (e-g. part-of-speech tags, model-based guesses for age or lan- guage)? The videos referenced by each video id instance consist solely of data that is directly observable (uploaded videos and captions). The selection of video ids is implicitly decided by combination of automatic and manual filtering in order to ensure a relatively high quality level.
What mechanisms or procedures were used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)? A combination of software programs and manual annotations were used to select preexisting YouTube videos.
If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)? Not applicable.
Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)? The authors collected the initial collection of video ids, and annotators were hired to help annotate the videos for filtering purposes.
Over what timeframe was the data collected? We collected videos up to January 2022.
Were any ethical review processes conducted (e.g., by an institutional review board)? No.
Did you collect the data from the individuals in question directly, obtain it via third parties or other sources (e.g., websites)? The dataset consists of references to videos that include people, but doesn't 't collect or release any new information about those people.
Were the individuals in question notified about the data collection? No, as we only provide video ids and no further information about the videos.
Did the individuals in question consent to the collection and use of their data? Not applicable.
If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? As we only provide ids and not the raw content, users removing the video will make them no longer available for use in our dataset.
Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., data protection impact analysis) been conducted? No.
B.4 Prepooesing/deeninghhhllng
Was any preproessingcctannng/leeinn the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? Yes, we filtered out videos that were not relevant, of poor quality, had poor captions, etc. The result is a list of video ids, which point to unmodified videos.
17

---

Was the "raw" data saved in addition to the preproeeseedddeerrdd/bbeelee data (e.g,, to support unanticipated future uses)? The original set of video ids will not be made available.
Is the software that was used to preprocess/clenn/lbell the data available? No.
B.5
Uses
Has the dataset been used for any tasks already? None prior to the baselines provided in this paper.
Is there repository that links to any or all papers or systems that use the dataset? No.
What (other) tasks could the dataset be used for? In addition to the intended task of ASL to English translation, the dataset could be used for related sign language understanding tasks like caption alignment, and potentially for sign language generation tasks like English to ASL translation, or sign language tasks that do not require captions.
Is there anything about the composition of the dataset or the way it was collected and prepro- cessed//leaned//abeeee that might impact future uses? The dataset was filtered for high-quality videos and captions, which includes variety of signing styles and skill levels, as long as they can be understood by an ASL user and are captioned correctly. This amount of variety may be unideal for tasks like generation where consistency is preferred. Even though it is varied, it is still not necessarily representative the signing community as a whole, and should be treated as such.
Are there tasks for which the dataset should not be used? This dataset should not be used as benchmark for comparing models across time, because the data that YouTube-ASL is derived from will change over time with video deletions or other modifications.
Any other comments?
B.6 Distribution
Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? The dataset is open sourced.
How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? GitHub,
When will the dataset distributed? currently available.
Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? We release the YouTube-ASL video ids under CC BY 4.0 International license, while the actual videos/captions on YouTube are preexisting and subject the YouTube Terms of Service (https: /wwww youtube com/ www/sttaiie wttp:stttttttppllcc//c aeaaiiiceelllttttteemee ate=terms).
Have any third parties imposed Ia-ed or other restrictions on the data associated with the instances? See above for license information.
Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? No.
B.7 Maintenance
Who will be supportiig/nssttngmmaaaaiiingggg the dataset? The authors will be responsible for maintaining the dataset.
How can the owner/curator/mmaager of the dataset be contacted (e.g., email address)? By contacting any of the authors listed on the publication.
Is there an erratum? No.
Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete in- stances)? It may be updated, and so, updates will be communicated via the associated GitHub page.
If the dataset relates people, are there applicable limits on the retention of the data asso- ciated with the instances (e.g., were the individuals in question told that their data would be
18

---

retained for fixed period of time and then deleted)? Our dataset is constructed similarly to past YouTube-related datasets, in that we only provide video ids. Thus, if user makes their YouTube video private or deletes it, this will then no longer be available for use.
Will older versions of the dataset continue to be suporredhnsteddmahtaince No, if need to remove older versions, these will be communicated on the associated GitHub page.
If others want to extend/augment/huil on/contribute to the dataset, is there mechanism for them do so? No, we are currently not planning to allow formal contributions to the dataset at this time, but others may extend the dataset on their own in accordance with the license.
C Additional Information
URL to the data: youtube_ asl
Hosting and maintenance: The data website is on GitHub under Google Research's shared GitHub repository, while the data itself is hosted on Google Research's shared Google Cloud Service.
D Author Statement
The authors bear all responsibility in case of violation of rights, and confirm that this dataset is open-sourced under the CC BY 4.0 International license.
19


Reviews:
[{"review_id": "x9WSz5Uy3t", "reviewer": "Reviewer_Px9y", "summary": "", "questions": "", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. This dataset is a timely and high-quality contribution to an under-investigated area with significant societal value.\n2. The improvement in model performance from a BLEU score of 1.22 to 12.39 is non-trivial and meaningful.\n3. The dataset has the potential to further advance the field by making Sign Language Translation (SLT) a real-world application. It provides a valuable foundation for future research in this area.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "4PgDfZsOoT", "reviewer": "Reviewer_f1hs", "summary": "", "questions": "", "limitations": "Yes, the paper has a section on ethical consideration and societal impact.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper is well-written, the limitations of the work is clearly explained, and justified, the rationale for the choices are presented.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "NBNBsvPYdd", "reviewer": "Reviewer_HbjK", "summary": "", "questions": "", "limitations": "The authors have adequately discussed limitations.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The dataset is much larger than existing ASL datasets, which is definitely helpful to the research on sign language translation.\n2. The collection process is simple and cheap, which can be easily migrated to other sign languages.\n3. Zero-shot experiments over How2Sign are conducted.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "RIUCR5lYtS", "reviewer": "Reviewer_3tD6", "summary": "", "questions": "", "limitations": "1. The dataset is created from YouTube and authors provide the links to the videos and do not release the data directly. But since YouTube videos can change and disappear, how is this dataset useful? In the future if someone wants to use this data, and reproduce results, how would they do that?\n2. It is not clear if the authors took the permission from content creator of the ASL YouTube video, there can be copyright issues. \n3. A more thorough error analysis of the model is missing. BLEU scores are not that high, it would be interesting to see why is that the case and how it could be addressed. ", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Authors create the largest ASL dataset with largest number of unique signers.\n2. Authors perform baseline experiments to show that having a larger dataset helps to improve the results. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "GYm2ANyr3s", "reviewer": "Reviewer_t1a9", "summary": "", "questions": "", "limitations": "The authors discussed the limitations and future wok in the conclusion. ", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The dataset is quite large, for which the number of singers is 2519, at least 10x as the existing largest datatset, and the scope of vocabularies as well as the length of hours all reach the largest values. \nThe authors discusses the properties the dataset in sufficient details. \nThe evaluation is conducted in a reasonable way, which is compared on the test data from How2Sign. By this way, the authors also compared the pretraining mode and zero-shot mode, showing the generalization ability. \n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper presents YouTube-ASL, the largest (as of now) American Sign Language to English captions dataset. The dataset is created from curating videos on YouTube and contains 1’000 hours of videos from 2’500+ unique signers.
The authors show the validity of the dataset by training a simple model on the dataset and show a 12.3 BLEU points improvement over the baseline.

There is a strong variance between reviewers, and some limitations have been noted: only the video ids are being provided (which is very common on web-crawled dataset), no code is provided to reproduce the ressource in other languages, the baseline is quite simple and more up-to-date approaches could have been used (such as adapting a pre trained model on YT-ASL).

While I agree with the limitations presented by the reviewers, this is an under-investigated area with significant societal value, which would benefit from the spotlight of the conference.

Recommendation: poster accept
Decision: Accept (Poster)