YouTube-SL-25: A Large-Scale, Open-Domain Multilingual Sign Language Parallel Corpus

## Abstract

Abstract Even for better-studied sign languages like American Sign Language (ASL), data is the bottleneck for machine learning research. The situation is worse yet for the many other sign languages used by Deaf/Hard of Hearing communities around the world. In this paper, we present YouTube-SL-25, a large-scale, open-domain multilingual corpus of sign language videos with seemingly well-aligned captions drawn from YouTube. With > > > 3000 hours of videos across > > > 25 sign languages, YouTube-SL-25 is a) > > > 3x the size of YouTube-ASL, b) the largest parallel sign language dataset to date, and c) the first or largest parallel dataset for many of its component languages. We provide baselines for sign-to-text tasks using a unified multilingual multitask model based on T5 and report scores on benchmarks across 4 sign languages. The results demonstrate that multilingual transfer benefits both higher- and lower-resource sign languages within YouTube-SL-25. 1 1 1 We release the YouTube-SL-25 video IDs at this link .

## 1 Introduction

There are [>300 sign languages](https://www.un.org/en/observances/sign-languages-day) used by Deaf/Hard of Hearing communities around the world. As minority languages, sign languages have relatively little data available and therefore—like low-resource spoken languages—are challenging to process with machine learning. The fact that sign languages are visuospatial languages represented as video with no commonly used written form creates extra challenges at every stage of development: they are more difficult to mine, filter, preprocess, and model. Datasets like YouTube-ASL [] for American Sign Language (ASL) and BOBSL [] for British Sign Language (BSL) make a focused effort to advance the status quo for a single sign language, but many of the world’s sign languages are being left behind.

In this paper, we present YouTube-SL-25, a large-scale, open-domain corpus of multilingual sign language videos with seemingly well-aligned captions, primarily intended for translation from each sign language to its region’s spoken language. By open-domain, we are distinguishing from datasets like AfriSign [] and JWSign [], which have made strides towards massive sign-multilinguality but feature a limited domain (e.g., Bible translations).

We mined these videos using a two-step process: First, like YouTube-ASL [], we used automatic classifiers on text metadata to identify potentially relevant videos. And second—in contrast to YouTube-ASL, which paid 3 native ASL signers to filter out individual videos with wrong or misaligned captions over the course of several months, a process that isn’t amenable to scaling across many sign languages—we used our own knowledge of sign languages and YouTube data to triage the videos over four days, ranking the priority of channels according to total hours of content and then auditing videos per channel, with particular attention to outliers by duration. This means that the annotations were performed with less expertise than YouTube-ASL, but in practice there are many signals that can be used to identify high-quality content even without complete understanding, as observed in work with written languages [, ].

The result is a dataset with 3207 hours of videos with seemingly well-aligned captions, featuring $>$3000 unique signers across $>$25 sign languages. This is $>$3x the size of YouTube-ASL (984 hours), which is a subset of YouTube-SL-25, and larger than the prior largest parallel sign language dataset, JWSign (2530 hours), which is closed-domain. YouTube-SL-25 is the first or largest parallel dataset for many of its component sign languages.

Following Kreutzer et al. []’s call to avoid “representation washing”—i.e., overstating the number of low-resource languages supported when their dataset size is marginal—we name YouTube-SL-25 after the $>$25 sign languages that have at least 15 hours of representation in it. This is roughly the minimum size of datasets that have previously been released for individual sign languages, but it is still considered extremely low resource. When including the long tail, YouTube-SL-25 includes data in at least 55 sign languages.

We provide baselines for YouTube-SL-25 on sign language understanding tasks, extending the multitask mixture from FLEURS-ASL’s baselines [] to support multiple source/target languages and the sign language identification task (which is extremely understudied [, , ]). Our results on benchmarks across 4 sign languages show that both higher- and lower-resource sign languages within YouTube-SL-25 benefit from multilingual transfer.

We publicly release the YouTube-SL-25 video IDs at [this link](https://github.com/google-research/google-research/tree/master/youtube_sl_25). We hope that YouTube-SL-25 can be further refined by language experts in the community and will be useful for a variety of purposes, such as general sign language pretraining and medium-quality finetuning for downstream tasks like translation, caption alignment, & sign language identification. In particular, we think that it is necessary to develop robust filtering and preprocessing tools for sign languages in order to realize the next order of magnitude increase in dataset size.

Figure: Figure 1: [World map](https://commons.wikimedia.org/wiki/File:BlankMap-World-noborders.png) showing the amount of content in YouTube-SL-25 for each sign language; the area of each circle is proportional to the number of hours. The circle in the middle of the Atlantic Ocean represents International Sign. Observe that the dataset is especially lacking in representation for Central & South America, Africa, West & Central Asia, and China & Southeast Asia.
Refer to caption: x1.png

## 2 The YouTube-SL-25 Corpus

YouTube-SL-25 is a massively multilingual corpus of sign language videos with seemingly well-aligned captions drawn from YouTube, intended primarily for pretraining sign-to-text translation models. YouTube-SL-25 is a superset of YouTube-ASL [].

YouTube-ASL used a two-step pipeline to construct its corpus: “first, retrieval using automatic content-based annotations, and second, filtering by skilled human annotators at a per-video level.” We adopt the first step with minimal modifications, but use a lower standard for the second step: triaging/auditing by a single non-native signer to identify seemingly high-quality channels for inclusion (even without full understanding of the content in every sign language).

### 2.1 Automatically Retrieving Candidate Videos

We retrieved candidate videos in a similar way to YouTube-ASL [], by fetching videos tagged with Knowledge Graph entities related to sign language generally or any individual sign language with its own tag as of July 2023.(^2^22We refreshed the dataset in May 2024 to include newer videos from accepted channels, with some light manual review.) As with YouTube-ASL, the recall of these tags is limited in that they are not aware of sign language in the video content itself (i.e., they do not use sign language detection and sign language identification classifiers), so they may miss videos or channels that use sign language but do not explicitly mention it. The pool of videos that we fetched from is also restricted in some generic ways, such as that the videos must be public and listed.

We applied the same filtering steps as YouTube-ASL: select only videos with manually uploaded captions, and remove videos with duration $<$10 seconds or $>$5 hours, width $<$480 pixels or height $360$ pixels, and frame rate $<$15fps or $>$60fps. We added one additional step: remove videos where captions cover $<$40% of the duration.

Unlike YouTube-ASL, we did not exclude videos of conversations with more than one signer in frame at a time. While our pose-based baselines only support one person’s input, we feel that there is no reason to exclude these videos from the corpus itself. We are not aware of any prior works that study sign language translation from videos with multiple signers; some datasets like the Public DGS Corpus [] consist of conversational data, but they are recorded with one camera per signer. Being able to handle this kind of data is important both in the near term to make the most of the limited resources available and in the long term to support multi-signer applications.

The result of this process was a list of 81,623 candidate videos that might contain signed content with high-quality captions.

### 2.2 Triaging Candidate Videos with Coarse Manual Annotations

YouTube-ASL used 3 native ASL signers as annotators to identify which of the individual candidate videos contained ASL and had high-quality, well-aligned English captions, labelling videos over the course of several months and several rounds of iteration on the annotation tool/labelling standards. This kind of annotation does not scale well to massively multilingual data because it is difficult and expensive to onboard native (or even just proficient) signers to annotate videos in tens of (extremely low resource) sign languages. Instead, we follow Kreutzer et al. []’s observation that many data filtering tasks can be done even without full understanding of the content. Therefore, the first author (a non-native hearing signer with experience in several sign languages) served as the annotator, triaging the corpus over the course of a four-day weekend using a combination of per-channel annotations and targeted audits. The triage proceeded as follows:

First, we automatically included the YouTube-ASL videos in our corpus and removed these from the manual review. Then we grouped videos by channel and sorted by total duration in descending order, with random ordering for videos within each channel. We used heuristic classifiers based on public text metadata for each video and channel (e.g., title, description, caption language) to provisionally label each video with a sign language. For each channel, the author sampled several random videos and judged whether they matched the acceptance criteria (described below), then corrected the sign language label if relevant. If all the sampled videos were good, the channel was tentatively accepted; if the videos were bad on balance, it was rejected; if it was mixed but promising, the author investigated further and was able to exclude individual bad videos. The author triaged the first 1000 channels of 18000 (down to about 1 hour of content per channel), then continued into the 3000-6000 range, skipping videos predicted to be in ASL in order to prioritize the long tail of sign languages.(^3^33An observation: the acceptance rates for videos in different sign languages were dramatically different. For example, the vast majority of content from Central & South America was live-interpreted events with small interpreters and poor caption alignment. We expect that high-quality produced content is more common in high-income countries, so it will be important for future work to leverage lower quality data in order to be more globally inclusive.) Finally, we sorted the videos for each channel by duration in descending order and the first author made a second pass, checking outlier videos with unusually long durations for their channels(^4^44Sometimes channels with high-quality produced content with relatively consistent duration $\sim$10 minutes would have long live-interpreted events with different translation and caption quality.) to reduce heavy-hitter errors.

Now, a description of our annotation standard for individual videos and some shallow signals to recognize these features even without full understanding:

- •
The video should feature sign language. While there is such a thing as [fake sign language interpreters](https://www.nbcnews.com/news/world/fake-sign-language-interpreter-nelson-mandela-memorial-provokes-anger-flna2d11723934), they do not typically create YouTube channels full of themselves gesturing nonsensically, so there is not much adversarial content to sift through. The closest thing to this is inexperienced sign language learners, especially producing loose translations of music.(^5^55We accept learners as long as they seem relatively competent. Models should be able to understand less proficient signers, especially given that many deaf people are not exposed to sign language in childhood and reach varying levels of proficiency []. Less proficient signing can be easily distinguished from native signing, so it should not harm the quality of sign language understanding (especially if the data is only used for pretraining), but more care is required for generation.) An annotator proficient in one sign language can generally distinguish these cases (and others like interpretive dance/gesture) even for other sign languages.
- •
The signer(s) should be the primary content. If a signer is interpreting spoken content, the video of the speaker (if present) should be significantly smaller than the signer. If the person is switching off between signing and speaking, or a speaker is the primary content for a significant fraction of the video, it should be rejected.
- •
The video should have relatively well-aligned captions. There are a few signals you can use for this: if the video has no speech track, the captions are virtually always well-aligned;(^6^66These are typically channels with natively Deaf-produced content.) you can validate this by seeing whether the changes in captions occur at boundaries in sign language phrases (which are marked pretty universally with pauses and nonmanual signals like head tilt or eyebrow movement). If there is a speech track, you can listen to whether the caption timing matches the speech track. We accepted videos where the captions were aligned to the speech track as long as they were not off by more than a couple of seconds; usually this corresponds to an after-the-fact voiceover translation, not live interpretation.
- •
Label which sign language the video is in. It was almost always possible to classify the sign language with high confidence based on the video title/description and caption language, or at worst the channel’s description (including its stated country). The failures in automatic classification mostly came from International Sign, often specified as just “IS”, which was not automatically tagged. In the rare cases where we could not determine the language with high confidence, we marked the sign language as unknown.

In order to sanity check the end-to-end quality of this triage process, the first author ran two additional audits on the final dataset (excluding the videos from YouTube-ASL, since we were already confident in their quality). The author rated two samples of 100 videos, one sampled uniformly per video id, and the other with videos sampled with probability proportional to their video duration. The author loaded each video and scrubbed through the frame previews across time to identify cuts where content and captioning habits may change.

In the sample with random videos by ID, we found: 1 video that strictly should not have been included in the dataset (a brief video with captioned narration), 1 borderline video with a waist-up interpreter next to a video of a speaker’s face only, and 3 videos with sign-along versions of songs that used appropriate vocabulary but didn’t really provide a complete translation.

In the sample proportional to video duration, we found: 0.76% of content that was speech rather than signing (from a channel with otherwise entirely signed content), and 0.96% that consisted of interstitials mixed within otherwise good content, where the interstitials had signing but no captions.

We consider these error rates acceptable.

Figure: (a) Monk skin tone ratings []. Dark skin is globally underrepresented; some of this is due to the regional bias towards the Global North.
Refer to caption: x2.png

### 2.3 Corpus Statistics & Comparison to Prior Datasets

The final human-triaged YouTube-SL-25 corpus consists of 39197 videos totaling 3207 hours of content (covered by 2980 hours of captions = 2.16M captions = 104M characters)(^7^77We used classifiers to evaluate whether the captions were toxic, hateful, violent, and sexual with confidence ¿0.8, which flagged ¡1% of captions. We manually audited the captions flagged with highest confidence and none merited exclusion from the dataset: the “toxic” and “hateful” content were false positives, the “violent” content was news about violent events, and the “sexual” content was news or educational content about sex.) across $>$25 sign languages.mom Like in YouTube-ASL, we approximately lower bound the number of unique signers using the number of unique channels, 3072, and expect that this is quite an underestimate as channels (or even individual videos) featuring multiple signers are common. This is only somewhat larger than YouTube-ASL’s 2519 channels, which is the main downside of our triage-based annotation approach: it captures the bulk of the hours but not the long tail of unique channels.

As discussed in Uthus et al. []’s related work section, to the best of our knowledge YouTube-ASL and now YouTube-SL-25 are the only sign language translation datasets that aggregate content from a wide variety of automatically mined channels, as opposed to others which either pay participants to create new data [, , , ] or curate content from a small number of content creators [, , , , , , ]. This has implications for diversity in topics/content and video production style.(^8^88Note that even though YouTube-SL-25 is open-domain overall, there may be discretization effects in the long tail of languages, where a language is represented by only a few channels and therefore in effect is closed-domain.)

See Figure 1 for a depiction of the relative weight of each language on a world map and Table 1 for a breakdown of hours by language, comparing to the largest prior parallel dataset for each language (both open- and closed-domain). Note that prior works have studied sign languages with no representation in YouTube-SL-25, like AfriSign [] & JWSign [] (primarily languages in Central & South America and Africa), and CSL-Daily [] (China); these languages aren’t included in the table. The distribution of language data can be attributed to population size, economic development, and political factors. Other supplementary sources of data, such as Bible translations [, ] and national interpreted broadcasts [, ], could provide more balanced data distributions.

See Figure 2 for an estimated demographic breakdown of YouTube-SL-25 by skin tone and perceived gender presentation, predicted by automated classifiers. Dark skin is globally underrepresented in the dataset, with only 1.9% (60 hours) of the data at Monk 6-10. Some of this can be explained by overrepresentation of countries in the Global North (and the only two countries in the Global South with a significant fraction of the dataset, Brazil and India, have substantial variation in skin tone). Ensuring demographic fairness remains an important problem for future work.

**Table 1: Hours of content in YouTube-SL-25 across sign languages, compared to the largest prior publicly available parallel dataset for each language (both open- and closed-domain). Some of these sign languages have more commonly used names, like “Libras” for “Brazilian SL”, but we refer by region here for convenience. Note that International Sign (ils) is a pidgin that arises at international Deaf conferences, in contrast to all the other sign languages on this list, which are complete natural languages. With the exception of YouTube-ASL, most prior datasets consist of mutually exclusive data that could supplement YouTube-SL-25.**
| sign language | iso 639 | #videos | #channels | #hours | largest prior<br>(open) | #hours | largest<br>prior | #hours |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| American | ase | 16724 | 2523 | 1394 | YouTube-ASL [] | 984 | " | " |
| International | ils | 1634 | 14 | 285 | - | - | - | - |
| Indian | ins | 3023 | 8 | 209 | ISLTransate [] | 55 | " | " |
| Polish | pso | 1698 | 34 | 137 | - | - | JWSign | 61 |
| German | gsg | 1024 | 65 | 108 | DGS Corpus [] | 50 | " | " |
| Brazilian | bzs | 846 | 30 | 101 | - | - | JWSign | 211 |
| British | bfi | 1026 | 60 | 74 | BOBSL [] | 1660 | " | " |
| Hungarian | hsh | 1687 | 9 | 70 | - | - | JWSign | 16 |
| Australian | asf | 1098 | 25 | 67 | Auslan-Daily [] | 45 | " | " |
| Italian | ise | 929 | 21 | 63 | - | - | JWSign | 115 |
| Japanese | jsl | 1075 | 21 | 62 | - | - | JWSign | 67 |
| Russian | rsl | 715 | 9 | 60 | - | - | JWSign | 111 |
| French | fsl | 900 | 23 | 49 | Mediapi-RGB [] | 86 | " | " |
| Korean | kvk | 325 | 10 | 38 | - | - | JWSign | 93 |
| Colombian | csn | 212 | 4 | 37 | - | - | JWSign | 128 |
| Spanish | ssp | 701 | 13 | 36 | - | - | JWSign | 85 |
| Dutch | dse | 450 | 14 | 35 | - | - | JWSign | 1 |
| Argentine | aed | 160 | 9 | 34 | - | - | JWSign | 107 |
| Quebec | fcs | 128 | 7 | 26 | - | - | JWSign | 20 |
| Catalan | csc | 239 | 7 | 26 | - | - | - | - |
| Pakistani | pks | 581 | 1 | 25 | - | - | - | - |
| Swedish | swl | 410 | 23 | 22 | - | - | JWSign | 14 |
| Turkish | tsm | 189 | 8 | 18 | E-TSL [] | 24 | " | " |
| Swiss German | sgg | 185 | 6 | 18 | SRF23 [] | 437 | " | " |
| Israeli | isr | 345 | 14 | 17 | - | - | JWSign | 1 |
| Fenno-Swedish | fss | 174 | 8 | 16 | - | - | - | - |
| Finnish | fse | 196 | 13 | 14 | Corpus FinSL [] | 15 | JWSign | 35 |
| Austrian | asq | 177 | 11 | 13 | - | - | JWSign | 1 |
| Taiwan | tss | 130 | 5 | 13 | - | - | JWSign | 14 |
| Czech | cse | 185 | 11 | 12 | - | - | JWSign | 18 |
| Icelandic | icl | 214 | 6 | 11 | - | - | - | - |
| Irish | isg | 34 | 2 | 10 | - | - | JWSign | 2 |
| Slovenian | - | 139 | 6 | 10 | - | - | JWSign | 1 |
| Indonesian | inl | 126 | 2 | 9 | - | - | JWSign | 30 |
| Jordanian | jos | 132 | 2 | 9 | - | - | - | - |
| Norwegian | nsl | 82 | 9 | 8 | - | - | JWSign | 1 |
| Singapore | sls | 102 | 5 | 8 | - | - | JWSign | <1 |
| Thai | tsq | 106 | 1 | 7 | - | - | JWSign | 8 |
| Hong Kong | hks | 117 | 9 | 6 | TVB-HKSL [] | 14 | " | " |
| Lithuanian | lls | 160 | 6 | 6 | - | - | - | - |
| Slovakian | svk | 59 | 4 | 4 | - | - | JWSign | 14 |
| Chilean | csg | 74 | 5 | 4 | - | - | JWSign | 100 |
| New Zealand | nzs | 79 | 8 | 4 | - | - | JWSign | 2 |
| Croatian | csq | 24 | 2 | 4 | - | - | JWSign | 5 |
| Mexican | mfs | 53 | 3 | 4 | - | - | JWSign | 184 |
| Philippines | psp | 26 | 2 | 3 | - | - | JWSign | 21 |
| Vietnamese | - | 36 | 1 | 3 | - | - | JWSign | 3 |
| Peruvian | prl | 71 | 4 | 2 | - | - | JWSign | 81 |
| Flemish | vgt | 57 | 4 | 2 | VGT-RAW [] | 100 | " | " |
| Swiss French | ssr | 33 | 2 | 2 | Signsuisse [] | 13 | - | - |
| Greek | gss | 22 | 1 | 2 | Elementary23 [] | 71 | " | " |
| French Belgian | sfb | 25 | 5 | 2 | - | - | JWSign | 3 |
| Swiss Italian | slf | 30 | 2 | 1 | SwissSLi [] | 10 | - | - |
| Danish | dsl | 21 | 5 | 1 | - | - | JWSign | 1 |
| Estonian | eso | 12 | 6 | 1 | - | - | JWSign | 1 |

## 3 Baselines

We demonstrate the value of YouTube-SL-25 with baselines for sign-to-text translation and sign language identification across benchmarks for 4 sign languages.

### 3.1 Setup

We build on the unified modeling approach adopted by the FLEURS-ASL baselines [], which are an evolution of the YouTube-ASL baselines []. In brief, this approach finetunes a pretrained encoder-decoder language model (T5-v1.1 Small) to take as input 256 tokens of text (control tokens and caption context) and 512 frames of MediaPipe Holistic landmarks at half frame rate (corresponding to a random video span), and output 256 tokens of text (including timestamps).

The only modification we make is to extend the mixture to support multiple source and target languages, as well as the sign language identification task with probability 0.1 (where it classifies the language and then proceeds to translate the input). Like in FLEURS-ASL, we train first on caption-level clips and then an equal mixture of caption-level and random clip-level data. See Figure 3 for a depiction of the modifications to the control token format.(^9^99We tried to change the pretrained model from T5-v1.1 Small [] to mT5 Small [] so that languages besides English could benefit from pretraining and better tokenization, but in initial experiments mT5 took about 1/3 more steps to converge and achieved worse results. We therefore ran the full set of experiments with T5 only.)

We compare three models: T5 with only text pretraining (i.e., the original pretrained T5 Small), T5 with continued training on YouTube-SL-25’s 1400-hour ASL subset, and T5 with continued training on the full YouTube-SL-25 dataset. See Appendix A for training details.

Figure: Figure 3: Unified document-level sign-to-text training, extended for multilinguality; modified from Figure 2 of Tanzer []. New additions circled in red. For caption alignment, source and target language are provided unconditionally. For translation, source and target language are provided w.p. 0.9 and predicted w.p. 0.1 (only when text context is not included). We avoid conditioning on the target language (including as text context) without the source language because each source language generally has one target language, making the language identification task easier or trivial.
Refer to caption: x4.png

### 3.2 Comparison to Prior Methods

See FLEURS-ASL [] for discussion of how their baseline approach (in particular, training on random clips and caption tracks rather than tightly clipped individual captions) relates to prior modeling work. Insofar as YouTube-SL-25 is slightly noisier than YouTube-ASL, we slightly push the boundaries in terms of training on weakly aligned data. With respect to our modifications: modeling multiple sign languages in one model is not new [, , , ]. We are not aware of any prior works that train sign language identification as a task within translation models (the few prior works on sign language identification [, ] use low-level phonetic features only), but this is not especially novel given its use in Whisper for speech input [].

### 3.3 Results

We provide sentence-level translation (scored with BLEURT []) and sign language identification (scored with top-1 accuracy) results on benchmarks for 4 sign languages: American Sign Language (ase,
FLEURS-ASL [] &
How2Sign []), Swiss German Sign Language (sgg, WMT23 SignSuisse []), Swiss French Sign Language (ssr, WMT23 SignSuisse []), and Swiss Italian Sign Language (slf, WMT23 SignSuisse []). We provide both zero-shot scores and finetuned scores where relevant.(^10^1010For translation, the model is separately finetuned for each dataset, checkpoint selected based on BLEU on the validation set. For sign language identification, zero-shot scores mean that the model is briefly finetuned on YouTube-SL-25 rebalanced to the 4 sign languages with equal weight, and finetuned scores mean that the model is finetuned on an equally weighted mixture of the benchmarks’ training sets. We don’t finetune on FLEURS-ASL, so the finetuned langid scores are after finetuning on How2Sign.)

Table 2 shows quantitative results on both tasks, and Table 3 shows a sample of qualitative translation examples. We see that on both translation and lang id, sign language pretraining is substantially better than none (as in Uthus et al. []) and multilingual transfer helps both the higher-resource sign language (ASL) and the lower-resource sign languages within YouTube-SL-25.(^11^1111Some prior works [, ] did not see strict improvement from modeling additional sign languages; we expect that this was a model capacity issue (for pretraining, or finetuning on multiple languages simultaneously rather than one at a time).) As expected, finetuning gives large gains—especially when the language is poorly represented in YouTube-SL-25—and benefits from multilingual pretraining. For example, our finetuned sgg result of 37.7 BLEURT (7.5 BLEU) far exceeds the top WMT23 score of 23.6 BLEURT (0.3 BLEU).

Note that for sign language identification, finetuning is somewhat ill-conceived because some datasets have train/test signer overlap or just generally unique recording conditions that could be used a shortcut. For example, langid quality on FLEURS-ASL decreases when pretrained on ASL data and finetuned on How2Sign, even compared to no sign language pretraining—with accuracy varying dramatically across the 3 signers within the test set, averaging 18.3% vs. 94.9% vs. 77.0%. This suggests that different kinds of pretraining may cause different features to be surfaced during finetuning, which may overfit or not transfer with respect to broader domains. Our zero-shot results are nontrivial despite the severe class imbalance. A proper sign language identification benchmark needs more consistency across languages; we were unable to use SP-10 [] for licensing reasons.

**Table 2: Baseline results for sentence-level translation and language identification on benchmarks for 4 sign languages. Translation is measured with BLEURT, lang id with top-1 accuracy: we report zero-shot / finetuned scores where relevant. We compare the results of no sign language pretraining vs. pretraining on the ASL subset of YouTube-SL-25 vs. the full YouTube-SL-25 dataset.**
|  | ase | sgg | ssr | slf |  |
| --- | --- | --- | --- | --- | --- |
| pretrain set | FLEURS-ASL | How2Sign | WMT23 SignSuisse |  |  |
| translation |  |  |  |  |  |
| None | -    / - | -   / 31.2 | -  / 26.2 | -  / 7.2 | -  / 22.8 |
| YT-SL-25 (ASL) | 33.7 / - | 30.2 / 46.6 | 9.0 / 27.9 | 5.4 / 11.1 | 9.3 / 20.8 |
| YT-SL-25 (Full) | 40.1 / - | 29.7 / 47.9 | 9.5 / 37.7 | 6.8 / 18.8 | 9.3 / 25.2 |
| lang id |  |  |  |  |  |
| None | -    / 98.4 | -    / 99.6 | -   / 19.6 | -  / 98.8 | -  / 87.6 |
| YT-SL-25 (ASL) | 100.0 / 66.7 | 89.7 / 99.9 | 54.0 / 89.6 | 6.0 / 100.0 | 1.6 / 99.6 |
| YT-SL-25 (Full) | 100.0 / 99.9 | 92.7 / 100.0 | 72.0 / 99.6 | 0.4 / 100.0 | 64.8 / 99.6 |

**Table 3: Qualitative examples for sentence-level translation across datasets. Examples selected randomly without cherrypicking (from examples already highlighted in previous papers if available).**
| language | setting | text |
| --- | --- | --- |
| ase | reference | And that’s a great vital point technique for women’s self defense. |
|  | prediction | It’s a great point for women to self-protection. |
| sgg | reference | Dieses Buch enthält ein Vorwort, welches sich Präambel nennt. |
|  |  | (This book contains a foreword called a preamble.) |
|  | prediction | Dieses Buch hat eine Wörtchenschilde. |
|  |  | (This book has a word shield.) |
| ssr | reference | Le fiancé de mon amie va bientôt s’engager dans l’armée. |
|  |  | (My friend’s fiancée is going to join the army soon.) |
|  | prediction | Mon amie est à l’armée. |
|  |  | (My friend is in the army.) |
| slf | reference | Lei è allergica alle punture d’api. |
|  |  | (She is allergic to bee stings.) |
|  | prediction | Lei ha un’ape migliore. |
|  |  | (She has a better bee.) |

## 4 Limitations

As discussed in Section 2, each step of the corpus curation comes with limitations: the automatic tagging step misses content in sign language that does not mention sign language (in the video itself or in metadata) (§ 2.1), the manual triage step trades off quality and small-channel exhaustiveness for annotation effort (§ 2.2), and the result has issues with representativeness—the distribution of languages and skin tone in particular (§ 2.3). The dataset is also small overall in comparison to MT datasets for spoken languages; we expect that more data will be needed to reach usable translation quality in generality, but this may be sufficient for more narrowly scoped tasks. Because YouTube-SL-25 (like YouTube-ASL) has so much signer variety (appearance, recording environment, proficiency, and now also number of signers in frame at a time), it may be difficult to use it to train consistent sign language generation models.

Our baselines are limited in that we just use FLEURS-ASL’s multitask training mixture as a way to pretrain on slightly noisy data and don’t engage with the discourse-level or timestamp tasks it enables. It is difficult to evaluate these tasks (plus sign language identification) on a collection of disparate benchmarks that primarily focus on sentence-level translation; this is on top of the fact that there are not enough benchmarks (especially ones with suitable licensing) to evaluate the vast majority of sign languages in YouTube-SL-25. This underscores the importance of extending FLEURS-ASL to even more sign languages.

## 5 Conclusion

In this paper, we presented YouTube-SL-25, a multilingual, open-domain corpus of sign language videos with seemingly well-aligned captions, with $>$3000 hours of content across $>$25 sign languages. We achieved this efficiently without hiring tens of language-specific annotators by performing a triage grouped by channel and sorted by duration of content; this comes at the expense of the long tail of unique signers and some quality assurance. We demonstrated the value of YouTube-SL-25 with experiments in sentence-level translation and sign language identification which demonstrate multilingual transfer benefits both higher- and lower-resource sign languages. We hope that YouTube-SL-25 and our account of how we curated it will serve as a foundation for research towards the ultimate goal of making technology inclusive for Deaf/Hard of Hearing signers worldwide.

## Ethics Statement

The ethical considerations of this work are very similar to our prior release of YouTube-ASL. We release publicly available YouTube videos only as video IDs so that deletions are automatically reflected in the corpus. We train our baseline models on MediaPipe Holistic skeletons as a form of anonymization. Another work that uses YouTube-ASL, Rust et al. [], explores an orthogonal approach for anonymization in direct video modeling: pretraining with the face blurred and finetuning with it unblurred on separate data. The field should continue to study this topic to develop a better understanding of the tradeoffs of different approaches for responsible use of the data.

While we provide estimated demographic breakdowns of the signers in YouTube-SL-25 along some dimensions, even if the dataset were globally representative, this would not guarantee that models trained on it would have equal performance across these attributes or others. And this intersects with sign multilinguality: models may perform worse for some demographics only in some sign languages due to cross-sectional effects. For example, sign language identification may be biased by the signer’s race as a proxy for nationality (itself a proxy for language) due to correlations in the training distribution. It is important to evaluate models in the context of their specific intended use cases to ensure that they robustly deliver on being useful for Deaf/Hard of Hearing users and avoid the trap of overpromising/underdelivering that has plagued sign language technology historically.

## Acknowledgements

We thank Manfred Georg and Caroline Pantofaru for institutional support; Anelia Angelova and Chris Dyer for feedback on drafts of this paper; David Uthus for general help with infrastructure; and Krishna Somandepalli, Katie Zhang, & Simon Wang for assistance on the fairness analysis.

## Appendix A Training Details

We train all of our models with Adafactor [] with base learning rate 0.001.

We pretrained our two T5v1.1 Small models (on Youtube-SL-25’s ASL and Full sets) on 64 TPUv3s for 210k and 430k steps respectively (switching from pure caption-level training to 1:1 caption-level:random clip-level training once the model appeared to have converged, then stopping again after re-convergence, both according to BLEU on the How2Sign val set, like in FLEURS-ASL []). Each 1k steps took about 8 minutes to train. We also pretrained an mT5 Small model for about 600k steps, which was underperforming so we didn’t run the complete set of experiments for it.

We finetuned the sentence-level translation models on 16 TPUv3s with a batch size of 32 until convergence; this took about 10k steps for WMT23 SS DSGS and at most 2.5k steps for the other datasets.

We finetuned the language identification models on a mixture of data for the four languages with equal weight, either subsets of YouTube-SL-25 (for zero-shot results) or the training splits from the downstream benchmarks (for finetuned results). We used 16 TPUv3s with a batch size of 32 until convergence, with up to 3k steps.

## Appendix B BLEU Scores

See Table 4 for sentence-level translation BLEU [] scores, rather than BLEURT from Table 2 in the body of the paper.

**Table 4: Baseline results for sentence-level translation on benchmarks for 4 sign languages, measured in BLEU (zero-shot / finetuned).**
|  | ase | sgg | ssr | slf |  |
| --- | --- | --- | --- | --- | --- |
| pretrain set | FLEURS-ASL | How2Sign | WMT23 SignSuisse |  |  |
| None | - / - | - / 2.6 | - / 2.2 | - / 1.3 | - / 1.1 |
| YT-SL-25 (ASL) | 3.4 / - | 3.6 / 14.5 | 0.1 / 5.4 | 0.4 / 5.9 | 0.1 / 4.0 |
| YT-SL-25 (Full) | 4.4 / - | 4.2 / 15.4 | 0.5 / 7.5 | 1.1 / 7.0 | 0.1 / 5.2 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Based on the provided excerpts from the paper "YOUTUBE-SL-25: A LARGE-SCALE, OPEN-DOMAIN MULTILINGUAL SIGN LANGUAGE PARALLEL CORPUS" and our previous conversation, here are some questions and suggestions for the authors:
Data Curation and Annotation:
1. Clarify the rationale for the 15-hour threshold for language inclusion. The paper mentions including languages with at least 15 hours of representation, aligning with the minimum size of previous datasets. However, the authors also acknowledge this as "extremely low resource". Further explanation of the trade-offs considered in setting this threshold and how it balances representativeness with data scarcity would be beneficial.
2. Elaborate on the specific challenges encountered during the manual triage process. The paper provides a high-level overview of the triage procedure but could benefit from a more detailed discussion of the practical difficulties faced. For instance, were there specific sign languages that proved particularly challenging to identify or assess for caption alignment? Providing insights into these challenges would shed light on the complexities of working with diverse and low-resource sign language data.
3. Address the potential for bias introduced by the single annotator. While the paper acknowledges this limitation, a more in-depth discussion of mitigation strategies would strengthen the analysis. For example, could the authors elaborate on the annotator's background and expertise in different sign languages? Could they discuss the feasibility of incorporating a second-level review by additional annotators, even for a subset of the data, to assess inter-annotator agreement and identify potential biases?

Dataset Analysis and Evaluation:
1. Expand the demographic analysis to encompass a wider range of factors. The current analysis focuses on skin tone and perceived gender presentation. However, considering other demographic variables, such as age, regional variations in signing, and disability intersectionality, would provide a more comprehensive understanding of the dataset's representativeness.
2. Consider including additional evaluation metrics that capture discourse-level features. The current evaluation focuses primarily on sentence-level translation accuracy. Incorporating metrics that assess the coherence, fluency, and expressiveness of translated output, particularly concerning discourse-level phenomena in sign language, would provide a more nuanced view of model performance.
3. Explore the potential of YouTube-SL-25 for sign language generation tasks. The paper primarily focuses on understanding tasks. 
Expanding the scope to include generation tasks could open up new research avenues. Discussing potential challenges and opportunities in this area, and perhaps presenting preliminary experiments on a subset of the data, would enrich the paper's contribution.

Generalizability and Future Work:
1. Discuss the generalizability of the findings to other sign language processing tasks. While the paper focuses on translation and language identification, addressing the dataset's potential for other tasks, such as sign language recognition, summarization, or sentiment analysis, would highlight its broader applicability.
2. Outline future directions for improving the corpus and addressing its limitations. This could include strategies for expanding the dataset to include more languages and signers, refining the annotation process, developing more robust noise filtering techniques, and exploring alternative data sources beyond YouTube.
These questions and suggestions aim to encourage a deeper exploration of the dataset's strengths, limitations, and potential impact. Addressing these points would contribute to a more robust and informative discussion
soundness: 3
strengths: Originality: 
The paper introduces YouTube-SL-25, a large-scale, open-domain, multilingual corpus of sign language videos with seemingly well-aligned captions. This is the largest supervised sign language dataset to date and the first or largest parallel dataset for many of its 25+ component languages. YouTube-SL-25 significantly expands upon YouTube-ASL, being over three times its size. The corpus creation methodology, while building upon prior techniques, innovates by utilizing a single annotator for manual triage, prioritizing channels by content duration. This approach allows for efficient curation, enabling the inclusion of a diverse array of sign languages. Additionally, the paper explores the novel integration of sign language identification as a task within translation models.

Quality: 
The paper presents a well-constructed and carefully curated dataset. The authors employ a two-step process, first using automatic classifiers to identify potentially relevant videos and then utilizing a manual triage process to ensure quality and alignment. While acknowledging the potential for limitations due to the single-annotator approach, the authors provide detailed descriptions of the annotation standards and conduct audits to assess and report error rates. The dataset's quality is further evidenced by the strong baseline results achieved in translation and language identification tasks.

Clarity: 
The paper is well-written and easy to follow. The authors clearly present their methodology, experiments, and findings using concise language and a logical structure. The paper effectively utilizes figures and tables to present data and illustrate key concepts. The authors provide a thorough literature review, contextualizing their work within the existing research landscape and citing relevant prior datasets and models.

Significance: 
The paper addresses a critical bottleneck in sign language processing: the scarcity of data, especially for less-studied sign languages. The creation and release of YouTube-SL-25 as an open resource has the potential to significantly advance the field by enabling the development and evaluation of more robust and inclusive sign language processing models. The paper's findings on multilingual transfer learning hold promise for improving the performance of sign language understanding systems across various languages, including those with limited resources. The dataset's size and diversity also make it a valuable asset for exploring novel architectures and training paradigms, potentially leading to breakthroughs in sign language recognition, translation, and generation
summary: This paper introduces YouTube-SL-25, a large-scale, open-domain, multilingual sign language video corpus with seemingly well-aligned captions. It features over 3,000 hours of videos spanning more than 25 sign languages, making it the largest supervised sign language dataset to date. The corpus surpasses the size of YouTube-ASL, a previous sign language corpus, by more than three times. 
Here are the key contributions of the paper:
1. Data Collection and Curation: The paper describes a two-step process for mining and filtering sign language videos from YouTube. First, videos tagged with sign language-related keywords are automatically retrieved. Then, a manual triage process is employed, where a single annotator identifies high-quality channels with well-aligned captions, prioritizing channels based on content duration
2. Multilingual and Open-Domain Focus: Unlike datasets like AfriSign and JWSign, which focus on specific domains (e.g., Bible translations), YouTube-SL-25 includes videos from a diverse range of topics and genres, making it "open-domain".
3. Baseline Models and Multilingual Transfer: The paper establishes baselines for sign language understanding tasks, utilizing a multilingual multitask model based on T5. Results indicate that multilingual transfer learning benefits both resource-rich and resource-poor sign languages within the corpus.
4. Addressing Data Bottleneck and Inclusivity: The paper aims to tackle the scarcity of data for sign language research, particularly for less-studied sign languages. It highlights the importance of developing robust filtering and preprocessing tools for sign language data to further expand dataset size and inclusivity.
5. Ethical Considerations: The authors acknowledge the ethical implications of working with videos featuring individuals and discuss their anonymization efforts using MediaPipe Holistic skeletons. They also emphasize the need for ongoing research to ensure responsible data usage and equitable model performance across different demographics
weaknesses: 1. Limited Evaluation Scope: While the corpus covers a wide range of sign languages, the baseline experiments presented in the paper only evaluate four sign languages. Expanding the evaluation to encompass a more diverse subset of the languages represented in YouTube-SL-25, especially those with fewer resources, would provide a more comprehensive understanding of the corpus's utility for multilingual transfer learning and the impact of data scale on model performance. This expanded evaluation could reveal specific challenges or opportunities associated with different sign languages, informing future research directions.
2. Single-Annotator Triage: The reliance on a single annotator for manual video triage, while efficient for corpus creation, raises concerns about potential biases and inconsistencies in the data selection process. The annotator's proficiency in certain sign languages and potential blind spots in recognizing subtle cues related to caption alignment or signer proficiency could introduce systematic biases. Involving multiple annotators, ideally with diverse linguistic backgrounds and sign language expertise, would enhance the reliability and objectivity of the triage process. This collaborative approach would help mitigate individual biases and ensure a more balanced representation of signing styles and dialects within each language.
3. Potential Data Noise: The authors acknowledge the potential for noise in the dataset, particularly related to caption quality and alignment. While they employ some filtering mechanisms and conduct audits to assess error rates, the inherent variability in YouTube content production and captioning practices poses a challenge. Developing more robust methods for detecting and filtering out noisy data, potentially leveraging techniques from automatic speech recognition or natural language processing, would improve the overall quality of the corpus. This could involve automated caption alignment algorithms, quality assessment metrics, and possibly even community-based validation initiatives to harness the collective expertise of the Deaf community.
4. Limited Demographic Analysis: The paper provides a basic analysis of signer demographics using automated classifiers for skin tone and perceived gender presentation. However, a more comprehensive and nuanced analysis of demographic representation, considering factors such as age, regional variations, signing styles, and disability intersectionality, is crucial for understanding potential biases and ensuring the inclusivity of models trained on the data. It would also be helpful to discuss the potential implications of these findings for the development of fair and unbiased sign language processing models.
5. Limited Focus on Sign Language Generation: While the paper demonstrates the corpus's usefulness for sign language understanding tasks, it does not address its potential for sign language generation. Developing and evaluating sign language generation models, particularly those capable of producing fluent and expressive signing, remains a challenging area of research. Leveraging YouTube-SL-25 for this purpose could involve exploring techniques such as sequence-to-sequence modeling, conditional variational autoencoders, and reinforcement learning to generate realistic and natural-looking signing.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see weaknesses for suggestions and questions.
soundness: 3
strengths: (1) The paper is well-organized and easy to follow.

(2) YouTube-SL-25 addresses the shortage of data for sign language research. This large-scale dataset serves as a valuable resource for tasks like translation, caption alignment, and sign identification.

(3) Models pretrained on YouTube-SL-25, compared to YouTube-ASL, show improvements across various sign languages, highlighting the effectiveness of this multilingual dataset.
summary: The paper introduces YouTube-SL-25, a multilingual corpus of links to sign language videos. This dataset is designed to support pretraining for sign language models, and fine-tuning for various downstream sign language tasks. The paper provides benchmarks of sign-to-text translation and sign language identification across 4 sign languages.
weaknesses: (1) While the dataset offers significant benefits, much of the technical training relies on the FLEURS-ASL baseline [1]. Clarifying the unique technical contributions of this paper would enhance its value.

(2) The dataset’s value is dependent on the availability of the videos. If these videos were to be removed, the overall impact of the paper would be significantly limited. To mitigate this risk, open-sourcing the framework used to filter sign language videos would benefit researchers in the long run and facilitate further development.

(3) Currently, there is no information on whether the pretrained models will be open-sourced. Providing access to these models would greatly benefit the research community, allowing other researchers to leverage large-scale pretrained models and explore new approaches in sign language tasks. This is especially important as links to the videos may become unavailable over time.

-----

[1] Garrett Tanzer. Fleurs-asl: Including american sign language in massively multilingual multitask evaluation, 2024.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 2
questions: # Figures

- Figure 3: Please clarify the meaning of the acronym "w.p." in the figure caption.

# Main text

- Page 2: Please provide exact figures for the amount of unique signers and sign languages, instead of >3000 and >25.
- Page 3: Can you clarify in what way the first author is "experienced" with multiple sign languages?
- Page 4: You mention in footnote 5: "We accept learners as long as they seem relatively competent". How was their competence evaluated? Is this competence evaluated by just one signer (the manuscript's first author), and how significant is this evaluation?
- Page 5: Please add a reference [15] for the Monk skin tone ratings to the main text (final paragraph) as well, not just the figure caption.
soundness: 3
strengths: This manuscript presents a valuable new dataset for the sign language machine translation community. The dataset is large and multilingual, and opens up new possibilities for sign language machine translation research in many previously underrepresented sign languages. As such, this manuscript is a valuable contribution to the field of sign language processing.

# Originality

This manuscript presents a labeled multilingual corpus for sign language translation. Previous corpora for sign language machine translation have often been monolingual (including the YouTube-ASL dataset, on which this manuscript builds). Research (in linguistics and in sign language processing) shows that sign languages have common elements, and therefore multilingual corpora can be beneficial for discovering and exploiting these common elements. The dataset is the first of its kind for several of the included low-resource sign languages and enables sign language translation research for these languages.

# Quality

The authors perform a partial analysis into the demographics of the collected data, in terms of skin tone and perceived gender.

# Clarity

The sections of the manuscript that discuss the dataset are clearly written and easy to follow. This is in contrast to the section on baseline methods, which is harder to follow (see "Weaknesses").

# Significance

A major problem in the domain of sign language translation is a lack of labeled data. The authors present a new dataset of unprecedented scale, with data for more than 25 languages and more than 3000 hours. Such resources are highly sought after and valued by the sign language processing community.
summary: The authors present YouTube-SL-25: a new dataset for sign language machine translation. This dataset is the first of its kind, in that it is significantly larger than predecessors and includes many languages for which previously, no labeled sign language machine translation data was available.

The dataset contains more than 3000 hours of video data for more than 25 sign languages, and was collected in two steps. First, YouTube was mined based on video tags related to sign language. Second, the authors (non-signers) performed a manual audit of the videos to identify high-quality video channels. Videos from those channels were used. An overview of the included sign languages, and relation to previous datasets for these sign languages (if any) is given in Table 1.

Key features of the dataset are as follows: it is open-domain and it includes videos with multiple signers in the frame. These features make it an important dataset for future research that will eventually lead to sign language machine translation systems that are usable in the real world.

A baseline method, adapted from previous work, is also presented. It is evaluated for four sign languages included in YouTube-SL-25. This method shows that multilingual training benefits both low-resource and (surprisingly) high-resource sign languages.
weaknesses: # Writing style and presentation

The authors make excessive use of footnotes (11 footnotes on 8 pages), which makes the text difficult to follow. I suggest moving some footnotes to the main text (3, 4, 6, 8, and 11 can definitely be in main text).

Figure 3 is quite complex and is nearly impossible to understand without first thoroughly reading the previous work from which it was adapted ([32]). I suggest adding an explanation in the figure caption, or, if constrained by the page limits, in appendix. This way, readers do not have to read and understand [32] to continue reading this manuscript past Figure 3.

# Methodology

The auditing of channels was performed by only one non-native hearing signer. The authors claim that this person is "experienced" in several sign languages, but they do not clarify what this experience entails and how this experience makes them a suited candidate for this auditing process. This leads to doubts about the robustness of the auditing process. Could the authors specify what "experience" means in this case? Having at least three auditors would also lead to a more robust auditing process.

The baseline method that is presented appears to be rather complex, especially compared to simple but effective sign language machine translation methods (e.g., https://arxiv.org/abs/2003.13830). It would be to the benefit of the reader if the method was simple, such that results are easier to interpret (the same argument is made by [33], which is cited by the manuscript's authors). Simplifying the baseline method would also give more room for analyzing results on more than just four sign languages and for further analyses of the results. Perhaps the authors could explain in the text why such a complex baseline method was chosen.

The results of the baseline method are discussed in limited detail. Scores are only provided in one translation metric (two, if including the appendix), and for the four evaluated languages, only one translation example is given per language. It would be interesting to see at least one good and one bad translation per language, and more than one if possible.

# Relation to previous work

The authors compare their dataset primarily to two previously released datasets:
- YouTube-ASL, because the same approach to data collection was used
- JWSign, because it is also a multilingual sign language machine translation dataset

The authors do not mention RWTH-PHOENIX-Weather 2014 T (https://www-i6.informatik.rwth-aachen.de/~koller/RWTH-PHOENIX-2014-T/) at all, despite it being a pivotal historical contribution in the field. Despite the limitations of PHOENIX 2014T (see, e.g., https://aclanthology.org/2022.wmt-1.71.pdf), I believe it is important to at least mention the dataset in passing to place this manuscript in the broader context of sign language machine translation.

The baseline method is adapted from [32], but there is little comparison to previous work in the field. The manuscript, in particular section 3, would benefit from an explanation as to why this method and these input features (MediaPipe) are used. If space permits, the authors should present a short section on related work in the design of sign language machine translation. If space is limited, they can refer to recent survey papers.

# Quality

On page 4, the authors mention "whether the changes in captions occur at boundaries in sign language phrases (which are marked pretty universally with pauses and nonmanual signals like head tilt or eyebrow movement)". It is my understanding that the boundaries of sign language phrases are a contentious topic in sign language linguistics, and that there certainly is no consensus on signals that are universal with respect or multiple sign languages. The authors should include a citation for this claim, or do not express this claim so strongly.

Meta-review: The article has received evaluations from three reviewers, all of which are positive. The authors have also provided detailed responses to the related issues. Therefore, the article will be accepted.

Decision: Accept (Poster)