YouTube-SL-25: A Large-Scale, Open-Domain Multilingual Sign Language Parallel Corpus
Authors: Garrett Tanzer
Biao Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: sign language translation
sign language
multilinguality
data curation
data auditing
Primary area: datasets and benchmarks
TL;DR: YouTube-SL-25: >3000 hours of parallel sign language videos across >25 sign languages

Abstract: Even for better-studied sign languages like American Sign Language (ASL), data is the bottleneck for machine learning research. The situation is worse yet for the many other sign languages used by Deaf/Hard of Hearing communities around the world. In this paper, we present YouTube-SL-25, a large-scale, open-domain multilingual corpus of sign language videos with seemingly well-aligned captions drawn from YouTube. With >3000 hours of videos across >25 sign languages, YouTube-SL-25 is a) >3x the size of YouTube-ASL, b) the largest parallel sign language dataset to date, and c) the first or largest parallel dataset for many of its component languages. We provide baselines for sign-to-text tasks using a unified multilingual multitask model based on T5 and report scores on benchmarks across 4 sign languages. The results demonstrate that multilingual transfer benefits both higher- and lower-resource sign languages within YouTube-SL-25.

Published conference paper at ICLR 2025
YOUTUBE-SL-25: A LARGE-SCALE, OPEN-DOMAIN MULTILINGUAL SIGN LANGUAGE PARALLEL CORPUS
Garrett Tanzer* Google DeepMind
Biao Zhang
Google DeepMind
ABSTRACT
Even for better-studied sign languages like American Sign Language (ASL), data is the bottleneck for machine learning research. The situation is worse yet for the many other sign languages used by Deaf/Hard of Hearing communities around the world. In this paper, we present Tub--bS-S-- large-scale, open-domain multilingual corpus of sign language videos with seemingly well-aligned captions drawn from YouTube. With 3300 hours of videos acToss 25 sign languages, SL-2u is a) >3x the size of Aou, b) the largest supervised sign language dataset to date, and e) the first or largest parallel dataset for many of its component languages. We provide baselines for sign-to-text tasks using unified multilingual multitask model based on T5 and report scores on benchmarks across 4 sign languages. The results demonstrate that multilingual transfer benefits both higher- and lower-resource sign languages within YouTube-SL-
INTRODUCTION
There are >300 sign languages used by Deaf/Hard of Hearing communities around the world. As minority languages, sign languages have relatively little data available and therefore-like low- resource spoken languages-are challenging process with machine learning. The fact that sign languages are visuospatial languages represented as video with no commonly used written form creates extra challenges at every stage of development: they are more difficult mine, filter, preprocess, and model. Datasets like YouTube-ASL [35] for American Sign Language (ASL) and BOBSI. [1] for British Sign Language (BSL) make focused effort to advance the status quo for single sign language, but many of the world's sign languages are being left behind.
In this paper, we present YouTube-SLee -L-225, a large-scale, aomain corpus of multilingual sign language videos with seemingly well-aligned captions, primarily intended for translation from each sign language to its region's 's spoken language. By open-domain, we are distinguishing from datasets like AfriSign [9] and JWSign [8]. which have made strides towards massive sign-multne multilinguality but feature a limited domain (c.g., Bible translations).
We mined these videos using two-step process: First, like YouTube-ASL [35], we used automatic classifiers on text metadata to identify potentially relevant videos. And second-in contrast to YouTube-ASLL, which paid native ASL signers to filter out individual videos with wrong or misaligned captions over the course of several months, a process that isn't amenable to scaling across many sign languages- used our own knowledge of sign languages and YouTube data to triage the videos over four days, ranking the priority of channels according to total hours of content and then auditing videos per channel, with particular attention outliers by duration. This means that the annotations were performed with less expertise than You ASL, but in practice there are many signals that can be used to identify high-quality content even without complete understanding, as observed in work with written languages [15; 16].
"Correspondence gtanzer@google. 'We release the YouTube- SL-25 video IDs under CC BY 4.0 at this link. Note that this license only applies to the video IDs and ISO 639-3 language codes. which we selected and labelled. The underlying video and caption content, as with all datasets consisting of YouTube video IDs, is subject to different licenses and should he accessed/used in accordance with the You Tube Terms of Service.

---

Published conference paper at ICLR 2025
Figure 1: World map showing the amount of content in YouTube-SLL225 for each sign language; the area each circle is proportional to the number of hours. The circle in the middle the Atlantic Ocean represents International Sign. Observe that the dataset is especially lacking in representation for Central South America, Africa, West & Central Asia, and China & Southeast Asia.
The result is dataset with 3207 hours of videos with seemingly well-aligned captions, featuring >3000 unique signers across >> 25 sign languages. This is >3x the size of You'uube-SSL. (984 hours), which is subset of YouTube-SL-25. and larger than the prior largest parallel sign language dataset, JWSign (2530 hours), which is closed-domain. YouTube-SL-25 is the first or largest parallel dataset for many of its component sign languages.
Following Kreutzer et al. [15]'s call to avoid "representation washing"- -iiiinggie.... overstating the number of low-resource languages supported when their dataset size is marginal- wwe name YouTube-SL-22 after the 25 sign languages that have at least 15 hours of representation in it. This is roughly the minimum size of datasets that have previously been released for individual sign languages, but is still considered extremely low resource. When including the long tail, YouTube-SL-22 includes data at least 55 sign languages.
We provide baselines for YouTube-SL-25 on sign language understanding tasks, extending the multitask mixture from FLEURS-ASL'S baselines [34] support multiple source/target languages and the sign language identification task (which is extremely understudied [6; 7; 33]). Our results on benchmarks across 4 sign languages show that both higher- and lower-resource sign languages within YouTube-SSL benefit from multilingual transfer.
We publicly release the YouTube-SL--2 video IDs at this link. We hope that YouTubeeSSL-25 can be further refined by language experts in the community and will be useful for variety of purposes, such as general sign language pretraining and medium-quality finetuning for downstream tasks like translation, caption alignment, & sign language identification. In particular, we think that it is necessary to develop robust filtering and preprocessing tools for sign languages in order to realize the next order of magnitude increase in dataset size.
THE YOUTUBE-SL- CORPUS
YouTube-SL-25 a massively multilingual corpus of sign language videos with seemingly well- aligned captions drawn from YouTube, intended primarily for pretraining sign-to-text translation models. You- is a superset of YouTube-ASL. [35].
YouTube-ASL used two-step pipeline to construct its corpus: "first, retrieval using automatic content-based annotations, and second, filtering by skilled human annotators at per-video level" [35]. We adopt the first step with minimal modifications, but lower standard for the second step:
2

---

Published conference paper at ICLR 2025
triaging/auditing by a single non-native signer to identify seemingly high-quality channels for inclusion (even without full understanding of the content in every sign language).
2.1 AUTOMATICALLY RETRIEVING CANDIDATE VIDEOS
We retrieved candidate videos in a similar way to YouTube-ASL [35], by fetching videos tagged with Knowledge Graph entities related to sign language generally or any individual sign language with its own tag of July 2023.² As with YouTube-ASI. the recall of these tags is limited in that they are not aware of sign language in the video content itself (i.e., they do not use sign language detection and sign language identification classifiers), they may miss videos or channels that use sign language but do not explicitly mention it. The pool of videos that we fetched from is also restricted some generic ways, such as that the videos must be public and listed.
We applied the same filtering steps as YouTube-ASI select only videos with manually uploaded captions, and remove videos with duration < <10 seconds or >5 hours, width <480 pixels or height 360 pixels, and frame rate <15fps or 660ps. We added one additional step: remove videos where captions cover <40% of the duration.
Unlike You ASL, we did not exclude videos of conversations with more than one signer in frame at time. While our pose-based baselines only support one person's 's input, feel that there is no reason to exclude these videos from the corpus itself. We are not aware of any prior works that study sign language translation from videos with multiple signers; some datasets like the Public DGS Corpus [10] consist of conversational data, but they are recorded with one camera per signer. Being able to handle this kind of data is important both in the near term make the most of the limited resources available and in the long term to support multi-signer applications.
The result of this process was a list of 81,623 candidate videos that might contain signed content with high-quality captions.
2.2 TRIAGING CANDIDATE VIDEOS WITH COARSE MANUAL ANNOTATIONS
YouTube-ASL used 3 native ASL signers as annotators to identify which of the individual candidate videos contained ASL and had high-quality, well-aligned English captions, labelling videos over the course of several months and several rounds of iteration on the annotation tool/labelling standards. This kind of annotation does not scale well to massively multilingual data because it is difficult and expensive to onboard native (or even just proficient) signers to annotate videos in tens of (extremely low resource) sign languages. Instead, we follow Kreutzer et al. [15]'s observation that many data filtering tasks can be done even without full understanding of the content. Therefore, the first author (a non-native hearing signer with experience in several sign languages, primarily ASL) served as the annotator, triaging the corpus per-channel annotations and targeted audits. The triage proceeded as follows:
the course of four-day weekend using combination of
First, automatically included the YouTube-ASL videos in our corpus and removed these from the manual review. Then we grouped videos by channel and sorted by total duration in descending order, with random ordering for videos within each channel. We used heuristic classifiers based on public text metadata for each video and channel (e.g., title, description, caption language) to provisionally label each video with a sign language. For each channel, the author sampled several random videas and judged whether they matched the acceptance criteria (described below), then corrected the sign language label if relevant. If all the sampled videos were good, the channel was tentatively accepted; if the videos were bad on balance, it was rejected; if it was mixed but promising, the author investigated further and was able to exclude individual bad videos. The author triaged the first 1000 channels of 18000 (down to about 1 hour of content per channel), then continued into the 3000-6000 range, skipping videos predicted to be in ASL in order to prioritize the long tail of sign languages." Finally, we sorted the videos for each channel by duration in descending order and the first author made a second pass, checking outlier videos with unusually long durations for their
We refreshed the dataset in May 2024 include newer videos from accepted channels, with some light manual review.
An observation: the acceptance rates for videos in different sign languages were dramatically different. For example, the vast majority of content from Central & South America was live-interpreted events with small interpreters and poor caption alignment. We expect that high-quality produced content is more common in
3

---

Published conference paper at ICLR 2025
channels to reduce heavy-hitter errors. (Channels that produce mostly high-quality, shorter content sometimes had longer live-interpreted events mixed in, which have different translation and caption quality.)
Now, a description of our annotation standard for individual videos and some shallow signals to recognize these features even without full understanding:
The video should feature sign language. While there is such thing as fake sign language inter- preters, they do not typically create YouTube channels full of themselves gesturing nonsensically, there is not much adversarial content to sift through. The closest thing to this is inexperienced sign language learners." 4 An annotator proficient in one sign language can generally distinguish these cases (and others like interpretive dance/gesture) even for other sign languages.
The signer(s) should be the primary content. If signer is interpreting spoken content, the video of the speaker (if present) should be significantly smaller than the signer. If the person is switching off between signing and speaking, or speaker is the primary content for significant fraction of the video, it should be rejected.
The video should have relatively well-aligned captions. There are a few signals you can use for this: if the video has speech track, the captions are virtually always well-aligned; these are typically channels with natively Deaf reafueroduced content that was captioned after the fact. You can validate the alignment by seeing whether the changes in captions occur at boundaries in sign language phrases (which are marked by prosodic signals that generally do not require knowledge of the particular sign language identify [5]). If there
speech track, you can listen to whether the caption timing matches the speech track. We accepted videos where the captions were aligned to the speech track as long as they were not off by more than a couple of seconds; usually this corresponds to an after-the- fact voiceover translation, not live interpretation.
Label which sign language the video is in. It was almost always possible to classify the sign language with high confidence based on the video title/description and caption language, or at worst the channel's description (including its stated country). The failures in automatic classification mostly came from International Sign, often specified as just "IS", which was not automatically tagged. In the rare cases where we could not determine the language with high confidence, we marked the sign language as unknown.
In order to sanity check the end-to-end quality of this triage process, the first author ran two additional audits on the final dataset (excluding the videos from YouTube-ASL, since we were already confident in their quality). The author rated two samples of 100 videos, one sampled uniformly per video id, and the other with videos sampled with probability proportional to their video duration. The author loaded each video and scrubbed through the frame previews across time identify cuts where content and captioning habits may change.
In the sample with random videos by ID, we found: video that strictly should not have been included in the dataset (a brief video with captioned narration), 1 borderline video with a waist-up interpreter next to video of a speaker's face only, and 3 videos with content from learners that used appropriate vocabulary but didn't 't really provide a complete translation.
In the sample proportional to video duration, we found: 0.76% of content that was speech rather than signing (from channel with otherwise entirely signed content), and 0.96% that consisted of interstitials mixed within otherwise good content, where the interstitials had signing but no captions.
We consider these error rates acceptable.
high-income countries, so it will be important for future work to leverage lower quality data in order to be more globally inclusive.
*We accept learners as long as they seem relatively competent. Models should be able to understand less proficient signers, especially given that many deaf people are not exposed to sign language in childhood and reach varying levels of proficiency [19]. Less proficient signing can be easily distinguished from native signing, so it should not harm the quality of sign language understanding (especially if the data is only used for pretraining). but more care is required for generation.
4

---

Published conference paper at ICLR 2025
Monk 05
Monk 01
1,7%
3.3%
Monk 04 5.3%
Monk 03 8.2%
Masculine
50.7%
49.3%
Monk 02
79.8%
(a) Monk skin tone ratings [17]. Dark skin is globally underrepresented: some of this is due to the regional bias towards the Global North.
(b) Perceived gender presentation (uonbinary presen- tation not supported by the classifier). Masculine and feminine presentation are at parity.
Figure 2:
Demographic representation of You Tube-S1-25 content (proportion of hours), pre- dicted with proprietary classifiers. These predictions should only be interpreted in aggregate.
2.3 CORPUS STATISTICS & COMPARISON TO PRIOR DATASETS
The final human-triaged YouTube-SI-22 corpus consists of 39197 videos totaling 3207 hours of content (covered by 2980 hours of captions 2.16M captions 104M characters) across >25 sign languages. Like in YouTube-ASL, we approximately lower bound the number of unique signers using the number of unique channels, 3072, and expect that this is quite an underestimate channels (or even individual videos) featuring multiple signers are common. This is only somewhat larger than YouTube-ASL:s 2519 channels, which is the main downside of our triage-based annotation approach: it captures the bulk of the hours but not the long tail of unique channels.
As discussed in Uthus al. [35]'s 's related work section, the best of our knowledge YouTube-ASI. and now YouTube-S1-225 are the only sign language translation datasets that aggregate content from wide variety automatically mined channels, as opposed to others which either pay participants to create new data [4; 36; 14; 10] or curate content from small number of content creators [2; 32; 1; 31; 21; 8; 12; 18]. This has implications for diversity in topies/content and video production style. Note that even though YouTube-SL--2. is open-domain overall, there may be discretization effects in the long tail of languages, where language represented by only few channels and therefore in effect is closed-domain
See Figure for a depiction of the relative weight of each language on a world map and Table for breakdown of hours by language, comparing to the largest prior parallel dataset for each language (both open- and -oosadddomain)...... Note that prior works have studied sign languages with no representation in YouTube-SL-25. like AfriSign [9] & JWSign [8] (primarily languages in Central & South America and Africa), and CSL--aaily [40] (China); these languages aren't 't included the table. The distribution of language data be attributed to population size, economic development, and political factors. Other supplementary sources of data, such as Bible translations [9; 8] and national interpreted broadcasts [1: 3], could provide more balanced data distributions.
See Figure for an estimated demographic breakdown of YouTubee-SL-25 by skin tone and perceived gender presentation, predicted by automated classifiers. Dark skin is globally underrepresentee in the dataset, with only 1.9% (60 hours) of the data at Monk 6-10. Some of this can be explained by overrepresentation of countries in the Global North (and the only two countries the Global South with significant fraction of the dataset, Brazil and India, have substantial variation in skin tone). Ensuring demographic fairness remains an important problem for future work.
'We used classifiers to evaluate whether the captions were toxic, hateful, violent, and sexual with confidence >0.8, which flagged <1% of captions. We manually audited the captions flagged with highest confidence and none merited exclusion from the dataset: the "toxie" and "hateful" content were false positives, the "violent" content was about violent events, and the "sexual" content
educational content about sex.
5

---

Published conference paper at ICLR 2025
Table I: Hours of content in YouTube-SS-225 across sign languages, compared the largest prior publicly available parallel dataset for each language (both open- and closed-domain). Some of these sign languages have more commonly used names, like "Libras" for "Brazilian SL", but we refer by region here for convenience. Note that International Sign (ils) is pidgin that arises international Deaf conferences, contrast to all the other sign languages on this list, which are complete natural languages. With the significant exception of You AS,, most prior datasets consist mutually exclusive data that could supplement YouTube- 2L-25.
largest prior (open)
largest prior
sign language
iso 639
#videos
#channels
#hours
#hours
#hours
American
ase
16724
2523
1394
YouTube-ASi ASLbe--SLLA [35]
984
International
ils
1634
14
285
Indian
ins
3023
8
209
iSign [13]
252
Polish
pso
1698
34
137
JWSign
61
German
gsg
1024
108
DGS Corpus [10]
Brazilian
bzs
846
30
101
JWSign
211
British
bfi
1026
60
74
BOBSL [1]
1660
Hungarian
hsh
1687
9
70
JWSign
16
Australian
1098
25
67
Auslan-Daily Daily [31]
Italian
ise
929
21
63
JWSign
115
Japanese
jsl
1075
21
62
JW
67
Russian
rsl
715
9
60
JWSign
111
French
fsl
900
23
49
Mediapi-RGr [21]
Korean
325
38
JWSign
93
Colombian
csn
212
4
37
SW
128
Spanish
701
13
36
JW
85
Dutch
dse
450
14
35
JWSign
1
Argentine
160
34
JWSign
107
Quebec
fcs
128
7
26
JWSign
20
Catalan
csc
239
7
26
Pakistani
pks
581
25
Swedish
swl
410
23
22
JWSign
14
Turkish
189
E-TSL [41]
Swiss German
sgg
185
18
SRF23 [18]
437
Israeli
isr
345
14
17
JWSign
Fenno-Swedish
fss
174
8
16
Finnish
fse
196
Corpus FinSL [28]
15
JWSign
35
Austrian
asq
177
11
13
JWSign
Taiwan
tss
130
5
13
JW
14
Czech
cse
185
12
SW
18
Icelandic
icl
214
11
Irish
isg
34
JWSign
Slovenian
139
10
JWSign
Indonesian
inl
126
2
9
JW
30
Jordanian
jos
132
2
9
Norwegian
nsl
82
8
JWSign
Singapore
sls
102
8
JWSign
Thai
tsq
106
7
JWSign
8
Hong Kong
hks
117
6
TVB-HKSL [20]
14
Lithuanian
11s
160
Slovakian
59
JWSign
14
Chilean
csg
74
4
JWSign
100
New Zealand
nzs
79
8
4
JW
2
Croatian
24
2
4
JW
5
Mexican
mřs
53
JWSign
184
Philippines
psp
26
3
JWSign
21
Vietnamese
36
3
JWSign
3
Peruvian
prl
71
4
2
JW
81
Flemish
vgt
57
4
2
VGT-RAW
100
Swiss French
33
2
2
Signsuisse [18]
13
Greek
gss
22
2
Elementary23 [36]
71
French Belgian
sfb
25
5
2
JW
3
Swiss Italian
slf
30
2
SwissSLi [1]]
10
Danish
dsl
21
5
1
JWSggn
Estonian
eso
12
JWSign

---

Published conference paper at ICLR 2025
3 BASELINES
We demonstrate the value of YouTube-SL-22 with baselines for sign-to-text translation and sign language identification across benchmarks for 4 sign languages.
3.1 SETUP
We build on the unified modeling approach adopted by the FLEURS-ASL bascline mixture [34], which are an evolution of the YouTube-ASL baselines [35]. We adopt this approach as a convenient vehicle for our data ablations and to illustrate the range of tasks that can be constructed from the YouTube-SS--25 data; the particular details are not core to this work. In brief, the approach finetunes pretrained encoder-decoder language model (T5-v1.1 Small) to take input 256 tokens of text (control tokens and caption context) and 512 frames of MediaPipe Holistic landmarks at half frame rate (corresponding to random video span), and output 256 tokens of text (including timestamps). The training examples are randomly sampled 34- second clips from the YouTube-SL- source videos, where the captions preceding and following the clip may be used as context to mitigate caption misalignment.
The only modification we make is to extend the mixture to support multiple source and target languages, as well as the sign language identification task with probability 0.1 (where it classifies the language and then proceeds to translate the input). Like in FLEURS-ASL, we train first on caption- level clips and then equal mixture of caption-level and random clip-level data. See Figure 3 for depiction of the modifications to the control token format.
We compare three models: T5 with only text pretraining (i.e., the original pretrained T5 Small), T5 with continued training on YouTube-SL-25 1400-hour ASL subset, and T5 with continued training on the full SutTubbb-S-255 dataset. See Appendix A for training details.
3.2 COMPARISON TO PRIOR METHODS
See FLEURS-ASL [34] for discussion of how their baseline approach (in particular, training on random clips and caption tracks rather than tightly clipped individual captions) relates to prior modeling work. Insofar as YouTube-SL-25 is slightly noisier than YouTube-ASL, we slightly push the boundaries in terms of training on weakly aligned data. With respect to our modifications: modeling multiple sign languages in one model is not new [38; 39; 9; 8J. We are not aware of any task within translation models (the few prior
prior works that train sign language identification works on sign language identification [6; 7] use low-level phonetic features only), but this is not especially novel given its use in Whisper for speech input [25].
3.3 RESULTS
We provide sentence-level translation (scored with BLEURT (29]) (specifically, BLEURT-20 (24]) and sign language identification (scored with top-1 accuracy) results on benchmarks for 4 sign languages: American Sign Language (ase, FLEURS-ASH [34] & How2Sign [4]), Swiss German Sign Language (sgg, WMT23 SignSuisse [18]), Swiss French Sign Language (ssr, WMT23 SignSuisse [18]), and Swiss Italian Sign Language (slf, WMT23 SignSuisse [18]). We provide both zero-shot scores and finetuned scores where relevant.?
Table 2 shows quantitative results on both tasks, and Table shows a sample of qualitative translation examples. We see that on both translation and lang id, sign language pretraining is substantially better than none (as in Uthus et al. [35]) and multilingual transfer helps both the higher-resource
'We tried to change the pretrained model from T5-v1.1 Small [26] to mT5 Small [37] so that languages besides English could benefit from pretraining and hetter tokenization, but initial experiments mT5 took about 1/3 more steps converge and achieved worse results. We therefore ran the full set of experiments with T5 only. FFor translation, the model separately finetuned for each dataset, checkpoint selected based on BLEU on the validation set. For sign language identification, zero-shot scores mean that the model is briefly finetuned on Yubeube-SL rebalanced to the sign languages with equal weight, and finetuned scores mean that the model is finetuned on equally weighted mixture of the benchmarks' training sets. We don't 't finctune on TLEURS-ASI. the finetuned langid scores are after finetuning on How2Sign.
7

---

Published conference paper at ICLR 2025
untimeed:
captson
sep
caption R
sep
captious
caption
caption.
input tokens
AVg cup
left edge
owerllew
CUT opdce
aligs
STC long
1gt long
0.2
subset
OUT oedica
Inusslate
ustimed
0,96
edge
0,8
0.5
0.1
sPc lang
lgt ling
0.5
0.9
0.2
right ddge
farget tokens
DOS
timed] Jnnineed cuptions
BOS
ling il
long
controll
text tolkns
timessturp
tokess
Figure 3: Unified document-level sign-to-text training, extended for multilingualityy; modified from Figure 2 of Tanzer [34]. New additions circled in red. At high level: The mixture allows training on random crops from videos with timed caption tracks (without strict sentence-level alignment). It supports multiple tasks such as translation and caption alignment using timestamps represented as text, and captions surrounding the given clip as context. We straightforwardly extend the task format from monolingual to multilingual. Ar low level: For caption alignment, source and target language are provided unconditionally. For translation, source and target language are provided with probability 0.9 and predicted with probability 0.1. When the sign language identification task is active, we avoid conditioning on the target spoken language (either with an explicit language code, or by providing text context in the input) because it makes the task casier or trivial. (Most naturally occurring data pairs one sign language with one spoken language.)
sign language (ASL) and the lower-resource sign languages within You'ubee-SL-25. Some prior works [9; 8] did not see strict improvement from modeling additional sign languages; we expect that this was model capacity issue (during pretraining, or from finetuning on multiple languages simultaneously rather than one at a time). As expected, finetuning gives large gains especially when the language is poorly represented in YouTube-SL
and benefits from multilingual pretraining. For example, our finetuned sgg result of 37.7 BLEURT (7.5 BLEU) far exceeds the top WMT23 score of 23.6 BLEURT (0.3 BLEU).
Note that for sign language identification, finetuning is somewhat ill-conceived because some datasets have train/test signer overlap just generally unique recording conditions that could be used shortcut, For example, langid quality on FLEURS-ASI. decreases when pretrained on ASL. data and finetuned on How2Sign, even compared to no sign language pretraining- with accuracy varying dramatically across the 3 signers within the test set, averaging 18.3% vs. 94.9% vs. 77.0%. This suggests that different kinds of pretraining may cause different features be surfaced during finetuning, which may overfit or not transfer with respect to broader domains. Our zero-shot results are nontrivial despite the severe class imbalance. A proper sign language identification benchmark needs more consistency across languages; we were unable to use SP-10 [38] for licensing reasons.
LIMITATIONS
As discussed in Section 2, each step of the corpus curation comes with limitations: the automatic tagging step misses content in sign language that does not mention sign language (in the video itself or in metadata) ($ 2.1), the manual triage step trades off quality and small-channel exhaustiveness for annotation effort ($ 2.2), and the result has issues with representativeness-ee the distribution of languages and skin tone in particular ($ 2.3). The dataset is also small overall in comparison to MT datasets for spoken languages; we expect that more data will be needed reach usable translation quality in generality, but this may be sufficient for more narrowly scoped tasks. Because YouTube-SL. 25 (like YoUT has so much signer variety (appearance, recording environment, proficiency,
8

---

Published conference paper at ICLR 2025
Table 2: Baseline results for sentence-level translation and language identification on bench- marks for 4 sign languages. Translation is measured with BLEURT, lang id with top-1 accuracy: we report zero-shot / finetuned scores where relevant. We compare the results of no sign language pretraining vs. pretraining on the ASL subset of YouTube-SL-22 vs. the full YouTube-SL- dataset.
ase
Sgg
SST WMT23 SignSuisse
slf
pretrain set
FLEURS-ASI.
How2Sign
translation
None
/31.2
126.2
/7.2
/22.8
YT-SL-25 (ASL)
33.7/-
30.2 46.6
9.0/
5.//1111
9.3/
YT-SL-25 (Full)
40.1/-
29.7 47.9
9.5/37.7
6.8/ 18.8
25.2
lang
None
/98.4
/99,6
19.66
198.8
887.6
YT-SL-25 (ASL)
100.0 66.7
89.7 99.9
54.0/89.6
6.0/ 100.0
1.6 19.6
YT-SL-25 (Full)
100.0 99.9
92.7 100.0
72.0/
0.4 100.0
64.8 99.6
Table 3: Qualitative examples for sentence-level translation across datasets. Examples selected randomly without cherrypicking (from examples already highlighted in previous papers if available).
language
setting
text
ase
reference
And that's a great vital point technique for women's is self defense. It's great point for women self protection.
prediction
sgg reference
Dieses Buch enthält ein Vorwort, welches sich Präambel nennt. (This book contains a foreword called a preamble.) Dieses Buch hat eine Wörtchenschilde.
prediction
(This book has a word shield.)
ssr
reference
Le fiancé de mon amic va bientôt s ennaager dans l'arméc. (My friend's 's funccé is going to join the army soon.)
prediction
Mon amie à a'améée.
(My friend the army.)
slf
reference
allergica alle punture d' d'api. (She allergic to bee stings.)
prediction
Lei ha un ape migliore.
(She has better bee.)
and now also number of signers in frame at time), it may be difficult to use it to train consistent sign language generation models.
Our baselines are limited in that we just use FLEURS-ASL' multitask training mixture as way to pretrain on slightly noisy data and don't engage with the discourse-level or timestamp tasks it enables. It is difficult to evaluate these tasks (plus sign language identification) benchmarks that primarily focus on sentence-level translation; this is on top of fact that there are not enough benchmarks (especially ones with suitable licensing) to evaluate the vast majority of sign languages in BouTube-SL-25 This underscores the importance of extending FLEURS-ASL to even more sign languages.
collection of disparate
CONCLUSION
In this paper, we presented YouTube-SL-25,, multilingual, open-domain corpus of sign language videos with seemingly well-aligned captions, with >3000 hours of content across >25 sign languages. We achieved this efficiently without hiring tens of language-specific annotators by performing triage grouped by channel and sorted by duration of content; this comes at the expense of the long tail of unique signers and some quality assurance. We demonstrated the value of YouTube-SL-22 with experiments in sentence-level translation and sign language identification which demonstrate multilingual transfer benefits both higher- and lower-resource sign languages. We hope that

---

Published conference paper at ICLR 2025
SL-25 and our account of how we curated it will serve as foundation for research towards the ultimate goal of making technology inclusive for Deaf/Hard of Hearing signers worldwide.
ETHICS STATEMENT
The ethical considerations of this work are very similar to YouTube-ASL. We release publicly available YouTube videos only video IDs so that deletions automatically reflected in the corpus. We train our baseline models on MediaPipe Holistic skeletons as a form of anonymization. Another work that uses YouTube-ASI Rust et al. [27], explores orthogonal approach for anonymization in direct video modeling: pretraining with the face blurred and finetuning with it unblurred on separate data. The field should continue study this topic develop a better understanding of the tradeoff's of different approaches for responsible use of the data.
While we provide estimated demographic breakdowns of the signers in YouTube-SL along some dimensions, even if the dataset were globally representative, this would not guarantee that models trained on it would have equal performance across these attributes others. And this intersects with sign multilinguality: models may perform worse for some demographics only in some sign languages due cross-sectional effects. For example, sign language identification may be biased by the signer's race proxy for nationality (itself proxy for language) due to correlations in the training distribution to the extent that race can be inferred from the input representation. evaluate models in the context of their specific intended use cases to ensure that they robustly deliver on being useful for Deaf/Hard of Hearing users and avoid the trap of overroomssnn/uundedeelieeee that has plagued sign language technology historically.
important to
ACKNOWLEDGEMENT
We thank Manfred Georg and Caroline Pantofaru for institutional support; Anelia Angelova, Chris Dyer, and anonymous reviewers for feedback on drafts of this paper; David Uthus for general help with infrastructure; and Krishna Somandepalli, Katie Zhang, & Simon Wang for assistance on the fairness analysis.
REFERENCES
[1] Samuel Albanie, Güi Varol, Liliane Momeni, Hannah Bull, Triantafyllos Afouras, Himel Chowdhury, Neil Fox, Bencie Woll, Rob Cooper, Andrew McParland, and Andrew Zisserman. Bbc-oxford british sign language dataset, 2021. URL httpt://arxi...oooo org/
www.ooggg//////eeeeeeo.e /aa////s11111/// 03635..
[2] Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. Neural sign language translation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018
[3] Necati Cihan Camgoz, Ben Saunders, Guillaume Rochette, Marco Giovanelli, Giacomo Inches, Robin Nachtrab-Ribel and Richard Bowden. Content4all open research sign language translation datasets, 2021.
[4] Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giro-i Nieto. How2Sign: Large-scale Multimodal Dataset for Continuous American Sign Language. In Conference Computer Vision and Pattern Recognition (CVPR), 2021.
[5] Jordan Fenlon, Tanya Denmark, Ruth Campbell, and Bencie Woll. Seeing sentence boundaries. Sign Language and Linguistics, 10(2):177-200, 2007. ISSN 1387-9316. doi: 10.1075/sl1.10.2 06fen.
[6] Binyam Gebrekidan Gebre, Peter Wittenburg, and Tom Heskes. Automatic sign language identification. 2013 IEEE International Conference on Image Processing, pages 2626 2626-2630, 2013. doi: 10.1 109//CCPP2013.673888
10

---

Published conference paper at ICLR 2025
[7] Binyam Gebrekidan Gebre, Onno Crasbom, Peter Wittenburg, Sebastian Drude, and Tom Heskes. Unsupervised feature learning for visual sign language identification. In Kristina Toutanova and Hua Wu, editors, Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 370-376, Baltimore, Maryland, June 2014. Association for Computational Linguistics. doi: 10.3115/v//P1-20611. URL
https: https///aaaaaaaaoooooeeoooo wwwpthoooggooooooP..oooo org/P1 2966...--201
[8] Shester Gueuwou, Sophie Siake, Colin Leong, and Mathias Müller. Jwsign: A highly multilin- gual corpus of bible translations for more diversity in sign language processing, 2023.
[9] Shester Gueuwou, Kate Takyi, Mathias Müller, Marco Stanley Nyarko, Richard Adade, and Rose-Mary Owusuaa Mensah Gyening. Afrisign: Machine translation for african sign languages. In 4th Workshop on African Natural Language Processing, 2023. URL ht https: /tppss/////peevieww net/forum sett/frrumiid-EHllkk3
[10] Thomas Hanke, Mare Schulder, Reiner Konrad, and Elena Jahn. Extending the Public DGS Corpus in size and depth. In Eleni Efthimiou, Stavroula-Evita Fotinea, Thomas Hanke, Julie A. Hochgesang, Jette Kristoffersen, and Johanna Mesch, editors, Proceedings of the LREC2020 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives, pages 75-82, Marseille, France, May 2020. European Language Resources Association (ELRA). ISBN 979-10-95546-54-. URL https: /wwps//holaaalaoyy.ocooooooo org/2020. signlang-1. 1.
[11] Zifan Jiang, Anne Göhring, Amit Moryossef, Rico Sennrich, and Sarah Ebling. SwissSLi: The multi-parallel sign language corpus for Switzerland. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors, Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages -5446, Torino, Italia, May 2024. ELRA and ICCL. URL htttss://aalaanhllllgggooooo//// org/ 2024. 1rec- 342.
[12] Abhinav Joshi, Susmit Agrawal, and Ashutosh Modi. Isttranslate: Dataset for translating indian sign language, 2023.
[13] Abhinav Joshi, Romit Mohanty, Mounika Kanakanti, Andesha Mangla, Sudeep Choudhary, Monali Barbate, and Ashutosh Modi. iSign: A benchmark for indian sign language processing. In Findings of the Association for Computational Linguistics: ACL 2024, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[14] Sang-Ki Ki Ko, Chang Jo Kim, Hyedong Jung, and Choongsang Cho. Neural sign language translation based on human keypoint estimation, 2019.
[15] Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii- Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoî t Sagot, Clara Rivera, Annette Rios. Is- abel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niy- ongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F, Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Balli, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. Quality a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguistics, 10:50-72, 2022. doi: 10. 10.1/al/ a 00447. URL httpp::/doon http:////ooooooo////// /0. www.oo.......11.aeeeeeeeeeo 1112%2Ftacl_a.0000
[16] Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Christopher A. Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. Madlad-400: A multilingual and document-level large audited dataset, 2023.
[17] Ellis Monk. Monk skin tone scale, 2019. URL htt.hht:///kkittneeee.... google.

---

Published conference paper at ICLR 2025
[18] Mathias Müller, Malihe Alikhani, Eleftherios Avramidis, Richard Bowden, Annelies Braffort, Necati Cihan Camgöz, Sarah Ebling, Cristina España-Bonet, Anne Göhring, Roman Grund- kiewicz, Mert Inan, Zifan Jiang. Oscar Koller, Amit Moryossef, Annette Rios, Dimitar Shteri- onov, Sandra Sidler-Miscrez, Katja Tissi, and Davy Van Landuyt. Findings of the second WMT shared task on sign language translation (WMT-SLT23), In Philipp Kochn, Barry Haddow, Tom Kocmi, and Christof Monz, editors, Proceedings of the Eighth Conference on Machine Transla- tion, pages 68-94, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10118665/11/22233 wmm2233wmmm1mm wmt-w.44 1.4. URL https: / ac acw.anholoooyyyyoooooooooooooloa www.///........oooo/.../... .mm0223.wmtt..mmm 1, .4.
[19] Joseph J Murray, Wyatte C Hall, and Kristin Snoddon. Education and health of children with hearing loss: the necessity of signed languages., 2019. URL https: htwp:://wwwwwttttt www...ccbi.... nlm. nih. how//pnnc/rrticcee////77777
[20] Zhe Niu, Ronglai Zuo, Brian Mak, and Fangyun Wei. A hong kong sign language corpus collected from sign-interpreted news, 2024.
[21] Yanis Ouakrim, Hannah Bull, Michèle Gouiffès. Denis Beautemps, Thomas Hueber, and Annelies Braffort. Mediapi-RGB: Enabling Technological Breakthroughs in French Sign Language (LSF) Research through an Extensive Video-Text Corpus. VISAPP 2024 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, volume 2, Rome, Italy, February 2024. doi: 11.52200011232200003060 URL https: /hall sci httt::::/haa.scceeneeeeeee
[22] Kishore Papineni. Salim Roukos. Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02. page 311-311 1-318, USA, 2002. Association for Computational Linguistics. doi: 10.3115/1073003 1073135 1073135.
URL https:://bi.org////eee 101.://.000/111111111111100
[23] Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Belgium, Brussels, October 2018. Association for Computational Linguistics. URL https: hwtp://wwwwwwwo aclweb. orgooogg////o/ hngggoghology///// W1 W18-6319.
[24] Amy Pu, Hyung Won Chung, Ankur P Parikh, Sebastian Gehrmann, and Thibault Sellam. Learning compact metrics for mt. In Proceedings of EMNLP. 2021.
[25] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision, 2022.
[26] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2020.
[27] Phillip Rust, Bowen Shi, Skyler Wang, Necati Cihan Camgöz, and Jean Maillard. Towards privacy-aware sign language translation scale, 2024.
[28] Juhana Salonen, Antti Kronqvist, and Tommi Jantunen. The corpus of Finnish Sign Lan- guage. In Eleni Efthimiou, Stavroula-Fyita Fotinea, Thomas Hanke, Julie A. Hochgesang. Jette Kristoffersen, and Johanna Mesch, editors, Proceedings of the LREC2020 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives, pages 197-202, Marseille, France, May 2020. European Language Resources Association (ELRA). ISBN 979-54-9554---4-22 URL wwtttsss//acleeeeeeeeeteeeeeeeee hwwpa/aaaaannooooooooooooooooota org/2020. 00220.signlann signlang-1..
[29] Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. Bleurt: Learning robust metrics for text generation, 2020.
[30] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost, 2018.
12

---

Published conference paper at ICLR 2025
[31] Xin Shen, Shaozu Yuan, Hongwei Sheng, Heming Du, and Xin Yu. Auslan-daily: Australian sign language translation for daily communication and news. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL.
https:
[32] Bowen Shi, Diane Brentari, Greg Shakhnarovich, and Karen Livescu. Open-domain sign language translation learned from online video, 2022. URL htttss:/aaxxiv.orr//bb///22 12870.
[33] Ahmed Sultan, Walied Makram, Mohammed Kayed, and Abdelmaged Amin Ali. Sign lan- guage identification and recognition: A comparative study. Open Computer Science, 12 (1):191-210, 2022. doi: dot:10.1515/comp-2200 comp- 2022-0240.
URL https:://ol..org//// 1515/
[34] Garrett Tanzer. Fleurs-asl: Including american sign language in massively multilingual multitask evaluation, 2024. URL
13585.
[35] David Uthus, Garrett Tanzer, and Manfred Georg. Youtube-asl: A large-scale, open-domain american sign language-english parallel corpus, 2023.
[36] Andreas Voskou, Konstantinos P. Panousis, Harris Partaourides, Kyriakos Tolias, and Sotirios Chatzis, A new dataset for end-to-end sign language translation: The greek elementary school dataset, 2023.
[37] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text trans- former, 2021.
[38] Aoxiong Yin, Zhou Zhao, Weike Jin, Meng Zhang, Xingshan Zeng, and Xiaofei He. Mislt: Towards multilingual sign language translation. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5099-5109, 2022. doi: 10. 10.1109/CVPR52688. 2022.00505.
[39] Biao Zhang, Mathias Müller, and Rico Sennrich. Sltunet: A simple unified model for sign language translation, 2023.
[40] Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li. Improving sign language translation with monolingual data by sign back-translation, 2021.
[41] Şükrü Öztürk and Hacer Yalim Keles. E-tsl: A continuous educational turkish sign language dataset with baseline methods, 2024.
13

---

slf Published conference paper at ICLR 2025
A TRAINING DETAILS
We train all of our models with Adafactor [30] with base learning rate 0.001.
pretrained our two T5v1.1 Small models (on Youtube-SL-25's ASL and Full sets) on TPUv3s for 210k and 430k steps respectively (switching from pure caption-level training 1:1 caption- revdoorandom clip-level training once the model appeared to have converged, then stopping again after re-convergence, both according to BLEU on the How2Sign val set, like in FLEURS-ASL [34]). Each Ik steps took about minutes to train. We also pretrained an mT55 Small model for about 600k steps, which underperforming
didn't 't run the complete set of experiments for it.
We finetuned the sentence-level translation models on 16 TPUv3s with a batch size 32 until convergence; this took about 10k steps for WMT23 SS DSGS and at most 2.5k steps for the other datasets.
We finetuned the language identification models mixture of data for the four languages with equal weight, either subsets of YouTube-SL-25 (for zero-shot results) or the training splits from the downstream benchmarks (for finetuned results). We used 16 TPUv3s with batch size of 32 until convergence, with up to 3k steps.
B BLEU SCORES
See Table 4 for sentence-level translation BLEU [22] scores, rather than BLEURT from Table 2 in the body of the paper. We use sacreBLEU [23] with intl tokenizer.
Table 4: Baseline results for sentence-level translation on benchmarks for sign languages, measured in BLEU (zero-shot finetuned).
ase
sgg
SST
pretrain set
FLEURS-ASL
How2Sign
WMT23 SignSuisse
None
-/-
-/2.6
-.2
1/3
-/1.1
YT-SL-25 (ASL)
3.4/-
3.6/ 14.5
0.1
0.1 0.4/5.9 1.1/7.0
YT-SL-25 (Full)
4.4/-
4.2/ 15.4
0.5
0.1
14


Reviews:
[{"review_id": "7XcCKCtzf1", "reviewer": "Reviewer_f36H", "summary": "This paper introduces YouTube-SL-25, a large-scale, open-domain, multilingual sign language video corpus with seemingly well-aligned captions. It features over 3,000 hours of videos spanning more than 25 sign languages, making it the largest supervised sign language dataset to date. The corpus surpasses the size of YouTube-ASL, a previous sign language corpus, by more than three times. \nHere are the key contributions of the paper:\n1. Data Collection and Curation: The paper describes a two-step process for mining and filtering sign language videos from YouTube. First, videos tagged with sign language-related keywords are automatically retrieved. Then, a manual triage process is employed, where a single annotator identifies high-quality channels with well-aligned captions, prioritizing channels based on content duration\n2. Multilingual and Open-Domain Focus: Unlike datasets like AfriSign and JWSign, which focus on specific domains (e.g., Bible translations), YouTube-SL-25 includes videos from a diverse range of topics and genres, making it \"open-domain\".\n3. Baseline Models and Multilingual Transfer: The paper establishes baselines for sign language understanding tasks, utilizing a multilingual multitask model based on T5. Results indicate that multilingual transfer learning benefits both resource-rich and resource-poor sign languages within the corpus.\n4. Addressing Data Bottleneck and Inclusivity: The paper aims to tackle the scarcity of data for sign language research, particularly for less-studied sign languages. It highlights the importance of developing robust filtering and preprocessing tools for sign language data to further expand dataset size and inclusivity.\n5. Ethical Considerations: The authors acknowledge the ethical implications of working with videos featuring individuals and discuss their anonymization efforts using MediaPipe Holistic skeletons. They also emphasize the need for ongoing research to ensure responsible data usage and equitable model performance across different demographics", "questions": "Based on the provided excerpts from the paper \"YOUTUBE-SL-25: A LARGE-SCALE, OPEN-DOMAIN MULTILINGUAL SIGN LANGUAGE PARALLEL CORPUS\" and our previous conversation, here are some questions and suggestions for the authors:\nData Curation and Annotation:\n1. Clarify the rationale for the 15-hour threshold for language inclusion. The paper mentions including languages with at least 15 hours of representation, aligning with the minimum size of previous datasets. However, the authors also acknowledge this as \"extremely low resource\". Further explanation of the trade-offs considered in setting this threshold and how it balances representativeness with data scarcity would be beneficial.\n2. Elaborate on the specific challenges encountered during the manual triage process. The paper provides a high-level overview of the triage procedure but could benefit from a more detailed discussion of the practical difficulties faced. For instance, were there specific sign languages that proved particularly challenging to identify or assess for caption alignment? Providing insights into these challenges would shed light on the complexities of working with diverse and low-resource sign language data.\n3. Address the potential for bias introduced by the single annotator. While the paper acknowledges this limitation, a more in-depth discussion of mitigation strategies would strengthen the analysis. For example, could the authors elaborate on the annotator's background and expertise in different sign languages? Could they discuss the feasibility of incorporating a second-level review by additional annotators, even for a subset of the data, to assess inter-annotator agreement and identify potential biases?\n\nDataset Analysis and Evaluation:\n1. Expand the demographic analysis to encompass a wider range of factors. The current analysis focuses on skin tone and perceived gender presentation. However, considering other demographic variables, such as age, regional variations in signing, and disability intersectionality, would provide a more comprehensive understanding of the dataset's representativeness.\n2. Consider including additional evaluation metrics that capture discourse-level features. The current evaluation focuses primarily on sentence-level translation accuracy. Incorporating metrics that assess the coherence, fluency, and expressiveness of translated output, particularly concerning discourse-level phenomena in sign language, would provide a more nuanced view of model performance.\n3. Explore the potential of YouTube-SL-25 for sign language generation tasks. The paper primarily focuses on understanding tasks. \nExpanding the scope to include generation tasks could open up new research avenues. Discussing potential challenges and opportunities in this area, and perhaps presenting preliminary experiments on a subset of the data, would enrich the paper's contribution.\n\nGeneralizability and Future Work:\n1. Discuss the generalizability of the findings to other sign language processing tasks. While the paper focuses on translation and language identification, addressing the dataset's potential for other tasks, such as sign language recognition, summarization, or sentiment analysis, would highlight its broader applicability.\n2. Outline future directions for improving the corpus and addressing its limitations. This could include strategies for expanding the dataset to include more languages and signers, refining the annotation process, developing more robust noise filtering techniques, and exploring alternative data sources beyond YouTube.\nThese questions and suggestions aim to encourage a deeper exploration of the dataset's strengths, limitations, and potential impact. Addressing these points would contribute to a more robust and informative discussion", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: \nThe paper introduces YouTube-SL-25, a large-scale, open-domain, multilingual corpus of sign language videos with seemingly well-aligned captions. This is the largest supervised sign language dataset to date and the first or largest parallel dataset for many of its 25+ component languages. YouTube-SL-25 significantly expands upon YouTube-ASL, being over three times its size. The corpus creation methodology, while building upon prior techniques, innovates by utilizing a single annotator for manual triage, prioritizing channels by content duration. This approach allows for efficient curation, enabling the inclusion of a diverse array of sign languages. Additionally, the paper explores the novel integration of sign language identification as a task within translation models.\n\nQuality: \nThe paper presents a well-constructed and carefully curated dataset. The authors employ a two-step process, first using automatic classifiers to identify potentially relevant videos and then utilizing a manual triage process to ensure quality and alignment. While acknowledging the potential for limitations due to the single-annotator approach, the authors provide detailed descriptions of the annotation standards and conduct audits to assess and report error rates. The dataset's quality is further evidenced by the strong baseline results achieved in translation and language identification tasks.\n\nClarity: \nThe paper is well-written and easy to follow. The authors clearly present their methodology, experiments, and findings using concise language and a logical structure. The paper effectively utilizes figures and tables to present data and illustrate key concepts. The authors provide a thorough literature review, contextualizing their work within the existing research landscape and citing relevant prior datasets and models.\n\nSignificance: \nThe paper addresses a critical bottleneck in sign language processing: the scarcity of data, especially for less-studied sign languages. The creation and release of YouTube-SL-25 as an open resource has the potential to significantly advance the field by enabling the development and evaluation of more robust and inclusive sign language processing models. The paper's findings on multilingual transfer learning hold promise for improving the performance of sign language understanding systems across various languages, including those with limited resources. The dataset's size and diversity also make it a valuable asset for exploring novel architectures and training paradigms, potentially leading to breakthroughs in sign language recognition, translation, and generation", "weaknesses": "1. Limited Evaluation Scope: While the corpus covers a wide range of sign languages, the baseline experiments presented in the paper only evaluate four sign languages. Expanding the evaluation to encompass a more diverse subset of the languages represented in YouTube-SL-25, especially those with fewer resources, would provide a more comprehensive understanding of the corpus's utility for multilingual transfer learning and the impact of data scale on model performance. This expanded evaluation could reveal specific challenges or opportunities associated with different sign languages, informing future research directions.\n2. Single-Annotator Triage: The reliance on a single annotator for manual video triage, while efficient for corpus creation, raises concerns about potential biases and inconsistencies in the data selection process. The annotator's proficiency in certain sign languages and potential blind spots in recognizing subtle cues related to caption alignment or signer proficiency could introduce systematic biases. Involving multiple annotators, ideally with diverse linguistic backgrounds and sign language expertise, would enhance the reliability and objectivity of the triage process. This collaborative approach would help mitigate individual biases and ensure a more balanced representation of signing styles and dialects within each language.\n3. Potential Data Noise: The authors acknowledge the potential for noise in the dataset, particularly related to caption quality and alignment. While they employ some filtering mechanisms and conduct audits to assess error rates, the inherent variability in YouTube content production and captioning practices poses a challenge. Developing more robust methods for detecting and filtering out noisy data, potentially leveraging techniques from automatic speech recognition or natural language processing, would improve the overall quality of the corpus. This could involve automated caption alignment algorithms, quality assessment metrics, and possibly even community-based validation initiatives to harness the collective expertise of the Deaf community.\n4. Limited Demographic Analysis: The paper provides a basic analysis of signer demographics using automated classifiers for skin tone and perceived gender presentation. However, a more comprehensive and nuanced analysis of demographic representation, considering factors such as age, regional variations, signing styles, and disability intersectionality, is crucial for understanding potential biases and ensuring the inclusivity of models trained on the data. It would also be helpful to discuss the potential implications of these findings for the development of fair and unbiased sign language processing models.\n5. Limited Focus on Sign Language Generation: While the paper demonstrates the corpus's usefulness for sign language understanding tasks, it does not address its potential for sign language generation. Developing and evaluating sign language generation models, particularly those capable of producing fluent and expressive signing, remains a challenging area of research. Leveraging YouTube-SL-25 for this purpose could involve exploring techniques such as sequence-to-sequence modeling, conditional variational autoencoders, and reinforcement learning to generate realistic and natural-looking signing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4ikX4sQHwR", "reviewer": "Reviewer_nLUa", "summary": "The paper introduces YouTube-SL-25, a multilingual corpus of links to sign language videos. This dataset is designed to support pretraining for sign language models, and fine-tuning for various downstream sign language tasks. The paper provides benchmarks of sign-to-text translation and sign language identification across 4 sign languages.", "questions": "Please see weaknesses for suggestions and questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) The paper is well-organized and easy to follow.\n\n(2) YouTube-SL-25 addresses the shortage of data for sign language research. This large-scale dataset serves as a valuable resource for tasks like translation, caption alignment, and sign identification.\n\n(3) Models pretrained on YouTube-SL-25, compared to YouTube-ASL, show improvements across various sign languages, highlighting the effectiveness of this multilingual dataset.", "weaknesses": "(1) While the dataset offers significant benefits, much of the technical training relies on the FLEURS-ASL baseline [1]. Clarifying the unique technical contributions of this paper would enhance its value.\n\n(2) The dataset’s value is dependent on the availability of the videos. If these videos were to be removed, the overall impact of the paper would be significantly limited. To mitigate this risk, open-sourcing the framework used to filter sign language videos would benefit researchers in the long run and facilitate further development.\n\n(3) Currently, there is no information on whether the pretrained models will be open-sourced. Providing access to these models would greatly benefit the research community, allowing other researchers to leverage large-scale pretrained models and explore new approaches in sign language tasks. This is especially important as links to the videos may become unavailable over time.\n\n-----\n\n[1] Garrett Tanzer. Fleurs-asl: Including american sign language in massively multilingual multitask evaluation, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uHPDJMhd0X", "reviewer": "Reviewer_Yfwn", "summary": "The authors present YouTube-SL-25: a new dataset for sign language machine translation. This dataset is the first of its kind, in that it is significantly larger than predecessors and includes many languages for which previously, no labeled sign language machine translation data was available.\n\nThe dataset contains more than 3000 hours of video data for more than 25 sign languages, and was collected in two steps. First, YouTube was mined based on video tags related to sign language. Second, the authors (non-signers) performed a manual audit of the videos to identify high-quality video channels. Videos from those channels were used. An overview of the included sign languages, and relation to previous datasets for these sign languages (if any) is given in Table 1.\n\nKey features of the dataset are as follows: it is open-domain and it includes videos with multiple signers in the frame. These features make it an important dataset for future research that will eventually lead to sign language machine translation systems that are usable in the real world.\n\nA baseline method, adapted from previous work, is also presented. It is evaluated for four sign languages included in YouTube-SL-25. This method shows that multilingual training benefits both low-resource and (surprisingly) high-resource sign languages.", "questions": "# Figures\n\n- Figure 3: Please clarify the meaning of the acronym \"w.p.\" in the figure caption.\n\n# Main text\n\n- Page 2: Please provide exact figures for the amount of unique signers and sign languages, instead of >3000 and >25.\n- Page 3: Can you clarify in what way the first author is \"experienced\" with multiple sign languages?\n- Page 4: You mention in footnote 5: \"We accept learners as long as they seem relatively competent\". How was their competence evaluated? Is this competence evaluated by just one signer (the manuscript's first author), and how significant is this evaluation?\n- Page 5: Please add a reference [15] for the Monk skin tone ratings to the main text (final paragraph) as well, not just the figure caption.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "This manuscript presents a valuable new dataset for the sign language machine translation community. The dataset is large and multilingual, and opens up new possibilities for sign language machine translation research in many previously underrepresented sign languages. As such, this manuscript is a valuable contribution to the field of sign language processing.\n\n# Originality\n\nThis manuscript presents a labeled multilingual corpus for sign language translation. Previous corpora for sign language machine translation have often been monolingual (including the YouTube-ASL dataset, on which this manuscript builds). Research (in linguistics and in sign language processing) shows that sign languages have common elements, and therefore multilingual corpora can be beneficial for discovering and exploiting these common elements. The dataset is the first of its kind for several of the included low-resource sign languages and enables sign language translation research for these languages.\n\n# Quality\n\nThe authors perform a partial analysis into the demographics of the collected data, in terms of skin tone and perceived gender.\n\n# Clarity\n\nThe sections of the manuscript that discuss the dataset are clearly written and easy to follow. This is in contrast to the section on baseline methods, which is harder to follow (see \"Weaknesses\").\n\n# Significance\n\nA major problem in the domain of sign language translation is a lack of labeled data. The authors present a new dataset of unprecedented scale, with data for more than 25 languages and more than 3000 hours. Such resources are highly sought after and valued by the sign language processing community.", "weaknesses": "# Writing style and presentation\n\nThe authors make excessive use of footnotes (11 footnotes on 8 pages), which makes the text difficult to follow. I suggest moving some footnotes to the main text (3, 4, 6, 8, and 11 can definitely be in main text).\n\nFigure 3 is quite complex and is nearly impossible to understand without first thoroughly reading the previous work from which it was adapted ([32]). I suggest adding an explanation in the figure caption, or, if constrained by the page limits, in appendix. This way, readers do not have to read and understand [32] to continue reading this manuscript past Figure 3.\n\n# Methodology\n\nThe auditing of channels was performed by only one non-native hearing signer. The authors claim that this person is \"experienced\" in several sign languages, but they do not clarify what this experience entails and how this experience makes them a suited candidate for this auditing process. This leads to doubts about the robustness of the auditing process. Could the authors specify what \"experience\" means in this case? Having at least three auditors would also lead to a more robust auditing process.\n\nThe baseline method that is presented appears to be rather complex, especially compared to simple but effective sign language machine translation methods (e.g., https://arxiv.org/abs/2003.13830). It would be to the benefit of the reader if the method was simple, such that results are easier to interpret (the same argument is made by [33], which is cited by the manuscript's authors). Simplifying the baseline method would also give more room for analyzing results on more than just four sign languages and for further analyses of the results. Perhaps the authors could explain in the text why such a complex baseline method was chosen.\n\nThe results of the baseline method are discussed in limited detail. Scores are only provided in one translation metric (two, if including the appendix), and for the four evaluated languages, only one translation example is given per language. It would be interesting to see at least one good and one bad translation per language, and more than one if possible.\n\n# Relation to previous work\n\nThe authors compare their dataset primarily to two previously released datasets:\n- YouTube-ASL, because the same approach to data collection was used\n- JWSign, because it is also a multilingual sign language machine translation dataset\n\nThe authors do not mention RWTH-PHOENIX-Weather 2014 T (https://www-i6.informatik.rwth-aachen.de/~koller/RWTH-PHOENIX-2014-T/) at all, despite it being a pivotal historical contribution in the field. Despite the limitations of PHOENIX 2014T (see, e.g., https://aclanthology.org/2022.wmt-1.71.pdf), I believe it is important to at least mention the dataset in passing to place this manuscript in the broader context of sign language machine translation.\n\nThe baseline method is adapted from [32], but there is little comparison to previous work in the field. The manuscript, in particular section 3, would benefit from an explanation as to why this method and these input features (MediaPipe) are used. If space permits, the authors should present a short section on related work in the design of sign language machine translation. If space is limited, they can refer to recent survey papers.\n\n# Quality\n\nOn page 4, the authors mention \"whether the changes in captions occur at boundaries in sign language phrases (which are marked pretty universally with pauses and nonmanual signals like head tilt or eyebrow movement)\". It is my understanding that the boundaries of sign language phrases are a contentious topic in sign language linguistics, and that there certainly is no consensus on signals that are universal with respect or multiple sign languages. The authors should include a citation for this claim, or do not express this claim so strongly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)