SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: Metrics for ASL to English translation on How2Sign. Our models either train from scratch or finetune a pretrained T5 checkpoint, and are trained on How2Sign (H2S) only, YouTube-ASL (YT-ASL) only, a mixture of H2S and YT-ASL, or YT-ASL and then finetuned on H2S.
Approach	Training Schedule	BLEU-1	BLEU-2	BLEU-3	BLEU	BLEURT
Álvarez et al. [3]	H2S	17.40	7.69	3.97	2.21	-
GloFE-VN [25]	H2S	14.94	7.27	3.93	2.24	31.65
Tarrés et al.[40]	H2S	34.01	19.30	12.18	8.03	-
Ours (no pretraining)	H2S	13.92	4.69	1.82	0.86	30.65
YT-ASL	14.53	5.47	2.61	1.41	29.55
YT-ASL + H2S	28.60	14.56	8.68	5.60	37.72
YT-ASL → H2S	28.38	15.41	9.55	6.26	39.40
Ours (pretrained)	H2S	14.96	5.11	2.26	1.22	29.98
YT-ASL	20.93	10.35	6.14	3.95	34.98
YT-ASL + H2S	36.35	23.00	16.13	11.89	44.78
YT-ASL → H2S	37.82	24.13	16.92	12.39	46.63
We ablate across four different training schedules:
- H2S. We train only on How2Sign, not YouTube-ASL, for a like-for-like comparison with prior methods.
- YT-ASL: We train only on YouTube-ASL, and evaluate on How2Sign zero-shot.
- YT-ASL + H2S: We train on a mixture of How2Sign and YouTube-ASL, mixed in proportion to the size of the datasets.
- YT-ASL  $\rightarrow$  H2S: We train on YouTube-ASL, then finetune on How2Sign.
We also ablate the effect of pretraining on English text by comparing models trained from scratch using the T5.1.1-Base architecture, vs. finetuned from the T5.1.1-Base pretrained checkpoint.
We train with a batch size of 128 and learning rate of 0.001 with Adafactor [36]; other hyperparameters are the T5X defaults. For models trained solely on How2Sign data, we train for 20,000 steps. For models trained on YouTube-ASL (including with How2Sign mixed in), we train for 200,000 steps. When finetuning on How2Sign after training on YouTube-ASL, we finetune for an additional 5,000 steps. Each 1,000 steps takes approximately 0.25 TPUv4-hours.
Following prior work, we present BLEU [29] and BLEURT [34] scores. BLEU scores are computed using SacreBLEU [30] version 2, with all default options. BLEURT scores are computed using checkpoint BLEURT-20 [31, 14]. We decode using beam search with a beam width of 5.
See Table 3 for metrics comparing our models to prior works on How2Sign [3, 25, 40]. The best results come from training on YouTube-ASL from a pretrained checkpoint, then finetuning on How2Sign, which achieves 12.39 BLEU vs. the state of the art of 8.03 BLEU [40]. The base model achieves 3.95 BLEU zero-shot, which is nontrivial but substantially worse than the finetuned score. Factors that could contribute to this gap include train/test leakage of signers and narratives, How2Sign's narrow domain, and the extra  $\sim 10\%$  training data it represents.
Results are substantially worse when training from scratch, which suggests that T5's English pretraining gives the model a better initialization, as De Coster et al. [11] found for frozen pretrained language models. Results are abysmal when trained without YouTube-ASL. The most direct comparison of our approach to prior work is T5 trained from scratch on How2Sign only, which reaches just 0.86 BLEU, despite training on the same data as Tarrés et al. [40]'s 8.03 BLEU. This might be explained by their use of a pretrained video encoder and various decisions they made to optimize for small amounts of data (smaller network, more text normalization, careful hyperparameter sweep), whereas we used a less tuned configuration that was intended for larger datasets.
Table 4: Qualitative examples from our best finetuned and zero-shot models, on sentences sampled from How2Sign by Tarrés et al. [40]. See Table 5 in the Appendix for the complete set of examples.
(1)	Reference	And that's a great vital point technique for women's self defense.
Tarrés et al.	It's really a great point for women's self defense.
Ours (zero-shot)	It's really great, especially for women who are facing barriers.
Ours (finetuned)	It's really great for women's self defense.
(2)	Reference	In this clip I'm going to show you how to tape your cables down.
Tarrés et al.	In this clip I'm going to show you how to improve push ups.
Ours (zero-shot)	This video will show how to use the code online.
Ours (finetuned)	In this clip we're going to show you how to cut a piece of clay.
(3)	Reference	In this segment we're going to talk about how to load your still for distillation of lavender essential oil.
Tarrés et al.	Ok, in this clip, we're going to talk about how to fold the ink for the lid of the oil.
Ours (zero-shot)	This video will discuss how to submit a digital form for the survey.
Ours (finetuned)	In this clip we're going to talk about how to feed a set of baiting lizards for a lava field oil.
See Table 4 for qualitative examples of the translations produced by our best finetuned and zero-shot models, on sentences sampled from How2Sign by Tarrés et al. [40]. The translations capture elements of the reference translation but are clearly not yet of usable quality. The zero-shot predictions hew less closely to the references, but the errors usually make sense in light of the sign language input. For example, in (1), the sign used to mean "defense" also means "barrier".
In this paper, we presented YouTube-ASL, a new, publicly available parallel corpus for American Sign Language and English that is  $\sim 3x$  the size and has  $\sim 10x$  as many unique signers as the largest prior ASL dataset. Our key improvement over prior work is that we used automatic tagging followed by human filtering to increase mining recall without harming precision. We demonstrated the value of this data with a simple baseline built from off-the-shelf components (MediaPipe Holistic and T5) that achieves a new finetuned state of the art in ASL to English translation on How2Sign, 12.39 BLEU. We also reported a zero-shot score of 3.95 BLEU, a first for sign language translation. We hope that YouTube-ASL will be immediately useful for research on methods for sign language translation and caption alignment, as well as tools for automatic annotation/filtering of new sign language datasets. Because YouTube-ASL has so much signer variety, including across dialect and skill level, it may be less useful for generation than recognition tasks.
While our baseline improves upon prior work, even the finetuned translations are subjectively low-quality and are not yet useful in the real world. We hope that more refined modeling approaches will provide better results with the same data, but despite our and prior efforts, ASL is still a low-resource language by modern standards [20]—let alone the many other sign languages of the world, most of which are even less resourced. Future work may look to address this by mining broader datasets with other kinds of supervision, and exploring multilingual transfer at larger scales. As model quality improves, future work should perform more comprehensive evaluations to understand differences across domains, dialects, levels of fluency, signer appearance, and other such factors.
7 Ethical Considerations
Sign language datasets pose privacy challenges, because the signer's appearance (body, facial expressions), which is personally identifying, is a vehicle for the language itself. Video anonymization techniques are not yet mature enough to be useful in this regard. Our corpus is composed of sign language content that uploaders made publicly visible on YouTube, and we release only video IDs so that changes to the underlying videos are automatically reflected in the corpus. While the corpus cov
ers a broader variety of channels than prior works, this does not mean it is necessarily representative of the signing population—or even if it were representative, that models trained on it would work equally well for everyone.
We train our models on reduced poses as a form of anonymization, but this is not suitable for all modeling approaches and may harm model quality. Until sign language translation models are closer to usable quality, there is little risk of societal harm, except that individuals or organizations mistakenly rely on models that are inadequate. As we approach that point, sign language processing will adopt the risks of natural language processing in general, but with a great potential to improve accessibility for Deaf/Hard of Hearing people.
[1] Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. YouTube-8M: A large-scale video classification benchmark, 2016. URL https://arxiv.org/abs/1609.08675.
[2] Samuel Albanie, Gül Varol, Liliane Momeni, Hannah Bull, Triantafyllos Afouras, Himel Chowdhury, Neil Fox, Bencie Woll, Rob Cooper, Andrew McParland, and Andrew Zisserman. BBC-Oxford British Sign Language dataset, 2021. URL https://arxiv.org/abs/2111.03635.
[3] Patricia Cabot Álvarez, Xavier Giro Nieto, and Laia Tarrés Benet. Sign language translation based on transformers for the How2Sign dataset. 2022. URL https://imatge.upc.edu/web/publications/sign-language-translation-based-transformers-how2sign-dataset.
[4] Danielle Bragg, Oscar Koller, Mary Bellard, Larwan Berke, Patrick Boudreault, Annelies Braffort, Naomi Caselli, Matt Huenerfauth, Hernisa Kacorri, Tessa Verhoef, Christian Vogler, and Meredith Ringel Morris. Sign language recognition, generation, and translation: An interdisciplinary perspective. In Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS '19, page 16-31, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450366762. doi: 10.1145/3308561.3353774. URL https://doi.org/10.1145/3308561.3353774.
[5] Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. Neural sign language translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018. URL https://doi.org/10.1109/CVPR.2018.00812.
[6] Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. Sign language transformers: Joint end-to-end sign language recognition and translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. URL https://doi.org/10.1109/CVPR42600.2020.01004.
[7] Necati Cihan Camgoz, Ben Saunders, Guillaume Rochette, Marco Giovanelli, Giacomo Inches, Robin Nachtrab-Ribback, and Richard Bowden. Content4all open research sign language translation datasets, 2021. URL https://arxiv.org/abs/2105.02351.
[8] Z. Cao, G. Hidalgo, T. Simon, S. Wei, and Y. Sheikh. OpenPose: Realtime multi-person 2D pose estimation using part affinity fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(01):172-186, jan 2021. ISSN 1939-3539. doi: 10.1109/TPAMI.2019.2929257. URL https://doi.ieeecomputersociety.org/10.1109/TPAMI.2019.2929257.
[9] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. URL https://doi.org/10.1109/CVPR.2017.502.
[10] Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. FLEURS: Few-shot learning evaluation of universal representations of speech. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 798-805, 2023. URL https://doi.org/10.1109/SLT54892.2023.10023141.
[47] Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li. Improving sign language translation with monolingual data by sign back-translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1316–1325, June 2021. URL https://doi.org/10.1109/CVPR46437.2021.00137.
This document contains supplementary material for the YouTube-ASL paper.
A.1 Full Qualitative Results
Table 5: The complete set of qualitative examples from our best finetuned and zero-shot models, on sentences sampled from How2Sign by Tarrés et al. [40].
(1)	Reference	And that's a great vital point technique for women's self defense.
Tarrés et al.	It's really a great point for women's self defense.
Ours (zero-shot)	It's really great, especially for women who are facing barriers.
Ours (finetuned)	It's really great for women's self defense.
(2)	Reference	In this clip I'm going to show you how to tape your cables down.
Tarrés et al.	In this clip I'm going to show you how to improve push ups.
Ours (zero-shot)	This video will show how to use the code online.
Ours (finetuned)	In this clip we're going to show you how to cut a piece of clay.
(3)	Reference	In this segment we're going to talk about how to load your still for distillation of lavender essential oil.
Tarrés et al.	Ok, in this clip, we're going to talk about how to fold the ink for the lid of the oil.
Ours (zero-shot)	This video will discuss how to submit a digital form for the survey.
Ours (finetuned)	In this clip we're going to talk about how to feed a set of baiting lizards for a lava field oil.
(4)	Reference	You are dancing, and now you are going to need the veil and you are going to just grab the veil as far as possible.
Tarrés et al.	So, once you're belly dancing, once you've got to have the strap, you're going to need to grab the thumb, and try to avoid it.
Ours (zero-shot)	he's dancing a lot. Now he needs a hat and a chain
Ours (finetuned)	Their hopping and dancing is now, they're going to need their squat and squat and they're going to be able to move independently.
(5)	Reference	But if you have to setup a new campfire, there's two ways to do it in a very low impact; one is with a mound fire, which we should in the campfire segment earlier and the other way to setup a low impact campfire is to have a fire pan, which is just a steel pan like the top of a trash can.
Tarrés et al.	And other thing I'm going to talk to you is a little bit more space, a space that's what it's going to do, it's kind of a quick, and then I don't want to take a spray skirt off, and then I don't want it to take it to the top of it.
Ours (zero-shot)	But if you have to set up a new campfire, you have to set up a campfire. You have to do it in a campfire, or set up a tentfire.
Ours (finetuned)	But if you have to set up a new campfire, there are two ways to do a low impact fire, one is a cone fire, which we have to do in the tent earlier, and the other one is to set up a campfire in a fire pan.
(6)	Reference	So, this is a very important part of the process.
Tarrés et al.	It's a very important part of the process.
Ours (zero-shot)	Wash your hands.
Ours (finetuned)	Alright, let's get started.
B Datasheets for Datasets
We provide documentation of the dataset based on Datasheets for Datasets 7.
<sup>7</sup>https://arxiv.org/pdf/1803.09010.pdf
For what purpose was the dataset created? The dataset was created primarily to serve as training data for ASL to English machine translation; prior datasets are smaller and have fewer unique signers. We used human annotators to identify high-quality ASL videos with well-aligned captions, but it was not feasible to manually correct or align any of the included captions. This is generally sufficient for translation, but slightly less ideal for tasks like ASL to English caption alignment, where consistent alignment standards might be desired. The dataset is probably also less suitable for English to ASL translation, due to the signing variation across videos, though this may be addressed with methods for improved controllability.
Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? This dataset was created by Dave Uthus, Garrett Tanzer and Manfred Georg for Google.
Who funded the creation of the dataset? Google.
What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Each instance is the video id of a YouTube video.
How many instances are there in total (of each type, if appropriate)? There are 11,093 video ids.
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? This contains a subset of videos available on YouTube of people signing in American Sign Language with English captions. This is not strictly representative of the larger set, as we have applied a combination of automatic and manual filtering techniques to find high quality videos with high-quality captions.
What data does each instance consist of? Each instance consists of a single video id, which represents an ASL video with associated English captions.
Is there a label or target associated with each instance? The English captions may be considered the target for each instance, but this depends on the task being attempted.
Are there recommended data splits (e.g., training, development/validation, testing)? No. YouTube datasets can change over time due to the nature of the platform (videos can be made private or deleted), thus there are no recommended splits.
Are there any errors, sources of noise, or redundancies in the dataset? Yes. During the annotation process, we allowed the annotators to mark whole channels with the same annotations, so there may be some videos which are not of the same quality as the rest of the channel. There may also be minor errors in the videos or captions that the annotators explicitly deemed acceptable.
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? The dataset is not self-contained, as it consists of YouTube video ids only. As such, there is no guarantee that the dataset will remain constant over time. The actual videos themselves fall under YouTube's Terms of Service https://www.youtube.com/static? template=terms.
Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor- patient confidentiality, data that includes the content of individuals' non-public communications)? No, the dataset consists of video ids for public videos only, and does not rehost any of the underlying data.
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? The video ids comprising the dataset itself are random identifiers. The videos referenced by our video ids are hosted by YouTube and therefore subject to YouTube's community guidelines. We or our annotators did not encounter any such content.
Does the dataset identify any subpopulations (e.g., by age, gender)? The dataset does not identify any subpopulations.
Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? The video ids comprising the dataset itself are random identifiers. The videos referenced by our video ids include people signing in ASL, which inherently includes the person's appearance. Our dataset does not provide any extra information about these people that was not already publicly available from their uploaded videos, and respects when videos are deleted/made private by virtue of using video ids.
Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals race or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? As with the previous question, the videos referenced by our video ids may contain information about many topics, if the signer chose to discuss that information in their publicly uploaded video. Our dataset does not provide any extra information and respects deletions.
How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? The videos referenced by each video id instance consist solely of data that is directly observable (uploaded videos and captions). The selection of video ids is implicitly decided by a combination of automatic and manual filtering in order to ensure a relatively high quality level.
What mechanisms or procedures were used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)? A combination of software programs and manual annotations were used to select preexisting YouTube videos.
If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)? Not applicable.
Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)? The authors collected the initial collection of video ids, and annotators were hired to help annotate the videos for filtering purposes.
Over what timeframe was the data collected? We collected videos up to January 2022.
Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)? The dataset consists of references to videos that include people, but doesn't collect or release any new information about those people.
Were the individuals in question notified about the data collection? No, as we only provide videos and no further information about the videos.
If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? As we only provide ids and not the raw content, users removing the video will make them no longer available for use in our dataset.
Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? Yes, we filtered out videos that were not relevant, of poor quality, had poor captions, etc. The result is a list of video ids, which point to unmodified videos.
Was the "raw" data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)? The original set of video ids will not be made available.
Is the software that was used to preprocess/clean/label the data available? No.
Has the dataset been used for any tasks already? None prior to the baselines provided in this paper.
What (other) tasks could the dataset be used for? In addition to the intended task of ASL to English translation, the dataset could be used for related sign language understanding tasks like caption alignment, and potentially for sign language generation tasks like English to ASL translation, or sign language tasks that do not require captions.
Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? The dataset was filtered for high-quality videos and captions, which includes a variety of signing styles and skill levels, as long as they can be understood by an ASL user and are captioned correctly. This amount of variety may be unideal for tasks like generation where consistency is preferred. Even though it is varied, it is still not necessarily representative of the signing community as a whole, and should be treated as such.
Are there tasks for which the dataset should not be used? This dataset should not be used as a benchmark for comparing models across time, because the data that YouTube-ASL is derived from will change over time with video deletions or other modifications.
Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? The dataset is open sourced.
How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? GitHub.
When will the dataset be distributed? It is currently available.