SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Aashaka Desai $^{\beta}$  Lauren Berger $^{\gamma}$  Fyodor O. Minakov $^{\alpha}$  Vanessa Milan $^{\alpha}$  Chinmay Singh $^{\alpha}$  Kriston Pumphrey $^{\gamma}$  Richard E. Ladner $^{\beta}$  Hal Daumé III $^{\alpha, \delta}$  Alex X. Lu $^{\alpha}$  Naomi Caselli $^{\gamma}$  Danielle Bragg $^{\alpha}$
$^{\alpha}$ Microsoft Research  $^{\beta}$ University of Washington  $^{\gamma}$ Boston University  $^{\delta}$ University of Maryland {hal3,lualex,dabragg}@microsoft.com {aashakad,ladner}@cs.washington.edu,nkc@bu.edu
Sign languages are used as a primary language by approximately 70 million D/deaf people world-wide. However, most communication technologies operate in spoken and written languages, creating inequities in access. To help tackle this problem, we release ASL Citizen, the first crowdsourced Isolated Sign Language Recognition (ISLR) dataset, collected with consent and containing 83,399 videos for 2,731 distinct signs filmed by 52 signers in a variety of environments. We propose that this dataset be used for sign language dictionary retrieval for American Sign Language (ASL), where a user demonstrates a sign to their webcam to retrieve matching signs from a dictionary. Through our generalizable baselines, we show that training supervised machine learning classifiers with our dataset achieves competitive performance on metrics relevant for dictionary retrieval, with  $63\%$  accuracy and a recall-at-10 of  $91\%$ , evaluated entirely on videos of users who are not present in the training or validation sets.
Communication in sign language is an essential part of many people's lives. As 70 million deaf people world-wide primarily use a sign language [57], the meaningful inclusion of signing deaf people requires widespread access to sign languages both in individual communities and society-wide (§2.1). Towards this, over 100,000 students per year enroll in American Sign Language (ASL) classes [42], and the number of UN countries mandating provision of services in sign language has grown from 4 in 1995 to 71 in 2021 [56].
Despite increasing support for equal access to sign language, most existing information resources (like search engines, news sites, or social media) are written, and do not offer equitable access. Requiring signing deaf people to navigate information in a written language like English necessarily means forcing them to operate in a different, and potentially non-native language. Adapting resources for sign language input and output introduces significant technical challenges, which have motivated the development of computational methods for sign language recognition, generation, and translation [36].
In this work, we focus on ASL dictionary retrieval. Many existing ASL dictionaries catalog signs with English glosses: an out-of-context translation of a sign into one or more English words (e.g., LOBSTER, CLIMB_LADDER or HAD_ENOUGH). However, English lookup relies on knowing the English translation, and on a 1:1 relationship with English words, which often does not exist. Demonstrating a sign to look it up may be more natural, but is computationally more challenging due to the rich visual format and linguistic complexities. We seek to help address this problem of video-based dictionary retrieval, where a person demonstrates a single sign by video, and the system
returns a ranked list of similar signs. Dictionary retrieval fills a need for language learners, who may see a sign but not know the meaning. Moreover, dictionaries can contribute to documenting sign languages, and allow established signers to navigate dictionary resources directly in sign language.
To help advance dictionary retrieval, we present a crowdsourced dataset of isolated ASL signs, to support data-driven machine learning methods overcome limitations of prior isolated sign language recognition (ISLR) datasets (see Table 1 and §2.2). Model development typically requires a large, high-quality training set (i.e. large vocabulary, minimal label noise, representation of diverse signers and environments). Existing datasets are often filmed in a lab or scraped from online sources, which limit scale and diversity. Web scraping often does not acquire participant consent, which erodes community trust and leads to labeling challenges. To overcome such limitations, we build upon recent crowdsourcing proposals [10] to collect and release ASL Citizen – the first large-scale crowdsourced sign language dataset. Our dataset is the largest isolated sign dataset to date, newly representative of real-world settings and signer diversity, and collected with permission and transparency.
Using this new dataset, we adapt previous approaches to ISLR [41, 51] (§2.3) to the dictionary retrieval task, and release a set of baselines for machine learning researchers to build upon (§4). Dictionary retrieval requires algorithms to return a ranked list of signs, given an input video. A variety of methods can satisfy this output, but we focus on supervised deep learning methods, taking advantage of recent ISLR methods. We show that even without algorithmic advances, training and testing on our dataset doubles ISLR accuracy compared to prior work, despite spanning a larger vocabulary and testing on completely unseen users (§5). We also evaluate our dataset against prior datasets by comparing performance on a subset of overlapping glosses, and by comparing performance of learned feature representations from models trained on these datasets, showing further improvements in each case. Finally, through a series of downsampled training set experiments, we show that dataset size contributes to our improved performance.
Throughout this research, we take a culturally-sensitive and participatory approach to sign language computation. Sign languages are a cornerstone of Deaf culture and identity.<sup>1</sup> Previous works have noted that many efforts in sign language computation promote misconceptions about sign language, exploit sign language as a commodity, and undermine Deaf political movements seeking recognition of sign languages [62, 7, 21, 5]. Other works question if technologies being designed actually benefit Deaf communities, and document patterns where new tech is rejected by signing communities for being intrusive, clunky, or insensitive [29, 39]. This work abides by calls issued by disability scholars for better collaboration with Deaf communities, and for focus on solving real needs [29, 7, 62, 9], and demonstrates that aligning with these calls can actually help advance machine learning results.
In summary, our primary contributions are:
1. We provide a benchmark dataset and metrics for the dictionary retrieval task, with released code. Not only does this application have real utility to the signing community, but it grounds ISLR in a real-world problem setting, informing data collection and metrics.
2. We release the first crowdsourced dataset of isolated sign videos, containing diverse signers in real-world settings representative of real-world dictionary queries. It is about four times larger than prior isolated sign datasets, and the only of its scale collected with Deaf community involvement, consent, and compensation.
3. Our general baselines improve over ISLR accuracy achieved by prior datasets by more than double while using the same architectures, highlighting the impact of appropriately collected data on model performance.
For links to the dataset, code, and additional supplementary materials please visit https://www.microsoft.com/en-us/research/project/asl-citizen/.
2.1 Sign Languages and Deaf Culture
Sign languages are complex, with large vocabularies and unique phonological rules. Analogous to the sounds of speech, signs are composed of largely discrete elements (e.g.,手势, location, and
<sup>1</sup>By established standards, we use "Deaf" to refer to cultural identity, and "deaf" to audiological status.
Dataset	Vocab size	Videos	Videos/sign	Signers	Collection	Consent
RWTH BOSTON-50	50	483	9.7	3 Deaf	Lab	✓
Purdue RVL-SLL	39	546	14.0	14 Deaf	Lab	✓
Boston ASLLVD	2,742	9,794	3.6	6 Deaf	Lab	✓
WLASL-2000	2,000	21,083	10.5	119 Unknown	Scraped	✗
ASL Citizen	2,731	83,399	30.5	52 Deaf/HH	Crowd	✓
Table 1: Prior ISLR datasets for ASL compared to ASL Citizen. HH stands for hard of hearing.
movement) according complex rules [12]. English translations of isolated signs are called glosses, are written in all-caps, and may be single words or multiple words, typically not mapping 1-1 to English (just as in any other language translation). Sign execution varies across contexts, signers, and sociolinguistic groups [44]. These factors complicate representative data collection and modeling.
Isolated signs like those in dictionaries are considered "core" parts of the lexicon, but are only a subset of sign languages [11]. The "non-core" lexicon is generally not well represented in dictionaries or lexical databases, and includes complex constructions like depicting verbs, classifier constructions, and verbs that use time and space in ways that can be difficult to decompose into discrete parts [65, 25]. Continuous signing also includes coarticulation and grammar expressed with the face, body, and signing space, in addition to the hands [59]. As such, ISLR only address a fraction of sign language recognition. However, since our goal is dictionary retrieval, this work focuses on isolated signs.
Sign languages play a critical cultural role in Deaf communities and identity [27]. While our work focuses on American Sign Language (ASL), which is primarily used in North America, over 300 sign languages are used worldwide. Sign languages have been suppressed by political and educational authorities to force deaf individuals to integrate into hearing society by favoring speech at the expense of individual welfare [40]. These oppressive movements promote misconceptions that persist today (e.g., that sign languages are lesser languages, or that ASL is signed English), and Deaf activists work to combat these ideas [29, 21, 62]. This cultural context informs our decision to formulate ISLR as a dictionary retrieval problem, which grounds research in a meaningful real-world use case.
2.2 Previous ISLR Datasets
Our work focuses on ASL, which has four main public ISLR datasets: WLASL [41], Purdue RVLSSL[58], BOSTON-ASLLV [3] and RWTH BOSTON-50 [63], summarized in Table 1. WLASL offers four different vocabulary sizes, the largest containing 2,000 signs (WLASL-2000 in our tables). While BOSTON-ASLLV contains a larger vocabulary of 2,742 signs, the number of videos per sign is limited. Real-world signs vary greatly by user due to dialectal (e.g., geographic region) and sociolectal (e.g., age, gender, identity) variation. Models trained on a small number of dataset contributors, as seen in prior work, may not generalize well to diverse signers [3, 63, 58].
Existing datasets use varied collection and labelling techniques, impacting quality and size. Labcollected data [58, 63, 3] is usually high-quality with clean labels, but limited in size, participant diversity, and real-world settings. Data scraped from the internet may capture more users or settings, but varied contributor fluency and difficulty identifying and segmenting signing in videos impacts quality. Including ASL teaching resources and relying on English text therein (e.g. [41, 37]) still creates unreliable labels, as even linguists struggle to label signs without a conventional notation system [24, 32]. Teaching videos are also often professionally-recorded, with similar limitations to lab data. Moreover, scraping is typically not allowed by content creators or hosting platforms.
Sign language dataset design has profound implications for responsible AI [9]. Videos feature identifiable faces and are expensive and labor-intensive to create; contributor consent is paramount. Prior work has proposed crowdsourcing [10] and addressing privacy concerns [8] as ways to collect larger representative datasets. Our work explores such ideas at scale, implementing a crowdsourcing platform with optimized versions of tasks proposed in [10] and partnering with Deaf community throughout. As a result, we present the first crowdsourced sign video dataset containing a large vocabulary and representative signers in everyday settings, collected with consent.
2.3 Isolated Sign Language Recognition Methods
Research on isolated sign language recognition (ISLR) is increasing, as evinced by the growing number of literature reviews in the space [36, 17, 55, 18, 53, 22, 48, 38, 1]. Early approaches relied on handcrafted features and classic machine learning classifiers [54, 26, 46, 14, 45], typically on small datasets and vocabularies. More recent approaches have shifted to deep learning as larger datasets have become available. Appearance-based methods operate directly on video frame inputs: approaches include spatially pooled convolutional neural networks [47, 41] and transformers [20, 4]. Alternatively, pose-based methods [41, 51, 4] fit models on keypoints extracted using human pose models (e.g. OpenPose [13], MediaPipe [43]). Despite the breadth of approaches, state-of-the-art ISLR methods still have relatively low recognition performance, Prior baselines for ISLR using standard approaches achieve about  $30\%$  accuracy [41, 51] over realistic vocabulary sizes  $(2,000+$  signs), which is largely inadequate for real-world use. More recent works have improved performance with algorithmic advancement: for example, using multiple modalities simultaneously (optical flow, depth map, depth flow) [35], or by introducing new priors (such as hand modeling) [33], or by leveraging cross-lingual semantic information to learn relations between classes[64]. While these models achieve better performance (about  $60\%$  accuracy), the complexity of these methods may obscure the impact of the dataset. Thus for our baselines, we use more generalizable techniques as seen in prior dataset works (such as I3D and ST-GCN) [15, 61].
2.4 Sign Language Dictionaries
Dictionaries are a meaningful application of ISLR to Deaf community members [34, 7]; they play a cultural role in language documentation, and are valuable tools for language users and learners. However, creating effective sign lookup systems is difficult. English-to-ASL dictionaries (e.g. [52]) accept written queries and so can leverage text-based matching methods, but ASL-to-English and ASL-to-ASL dictionaries cannot because no standard sign language writing system exists. This then leaves sign language users to operate in a non-native language to access dictionary resources. Similarly, novice signers may not know the English gloss to leverage English-to-ASL dictionaries.
To address these challenges, two main approaches are used in ASL-to-English dictionaries: feature-based lookup and example-based lookup. In feature-based lookup, users specify parameters of the sign they seek (e.g., the handshape, the body location involved, the motion, etc.), and a list of top matching signs are returned [6]. While this simplifies the lookup problem, unlike English spelling, these features are not conventional and are not widely used or taught, so users may not be familiar with how to make use of them. Novice learners may especially have trouble noticing and remembering these parameters.
In example-based lookup, users demonstrate a sign by video, and receive a list of top matching signs. While this may be more accessible for users, it is more challenging computationally, now requiring video processing to complete lookup [60]. Because this is largely unsolved, human-computer interaction (HCI) research has primarily focused on understanding potential dictionary use through wizard-of-oz methods [31, 2, 30, 28]. Our work advances ISLR for dictionary retrieval using generalizable methods, potentially enabling functional example-based dictionaries to be created and studied.
3 ASL Citizen Dataset Creation
We build on prior work [10], which piloted the first crowdsourcing tasks for sign language data collection in a small user study. In this work, we scale data collection with a longer-term deployment with fluent Deaf signers. We also enhanced the crowdsourcing tasks proposed in [10] to improve recording efficiency and data quality (details in Appendix). This method secures participant consent, and curates data appropriate for work on dictionary lookup; participants are asked to contribute videos for a communal dictionary, recording videos in real-world settings, similar to real-world dictionary queries. The task design also eliminates labelling challenges by collecting pre-labelled content.
The platform provided each user with the full vocabulary of 2,731 signs (from ASL-LEX [16, 50]), sorted so that signs with the fewest videos are first, to help balance the dataset across signs. For each
Train	Val	Test
Users	35	6	11
Videos	40,154	10,304	32,941
User distrib.	60% F	83% F	55% F
Video distrib.	54% F	71% F	55% F
Table 2: Statistics for ASL Citizen dataset splits.
Figure 1: Random ASL Citizen video stillframes.
sign, they first viewed a "seed" video of the sign filmed by a highly proficient, trained ASL model. This "seed signer" was a paid research member, and used a high-resolution camera. Participants then recorded their own version of the demonstrated sign. As in [10], users could re-play videos, and re-record or delete. We provided optional English gloss, hidden by default to encourage focus on the ASL. For every 300 videos, participants received a $30 gift card, for up to 3,000 signs. Those who completed the vocabulary re-visited the least-recorded signs. Providing demographics was optional.
We took steps to help ensure that our data collection was culturally sensitive and participatory. Deaf researchers were involved in every aspect of the research, and made direct contact with participants. All recruitment and consent materials were presented in an ASL-first format, featuring short ASL videos. Participants were recruited through relevant email lists and snowball sampling of Deaf researchers' social networks. All procedures were reviewed and approved by IRB.[2]
3.2 Data Verification and Cleaning
To help ensure data quality, we engaged in verification and cleaning procedures under close guidance from our ethics review board. First, we removed empty videos automatically, by removing those under  $150\mathrm{KB}$  in size or where YOLOv3 [49] did not detect a person ( 50 videos). We manually reviewed the first and last videos recorded by each participant on each day and random samples throughout, checking for a list of sensitive content provided by our ethics and compliance board. Three types of personal content were identified for redaction: another person, certificates, and religious symbols. To protect third parties, we used YOLOv3 to detect if multiple people were present. For these videos and others with identified personal content, we blurred the background using MediaPipe holistic user segmentation. We blurred a subset of pixels for one user, since the personal content was reliably limited to a small area. We also removed one user's videos, who recorded many videos without a sign in them, and videos of a written error message. Finally, we manually re-reviewed all videos.
In our reviews, we did not identify any inappropriate content or bad-faith efforts. In total, we blurred the background of 268 videos where a second person was detected automatically, 293 additional videos with sensitive content, and 32 additional videos with a person missed by the automatic detection. We blurred a small fixed range of pixels for 2,933 videos, and omitted 513 videos where the blurring was insufficient or an error message (resulting from the data collection platform) showed.
3.3 ASL Citizen Dataset Benchmark for Dictionary Retrieval
Our final cleaned dataset, ASL Citizen, contains 83,399 videos corresponding to 2,731 signs recorded by 52 participants, including the seed signer (Table 1). This is the first crowdsourced ISLR dataset, and the largest for ASL (or any other sign language) to date. Our vocabulary was taken from ASL-LEX [16], a database of lexical and phonological properties of signs. As such, each ASL Citizen video maps to an ASL-LEX entry, enabling the use of ASL-LEX's linguistic annotations (e.g., the手势 of each sign in our dataset). Each sign has multiple recordings ( $\mu = 30.5$ ,  $\sigma = 1.7$ ).
Of our participants, 49 identify as Deaf and 3 as hard of hearing; 32 as female and 20 as male; and with ages 20-72 years old ( $\mu = 36.16$ ,  $\sigma = 14.2$ ,  $n = 49$ ). These signers come from 16 U.S. states, with 2-65 years of ASL experience ( $\mu = 30$ ,  $\sigma = 15.12$ ,  $n = 48$ ), with mean ASL level of 6.45.
We expect our videos to be consistent with our chosen dictionary retrieval task (i.e. participants resemble webcam users demonstrating signs to a dictionary). First, all users were informed they were contributing to a dictionary. Second, participants recorded videos in a variety of everyday settings,
2Microsoft and Boston University IRBs reviewed the project. Microsoft serves as the IRB of record (#418).
In this paper, we use ASL Citizen Version 1.0. All stats and experiments were run with this dataset version.
spanning varied illumination, background, resolution, and angle. Since our videos are self-recorded, there is also variability in when users start and finish signing in videos (i.e. amount of padding), and the speed, repetition, and execution of signs. We consider this variability to be valuable as it is within the scope of "in-the-wild" dictionary queries, so we do not filter or standardize these variables.
Dataset Splits. We also release standardized splits of our 52 users into training, validation and test sets (Table 2), attempting to balance by female-to-male gender ratio (no participants self-identified as non-binary). Importantly, we establish splits such that users in the validation and test sets are unseen during training. While previous work randomly split videos [41] (so users in the test set may be seen in training), we felt it critical to evaluate on unseen users to align with our dictionary retrieval problem; it is unlikely that a user looking up a sign would be in the training set of a deployed model. Accordingly, the seed signer is placed in the training split. We also sought to provide a large test set. While still leaving sufficient videos for training (over  $2.5 \times$  that of the previous largest dataset [41]), a large test set with multiple users not only offers more robust performance estimates, but also offers the future potential for a wider range of methods (e.g., unsupervised domain adaptation).
Sign Ranking and Metrics. Dictionary retrieval requires models to return a ranked list of signs, so we consider standard information retrieval metrics: recall-at-K (for  $\mathrm{K} = 1, 5$ , and 10, where recall-at-1 is the same as accuracy), discounted cumulative gain (DCG) and mean reciprocal rank (MRR). Recall-at-K, measured by determining if the correct sign is in the top K rankings, allows us to consider scenarios where users may look at only the first K signs returned for their query [30]. Alternatively, DCG and MRR evaluate the overall ranking of the correct sign in the entire list. For all these metrics, a higher score indicates the correct sign is earlier in the ranking. DCG uses a log scale, so is more sensitive to order at the top of the ranking. The Appendix provides formulas and additional details.
4 Methods and Training Details
We train two fundamentally different types of machine learning methods on the ASL Citizen dataset: an appearance-based approach, I3D, which is based on a 3D convolutional network applied directly to the video frames [15]; and a pose-based approach, ST-GCN, which preprocesses the video to extract pose information, on which we train a temporal graph convolutional network [61]. By choosing simple representative models, I3D and ST-GCN, we aim to showcase the quality of the data while minimizing the assumptions encoded into the methods. Both are classifiers with an output space equal to the size of the vocabulary. For both, to generate a ranked list of retrieved signs, we sort the output probabilities across labels.