SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Data Preprocessing For our I3D model, we preprocess videos by standardizing to 64 frames. Due to variance in sign length and user execution, our dataset videos differ in length. While previous works use random temporal crops to standardize frame lengths during training [41], we reasoned this practice might alter sign semantics: some signs are compounds of multiple signs, and temporal cropping may reduce the sign to just one root sign. Instead, we standardize training and evaluation videos by skipping frames: for videos longer than 95 frames, we skip every other frame, and for videos longer than 159 frames, we take every third frame. Videos shorter than 64 frames are padded with the first or last frame, and longer videos had even numbers of frames removed from the start and end. Finally, we augmented with random horizontal flips to simulate left and right handed signers.
For our ST-GCN model, we extracted keypoints using MediaPipe holistic. We use a sparse set of 27 keypoints previously established by OpenHands [51]. Following previous practice, extracted keypoints are center scaled and normalized using the distance between the shoulder keypoints. Since our videos contain a higher frame-rate than ISLR datasets analyzed by OpenHands, we cap the maximum frames to 128, and downsample frames evenly if the video is longer. As with OpenHands, we apply random shearing and rotation transformations during training as data augmentation.
Model Structure and Training We train our I3D model for up to 75 epochs using learning rate 1e-3 and weight decay 1e-8, with an Adam optimizer and ReduceLRonPlateau scheduler with patience 5. As we observed that a cross-entropy loss on the entire video alone led to poor convergence, we also employ a weakly supervised per-frame loss previously used in the Charades Challenge [19], where the cross-entropy loss is also applied to each video frame. Our final loss is an average of the full-video cross-entropy loss and this per-frame cross-entropy loss. We train our ST-GCN model
Model	Train Data	Test Data	DCG	MRR	Rec@1	Rec@5	Rec@10
I3D	ASL Citizen	ASL Citizen	0.7913	0.7332	0.6310	0.8609	0.9086
ST-GCN	ASL Citizen	ASL Citizen	0.7637	0.6997	0.5952	0.8268	0.8813
Table 3: Appearance and pose-based baseline results, representing a phase transition in ISLR, up from mid-30% accuracy (Rec@1) in prior work. The best results on ASL Citizen test set are bold.
for a maximum of 75 epochs using a learning rate of 1e-3 using an Adam optimizer and a Cosine Annealing scheduler. For each model, we selected the best-performing checkpoint on the validation set for analysis on our test set. Code and model weights are released publicly alongside our dataset.
5 Results and Analysis
5.1 Novel Benchmarks
We trained our appearance-based I3D model [15] on ASL Citizen to establish an initial baseline, with  $63.10\%$  top-1 accuracy (recall-at-1), DCG 0.791, and MRR 0.733 (first row in Table 3). This accuracy is notable, given our problem difficulty – our dataset has completely unseen users, and spans one of the largest vocabulary sizes in ISLR to date (2,731 signs). The pose-based ST-GCN model performs similarly, but consistently worse by a few percentages across metrics, but still substantially better than any previous reported results on datasets of similar size and complexity. For comparison, due to the number of classes, random guessing would yield  $0.04\%$  expected accuracy.
In previous work, appearance-based and pose-based models have generally shown competitive performance. Pose-based methods reduce information and potentially introduce errors during keypoint extraction, at the benefit of making relevant features more accessible and standardized compared to raw pixels. However, this standardization may not outweigh errors and information loss when the training set is diverse enough to train general appearance-based methods. Consistent with this, our pose-based model performance slightly lags behind that of our appearance-based model.
5.2 Comparison to Prior Datasets
We subsequently seek to understand how much our dataset advances the performance of ISLR models in ASL, compared to previous datasets. To do this, we compared our model trained on our dataset, to a public model trained on the previously largest ASL ISLR dataset, WLASL-2000 [41]; see Table 1. We made no changes to model architecture compared to the WLASL I3D model, meaning that any improvements are because of the training data, not model capacity.
Directly comparing these models is challenging because they use independent gloss mappings. As a result, the number of classes differs, and the same English gloss in one model may refer to a different sign in the other. We overcome this challenge by comparing the models in two ways (Table 4).
First, we directly compared metrics on our test set to previously reported accuracy on the WLASL-2000 test set. While this comparison does not account for potential differences in test set difficulty, we believe our test dataset is more challenging than that of WLASL-2000. Unlike WLASL-2000, we evaluate on unseen signers, and our vocabulary is larger. Comparing top-1 accuracy, we achieve  $63.10\%$  on the ASL Citizen test set, while [41] reports  $32.48\%$  on the WLASL-2000 test set.
Second, we reduced our test set to a subset of glosses that refer to the same sign in ASL Citizen and WLASL-2000, and used this test set to compare models trained on ASL Citizen and WLASL-2000. We used a reduced version of ASL Citizen's test set to ensure it would not contain anyone seen in training by either model, enabling a more fair comparison. We excluded any sign with a documented variant in either dataset, and looked for exact gloss matches across datasets. This produced a set of 1,075 glosses ("Subset" in our tables). To verify matches, an author fluent in ASL examined one example per sign from each dataset for 100 random signs, and did not find any discrepancies. To control for models outputting different numbers of classes, we recalculated the softmax on only the logits for the 1,075 overlapping glosses (effectively excluding predictions outside this set). Under this comparison, top-1 accuracy is  $74.16\%$  for our model vs.  $8.49\%$  for the WLASL-2000 model. We hypothesize that the accuracy drop for WLASL-2000, also relative to its original test set  $(32.48\%)$ , is because the model was originally evaluated on seen users, and fails to generalize to unseen users.
Model	Train Data	Test Data	DCG	MRR	Rec@1	Rec@5	Rec@10
I3D	ASL Citizen	ASL Citizen	0.7913	0.7332	0.6310	0.8609	0.9086
I3D	WLASL-2000	WLASL-2000	--	--	0.3248	0.5731	0.6631
I3D	ASL Citizen	Subset	0.8631	0.8228	0.7416	0.9252	0.9529
I3D	WLASL-2000	Subset	0.2794	0.1477	0.0849	0.2011	0.2744
ST-GCN	ASL Citizen	ASL Citizen	0.7637	0.6997	0.5952	0.8268	0.8813
ST-GCN	WLASL-2000	WLASL-2000	--	--	0.2140	--	--
I3D Features	ASL Citizen	ASL Citizen	0.7652	0.6985	0.5803	0.8450	0.9023
I3D Features	WLASL-2000	ASL Citizen	0.3153	0.1791	0.0985	0.2528	0.3433
I3D Features	Kinetics	ASL Citizen	0.1234	0.0128	0.0041	0.0136	0.0227
I3D Features	ASL Citizen	Subset	0.8494	0.8039	0.7107	0.9216	0.9565
I3D Features	WLASL-2000	Subset	0.4039	0.2705	0.1649	0.3785	0.4852
Table 4: Results comparing across datasets and feature representations: ASL Citizen, prior WLASL-2000 datasets, and a Subset of ASL Citizen "matched" to WLASL-2000. Encoding of best results for test sets - ASL Citizen: **bold**, Subset: **italics**, other datasets: **underlined**. Gray rows are from Table 3.
DCG	MRR	R@1	R@5	R@10
0%	0.103	0.002	0.000	0.002	0.004
25%	0.393	0.262	0.162	0.365	0.467
50%	0.679	0.597	0.481	0.736	0.809
75%	0.763	0.698	0.592	0.830	0.880
100%	0.791	0.733	0.631	0.861	0.907
Table 5: Downsampling results. More videos per sign improves performance.
Figure 2: Impact of training data size on recall. We fit a polynomial function to each series.
Together, these results showcase the quality of data in ASL Citizen. Even with standard ISLR algorithms like I3D and ST-GCN, we achieve over  $63\%$  top-1 accuracy. In the most naive comparison with independent test sets, this indicates a significant improvement (approx. 2x) in performance compared to prior datasets (63.10% vs. 32.48%). In more standardized comparisons where both models are evaluated on the same test set, our model outperforms prior results by 8.7 times (74.16% vs. 8.49%).
5.3 Comparison of Learned Feature Representation
We reasoned that while the classifiers of the ASL Citizen and WLASL-2000 models may predict different sets of signs, both models may still be learning features to recognize signs more generally. To assess this, we extracted the globally pooled feature representation before the classification layer of each model, and built a nearest-neighbor classifier using the cosine distance from ASL Citizen training set representation. We report accuracies on both the full ASL Citizen test dataset, and the reduced overlapping version (Table 4). For the full test set, our model achieves  $58.03\%$  top-1 accuracy, while the WLASL-2000 model achieves  $9.85\%$ . For the reduced overlap, our model achieves  $71.07\%$  top-1 accuracy, while the WLASL-2000 model achieves  $16.49\%$ . This suggests that our dataset enables learning a more robust representation of ASL. We similarly test the performance of I3D feature representations trained on Kinetics (an action recognition dataset) [15]: the reported results are close to random, evincing that out-of-domain models do not generalize to sign language recognition.
5.4 Impact of Dataset Size
Finally, to understand how scaling data collection affects model performance, we systematically downsampled the training dataset. We generated three random splits:  $25\%$  (10,924 videos),  $50\%$  (20,077 videos),  $75\%$  (30,116 videos) of the original training dataset. We ensured that each sign was represented by at least 4 samples, and smaller splits were subsets of larger splits but otherwise sampled at random. We trained an appearance-based I3D model on each of these splits and report results in Table 5, and also experimented with a 0-shot model with no training data by testing performance on
an I3D model with randomly initialized weights. Our results confirm that the scale of our training set is critical to performance; as training set size decreases, so does accuracy on our test set.
In this work, we introduce a problem formulation for ISLR in the form of dictionary retrieval, provide the largest crowdsourced ISLR dataset to date, and release metrics and baselines showing that our new dataset achieves competitive performance for ISLR in ASL.
We caution dataset users against using ASL Citizen to solve sign language translation. Unlike ISLR, continuous sign recognition requires a system to handle co-articulation, fingerspelling, facial expressions, depictions, and classifiers constructions. Translation requires not only continuous sign recognition, but also the ability to move between ASL and English grammar. Researchers without domain expertise may assume that tokenizing a video into a sequence of signs and applying ISLR to these tokens is sufficient for translation. Underestimating sign language complexity in translation work has led to objections from Deaf communities [29, 39, 21, 23]. The dataset presented here enables technologies like dictionary search, that are focused on classifying signs and do not require considering syntax or even optimal English translation. For example, beyond dictionaries, ISLR can be used to create ASL-first user interfaces that enable a user to interact with signed commands or short responses. We discourage researchers from using this dataset alone (e.g., without also learning from continuous datasets) for more complex applications.
Our data collection process reflects the dictionary lookup problem formulation and yields a large-scale, high-quality ISLR dataset. Traditional lab collections offer high-quality data but limit diversity, and scraped data may promise diversity while introducing labelling and fluency errors and compromising consent. Our crowdsourcing method overcomes such limitations. First, our web platform allows users to contribute from everyday spaces, capturing real-world diversity and use of signing space representative of dictionary applications. Second, by prompting contributors with specific signs to demonstrate, we can generate automatic labels with high confidence. Third, recruiting fluent signers from trusted groups helps ensure quality and capture of ASL conventions. Our study informs future data collection efforts: participatory approaches with meaningful contribution from Deaf researchers can yield not just larger datasets, but higher-quality data.
6.1 Limitations and Future Work
While our benchmark models achieve high levels of ISLR performance on unseen signers, future work is still required to fully solve the real-life dictionary retrieval problem. In particular, real-life dictionaries present many use cases not fully addressed in this work.
First, our supervised models operate on a fixed vocabulary, and our benchmarks are unable to cope with signs outside of this vocabulary. Since sign languages are dynamic, with new signs emerging regularly, future work should consider methods that can adapt to signs that were unseen at training time. Second, we evaluated our models on fluent Deaf signers who can expertly replicate signs. Novice signers would likely have difficulty recalling and executing a sign. Therefore, we expect a performance gap for these users, which future models could address. As the vocabulary of ASL is in the magnitude of thousands, further data collection efforts can help create dictionaries more representative of the full vocabulary. While we capture a variety of users and environment in our videos, future work also includes collecting a diversity of signing styles and capturing variations that stem from geographical and learning background to support all users.
Future work also includes deepening evaluation. While we considered DCG, MRR, and recall-at-K in this work, these metrics may not fully align with user preferences. Metrics that better capture overall list relevance (e.g., by weighting relevance of signs with similar meanings or visual appearance high in the list) may reflect a better user experience [30]. Deployed dictionaries also must meet performance measures beyond accuracy (like ease of use or speed). Finally, although we report overall performance, our I3D model accuracy ranges substantially (43-75%) across users. Real-world applications are becoming increasingly viable, and future work should explore whether ISLR models are equitable, and how performance disparities between demographic groups might be addressed.
First and foremost, we thank the community members who participated in this research project by recording themselves signing. We also thank Mary Bellard, Miriam Goldberg, Hannah Goldblatt, Paul Oka, Philip Rosenfield, Bill Thies, and the Microsoft Research outreach team for thoughtful discussions, support, and contributions to the crowdsourcing platform.
This work was supported in part by the National Science Foundation Grants: BCS-1625954 and BCS-1918556 to Karen Emmorey and Zed Sehr, BCS-1918252 and BCS-1625793 to Naomi Caselli, and BCS-1625761 and BCS-1918261 to Ariel Cohen-Goldberg. Additional funding was from the National Institutes of Health National Institute on Deafness and Other Communication Disorders of and Office of Behavioral and Social Science Research under Award Number 1R01DC018279.
[54] Christian Vogler and Dimitris Metaxas. Adapting hidden markov models for asl recognition by using three-dimensional computer vision methods. In 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation, volume 1, pages 156-161. IEEE, 1997.
[55] Ankita Wadhawan and Parteek Kumar. Sign language recognition systems: A decade systematic literature review. Archives of Computational Methods in Engineering, 28(3):785-813, 2021.
[56] WFD. Legal recognition of national sign languages., 2022. https://wfdeaf.org/news/the-legal-recognition-of-national-sign-languages/.
[57] WFD. World federation of the deaf, 2022. URL http://wfdeaf.org/our-work.
[58]Ronnie Wilbur and Avinash C Kak.Purdue rvl-sll american sign language database. 2006.
[59] Ronnie B Wilbur. Phonological and prosodic layering of nonmanuals in american sign language. In *The signs of language revisited*, pages 196–220. Psychology Press, 2013.
[60] Chenchen Xu, Dongxu Li, Hongdong Li, Hanna Suominen, and Ben Swift. Automatic gloss dictionary for sign language learners. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 83–92, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-demo.8. URL https://aclanthology.org/2022.acl-demo.8.
[61] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Thirty-second AAAI conference on artificial intelligence, 2018.
[62] Kayo Yin, Amit Moryossef, Julie Hochgesang, Yoav Goldberg, and Malihe Alikhani. Including signed languages in natural language processing. arXiv preprint arXiv:2105.05222, 2021.
[63] Morteza Zahedi, Daniel Keysers, Thomas Deselaers, and Hermann Ney. Combination of tangent distance and an image distortion model for appearance-based sign language recognition. In Joint Pattern Recognition Symposium, pages 401-408. Springer, 2005.
[64] Ronglai Zuo, Fangyun Wei, and Brian Mak. Natural language-assisted sign language recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14890-14900, 2023.
[65] Inge Zwitsserlood. Classifiers. In Sign language: An international handbook. De Gruyter, 2012.
A Platform modifications
Changes to the platform introduced in prior work [8], with the goal of helping to meet our data collection needs are described below.
Platform change	Intended impact on data collection
Improved recording task flow (automatic count-down to recording start, video upload in the back-ground)	Scale: a more efficient contribution process enables participants to contribute more videos in a set amount of time
Visual design overhaul (updated color scheme, button design, and page layouts; replaced one-off video players with standard video players)	Scale: a better user experience attracts and encourages use, and increases trust in the platform creators, which lowers barriers to contributing
Updated platform structure and navigation (updated landing page with direct links to contribute; updated navigation links in header; updated info page with more project details in ASL and English)	Scale: a reduced learning curve lets participants contribute more data, and increased trust lowers barriers to contributing
Infrastructure scaling (set up infrastructure to handle many parallel contributions, large data storage, backup scripts)	Scale: provides technical capabilities to collect data at scale
Removed participant ability to share content with one another in real-time (instead creating two-phased implementation of collection followed by release)	Scale: removes potential for viewing off-putting content from other users
New set of seed sign videos executed by a well-known fluent signer who is not white-presenting	Participant diversity: representation creates a more welcoming environment and fosters contributions from a wider range of participants
Improved personal data view (enabling users to search through their own videos, sorting videos and demographics into tabs)	Data quality: participants can more easily review and update contributions
Overlaid the outline of a human figure on the web-cam feed	Data quality: videos are more likely to capture the entire upper body and are more standardized across participants
Table 6: Summary of platform feature changes, alongside potential impacts on the contributor and resulting dataset.
B Dictionary Retrieval Metrics
For a given query, if  $i$  is the placement of the desired gloss in the returned list of glosses, we calculate metrics using the following formulae:
- Discounted Cumulative Gain  $= \frac{1}{\log_2(i + 1)}$ , ranges in  $[\epsilon, 1]$  with 1 indicating that the correct gloss is always the top ranked item, and  $\epsilon = \frac{1}{\log_2(N + 1)}$  is the smallest attainable score when the correct gloss is ranked last (in our case,  $N = 2729$  so  $\epsilon = 0.088$ ). A completely random ordering with give an average DCG of approximately 0.15.
- Mean Reciprocal Rank  $= \frac{1}{i}$ , ranges in  $[\epsilon, 1]$  with 1 indicating that the correct class is always the top ranked item, and  $\epsilon = \frac{1}{N} = 0.00037$ . A completely random ordering will give an average MRR of approximately 0.0032.
The overall scores reported are averages of these metrics across respective data splits (e.g., average over all test instances).