SlowGuess/NeurIPS_2023_OCR huggingface dataset:
On the other hand, prompting with the English gloss has the benefit of collecting more variations of each sign, which could be valuable in making a more general-purpose sign recognition system (though labeling all of these unintended signs is future work). In this dataset, we observed five classes with significant variants: Compound signs such as BEDROOM, BACKYARD, and REFRIGERATOR may vary from the reference example by the ordering of the parts or the formation of one or both of the constituent parts. Lexicalization results in the parsing of the first letter of a sign's closest gloss and using the manual alphabet as the handshape of the sign. For example, a signer might use a R-hand shape for RED. Regional and cultural variations are commonplace. The sign for GARBAGE in this dataset would be understood as CABBAGE in many areas of the United States. PIZZA has a surprising number of regional variations. Iconic signs in the vocabulary often had variations. For example, TOOTHBRUSH may be formed by using the index finger as the toothbrush to mimic the back and forth motion. Alternatively, TOOTHBRUSH may be signed by mimicking holding a toothbrush in the palm. VACUUM may be formed by imitating the motion made while vacuuming or by using the hand to mimic the internal fan of the device. Contextual understanding of the English gloss also caused variations. BEFORE in time is signed differently than when something comes sequentially BEFORE something else. Other examples include PRETEND, SAME, and CLOSE.
We assign 31 signers to the training set, eight to the validation set, and eight to the test set. The eight chosen for the validation set capture the broadest possible demographic and linguistic range in order to make validation results as representative of real-world performance as possible. Of the eight users, four were noted for exhibiting high variance (differed from DPAN's example sign), while one was noted for exhibiting middle levels of variance. In addition to variability, we classified signers according to the Fitzpatrick scale, a numerical classification of human skin color. Of the validation signers, five had skin phenotype I, two had skin phenotype II, and two had skin phenotype V (a lower-numbered phenotype represents fairer skin). Finally, the gender of the validation users is evenly split between male and female. The test set has demographics proportional to the training set.
5 Experiments on PopSign ASL v1.0
Georgia Tech's PACE computing cluster [PACE, 2017] was used to extract features with MediaPipe Hands with a detection confidence of 0.5 and a tracking confidence of 0.1. MediaPipe Hands extracts 21 key points, each containing an x, y and z coordinate [Lugaresi et al., 2019]. Preprocessing includes removing NaNs for each key point. All videos where a left hand was used to sign were flipped along the vertical axis to convert them into right-handed signs. The PopSign game recognizes when a player is signing with the left hand and also flips the axis. Since there are much fewer left hand dominant signers, this technique can improve recognition rates for playing with either hand. All videos with zero frames of tracking were removed. To deal with variable sequence length, videos with less than 60 frames were padded with a row with values set to -1 at the end of the sample. For videos with more than 60 frames, the middle 60 frames were kept while the rest were discarded. Each of the labels was converted into a one-hot encoding.
Confused Pair	% Times Confused (% of Dataset)	Confused Pair	% Times confused (% of Dataset)
WAKE, AWAKE	47.0 (2.7)	PRETEND, HELLO	15.4 (0.8)
GLASSWINDOW, TOOTH	25.1 (1.2)	NAPKIN, FLOWER	26.0 (0.8)
EYE, TOOTH	19.8 (1.0)	INTO, ELEPHANT	11.6 (0.6)
CHIN, SAY	19.9 (1.0)	FINGER, WAIT	11.4 (0.6)
BESIDE, PERSON	18.8 (0.8)	THAT, STAY	10.8 (0.4)
Table 2: Most confused signs
Using Keras  $^{6}$ , the model is built using two bidirectional LSTM layers followed by a dropout layer and the output layer of 250 units with a softmax activation. The first bidirectional LSTM layer has 128 units and has return sequences set to True to allow timestep information to be preserved and passed to the second bidirectional LSTM layer. Default Keras LSTM parameter values are used. The input shape is manually set to (60, 63). The second bidirectional LSTM has 128 units and the default parameter values required for the LSTM layer in Keras remain. The third layer is a dropout layer with rate set to 0.5. The last layer is the 250 unit softmax output layer. During model training, a batch size of 32 is used along with the Adam optimizer and Categorical Cross Entropy Loss. The model is set to train for 40 epochs, and the validation data is used for early callback stopping if the validation loss converges. The weights that results in the lowest validation loss are restored. The model is trained in 30 minutes on one PACE GPU node (4x NVIDIA RTX 6000).
The recognition system achieved  $82.1\%$  accuracy (81.9% F1 score) on the verification set and  $84.2\%$  accuracy (83.9% F1 score) on the test set for all files that had at least one frame of MediaPipe tracking. When including the files where tracking failed completely (109/29,949 validation; 171/33,667 test), accuracy decreases to  $81.9\%$  validation and  $83.8\%$  test set, respectively. Upon inspection, the test set has some signers who are highly consistent with their signing, which explains the surprising increase in accuracy between verification and test. The top most confused pair of signs can be seen in Table 2. Many are near homophones mentioned earlier. For example, the video for AWAKE is often labeled WAKE, and vice versa. EYE versus TOOTH confusion is the result of Mediapipe Hands's features not being measured relative to the face. Adding Mediapipe Pose or switching to Mediapipe Holistic, if we can make it run efficiently enough with Unity for PopSign, would significantly improve results.
6 Discussion and Future work
With 200,686 total videos, PopSign ASL v1.0 is the largest isolated sign language dataset publicly available. Besides residing at signdata.cc.gatech.edu, we expect to host the 1.1TB PopSign ASL v1.0 dataset at RIT/NTID, UIUC, and DPAN. Unlike previous datasets, PopSign focuses on one-handed signing captured from smartphone selfie cameras. The face and signing hand represent a significant amount of the area of the image. Each video is reviewed manually, with a large majority being labeled as the expected sign. The videos show a large variety of lighting conditions and backgrounds.
The PopSign game only considers five signs at a time. Thus, sets of signs can be chosen to minimize potential confusion by the recognition system. Seven sets of signs were chosen to create PopSignAI Preview, an initial version of the PopSign game with embedded sign recognition that can be found on the Android Play and Apple App stores<sup>7</sup>. For this proof-of-concept a 2D-CNN was trained for each set of five signs and limited to approximately 1MB in size in order to run efficiently on older smartphones such as the Pixel 4A. In informal testing, this limited task increased recognition accuracy to approximately  $95\%$  when played by an experienced signer. Gameplay is quite compelling, suggesting that the PopSign ASL v1.0 dataset is suitable for creating sign language recognition based games. Of course, a 250-sign recognition system will allow many more levels of the PopSign game to be created. As the recognition system matures, it will be integrated as a plug-in for game creation engines such as Unity to enable more programmers to create games based on sign language.
$^{6}$ https://github.com/keras-team/keras
The app is available at https://play.google.com/store/apps/details?id=edu.gatech.popsignai
The lexicon was chosen with a particular application in mind: teaching hearing parents of deaf infants basic sign vocabulary. However, there are approximately 313 concepts in the MacArthur-Bates CDI not covered in the current dataset. A similar collection effort is underway to address these signs, which should expand the usefulness of the data. In speech recognition, small vocabulary isolated recognizers are used for voice dialing ("call" "father"), interactive phone menus ("press one or say 'balance' to get account balances"), multimodal control of windowing systems ("close" while pointing to a window), searching voice mail by voice, and controlling wearable computers ("ok Glass" "show" "compass"). We hope this dataset can support similarly creative use cases for ASL.
Since collection is performed remotely by mailing smartphones to participants, researchers can use our tools and procedures to purposely sample different variants of signing. For example, one can ask participants to intentionally provide different versions of the sign based on their background. Signers who were raised with Black ASL [Valli and Lucas, 2000] or with a local sign language like Hawaiian Sign [Shroyer and Shroyer, 1984] may have used a different set of signs at home than at university. Such a collection allows sign linguists to gain a better understanding of regional signs and enables a general purpose sign recognizer to be more inclusive.
7 Ethics and Safety Discussion
"Nothing about us without us" is an often-heard phrase when discussing accessibility with those in the community. Without guidance, such as can be found in the FATE paper Bragg et al. [2021], the many subtleties of sign collection can be confusing. Unfortunately, most machine learning researchers do not know about the existence of the Deaf culture, the fact that ASL has a different grammar than English, or even that there are many distinct sign languages. Here we follow the convention that "deaf" refers to the medical condition, whereas "Deaf" refers to the community, which focuses on sign language. Without Deaf members on the team, researchers can stray into creating technologies that are not useful or are even considered harmful by the community [Erard, 2017, Hill, 2020]. Deaf team members and the Deaf community have informed PopSign at every step. The idea that an educational game might be the first useful application of sign language recognition was posited in 2000 by Dr. Harley Hamilton, a sign linguist and a technology coordinator at the Atlanta Area School for the Deaf. After several iterations of desktop games tested at Deaf schools in Georgia, a student researcher suggested changing the focus to smartphones and introduced the team to the idea that ASL is evolving due to smartphone technology [Morris, 2022a]. PopSign provides the ML community with a clear need and a viable dataset for a goal that can be achieved by sign recognition in the near term. Signers for the dataset were recruited by Deaf researchers and informed before beginning participation about the intent to create a public video database. DPAN's consent procedures were informed by those used by Georgia Tech's IRB. The risks of the dataset videos are similar to any identifiable video made available on the Internet. While names are not affiliated with any of the videos, it was clear in the consent process that participants' faces would be identifiable. Participants were paid $300 for providing 5000 signs, a process that would take approximately six hours. Signers could stop participation without penalty; compensation is prorated. PopSign ASL v1.0 is provided under a Creative Commons CC-BY 4.0 license, specifically so both academics and industry may use it. Software created as part of the collection and review process is covered by the standard MIT license. The dataset is not meant to be representative of ASL, though there is a risk users might use it that way. The next section warns about the dataset's limitations.
While ASL is the most common sign language used in the United States, there are many sign languages both in the United States (e.g., Plains Indian Sign Language and Hawaiian Sign Language) and in the world at large (e.g., British Sign Language, French Sign Language, German Sign Language, Hindi Sign Language, etc.). In addition, there are many regional and cultural accents associated with sign in the United States, including Black American Sign Language. The PopSign dataset is designed for teaching one variant of a signed concept, focusing on one-handed smartphone signing, but it does not capture a representative sample of all the sign variations that would be commonly understood in conversation. ASL has a grammar that is very different from English, and isolated signs do not reflect its richness. A larger number of signers is necessary to better represent skin tones, hand features, ages and different levels of signing fluency. PopSign assumes one-handed sign from
the smartphone viewpoint and field of view, whereas a more general recognition system would also require two-handed signs and a greater variety of viewpoints and camera parameters. As collection required the use of a smartphone, communication via email, and a viable shipping address, it biases the dataset towards younger, more affluent signers with facility in English.
The Popsign ASL v1.0 dataset provides a large, reviewed dataset of signs with many examples per sign that demonstrate the concepts first taught to infants. It focuses on signing captured by the selfie camera on a smartphone, and the data has already proven viable in creating a sign language recognition based educational game for helping hearing parents of deaf children learn ASL.
We would like to thank all of the signers who contributed to the project and especially Nathan Qualls and Michaela Jitaru at DPAN for their hard work in data collection. Max Shengelia at RIT helped guide the project toward smartphone signing, based on his observations in the community, and performed initial reviews of the sign. Graham MacKenzie provided perspective as an ASL interpreter on how one-handed smartphone signing is becoming commonplace and provided an initial analysis of which concepts in the CDI could be signed with one hand. RIT/NTID CAT lab is our partner in creating the next PopSign games. Finally, thanks to the Kaggle team (Mark Sherwood, Ashley Chow, and Phil Culliton) for all their advice on how to best create useful datasets. This research was supported in part through research cyberinfrastructure resources and services provided by the Partnership for an Advanced Computing Environment (PACE) at the Georgia Institute of Technology, Atlanta, Georgia, USA.
DPAN is a non-profit, and its funding for this project came through non-restrictive gifts by private and public entities. Georgia Tech's contributions are through classwork and volunteer efforts. Google-affiliated personnel play an advisory role and participate on a volunteer basis.
YouTube, May 2017. URL https://www.youtube.com/watch?v=dQM8GRjj6vY.
D. Anderson and J. Reilly. The macarthur communicative development inventory: normative data for american sign language. Journal of deaf studies and deaf education, 7(2):83-106, 2002.
V. Athitsos, C. Neidle, S. Sclaroff, J. Nash, A. Stefan, Q. Yuan, and A. Thangali. The American Sign Language Lexicon Video Dataset. In 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pages 1-8, 2008. doi: 10.1109/CVPRW.2008.4563181.
L. Berger, J. Pyers, A. Lieberman, and N. Caselli. Parent american sign language skills correlate with child-but not toddler-asl vocabulary size. Language Acquisition, pages 1-15, 2023.
D. Bragg, N. Caselli, J. A. Hochgesang, M. Huenerfauth, L. Katz-Hernandez, O. Koller, R. Kushal-nagar, C. Vogler, and R. E. Ladner. The FATE Landscape of Sign Language AI Datasets: An Interdisciplinary Perspective. ACM Trans. Access. Comput., 14(2), July 2021. ISSN 1936-7228. doi: 10.1145/3436996. URL https://doi.org/10.1145/3436996. Place: New York, NY, USA Publisher: Association for Computing Machinery.
N. C. Camgoz, A. A. Kindiroglu, S. Karabuklu, M. Kelepir, A. S. Ozsoy, and L. Akarun. Bosphorus-Sign: A Turkish Sign Language Recognition Corpus in Health and Finance Domains. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 1383-1388, Porto-rož, Slovenia, May 2016. European Language Resources Association (ELRA). URL https://aclanthology.org/L16-1220.
N. K. Caselli, A. M. Lieberman, and J. E. Pyers. The asl-cdi 2.0: An updated, normed adaptation of the macarthur bates communicative development inventory for american sign language. Behavior Research Methods, 52:2071-2084, 2020.