SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xin Shen $^{1}$  Shaozu Yuan $^{2}$  Hongwei Sheng $^{1}$  Heming Du $^{1}$  Xin Yu $^{1*}$ $^{1}$ The University of Queensland  
 $^{2}$ JD AI, Beijing, China  
x.shen3@uqconnect.edu.au
Sally & Possum  
Auslan Daily Communication
Figure 1: Diversity of our curated Auslan-Daily dataset. Auslan-Daily involves various topics, signers and diverse environments. In particular, there are multiple persons in a scene and signers appear in different areas of the scene.
ABC News with Auslan  
Auslan Daily News
Sign language translation (SLT) aims to convert a continuous sign language video clip into a spoken language. Considering different geographic regions generally have their own native sign languages, it is valuable to establish corresponding SLT datasets to support related communication and research. Auslan, as a sign language specific to Australia, still lacks a dedicated large-scale dataset for SLT. To fill this gap, we curate an Australian Sign Language translation dataset, dubbed Auslan-Daily, which is collected from the Auslan educational TV series and Auslan TV programs. The former involves daily communications among multiple signers in the wild, while the latter comprises sign language videos for up-to-date news, weather forecasts, and documentaries. In particular, Auslan-Daily has two main features: (1) the topics are diverse and signed by multiple signers, and (2) the scenes in our dataset are more complex, e.g., captured in various environments, gesture interference during multi-signers' interactions and various camera positions. With a collection of more than 45 hours of high-quality Auslan video materials, we invite Auslan experts to align different fine-grained visual and language pairs, including video  $\leftrightarrow$  fingerspelling, video  $\leftrightarrow$  gloss, and video  $\leftrightarrow$  sentence. As a result, Auslan-Daily contains multi-grained annotations that can be utilized to accomplish various fundamental sign language tasks, such as signer detection, sign spotting, fingerspelling detection, isolated sign language recognition, sign language translation and alignment. Moreover, we benchmark results with state-of-the-art models for each task in Auslan-Daily. Experiments indicate that Auslan-Daily is a highly challenging SLT dataset, and we hope this dataset will contribute to the development of Auslan and the advancement of sign languages worldwide in a broader context. All datasets and benchmarks are available at  $\Theta$  Auslan-Daily.
(a.) ASL  $\leftrightarrow$  American English: Hello, my names Rebecca
(b.)  $BSL \leftrightarrow$  British English: Every spring, our planet is transformed
(c)  $CSL \leftrightarrow \text{Chinese}$ : 晚上太安静了。
(d.)  $DGS \leftrightarrow$  Germany: Sonne frisch bis zehn grad suedost auch
(e.) Auslan  $\leftrightarrow$  Australian English: Look Possum, you can see there ...
Auslan-Daily Communication
Figure 2: Comparisons of large-scale sign language translation datasets across diverse geographic regions. (a) How2Sign [1] (American), (b) BOBSL [2] (British), (c) CSL-Daily [3] (Chinese), (d) RWTH-PHOENIX-Weather 2014T [4] (Germany), (e) Our proposed Auslan-Daily (the yellow and black bounding-boxes indicate the signer and non-signer in the sign video clip).
Auslan-Daily News
Sign language (SL) is the primary way for deaf or people with hearing loss to express themselves. Similar to various spoken languages, sign languages have their vocabularies and grammar [5, 6, 7]. More importantly, diverse geographic regions usually have their native sign languages even though these regions share a commonly spoken language, such as America, Australia and the UK. To eliminate the communication barriers between the deaf and hearing communities, sign language translation (SLT) has been proposed to convert signs into spoken languages [1, 2, 4, 3].
With the emerging deep learning techniques and large-scale sign language datasets, SLT has achieved promising progress recently. As shown in Figure 2, researchers from various countries have constructed their sign language datasets and thus thrust SLT in their respective sign languages, such as American sign language (ASL) [1], British sign language (BSL) [2], Chinese sign language (CSL) [3] and Germany sign language (DGS) [4]. However, to the best of our knowledge, there is no publicly available large-scale Auslan dataset for continuous sign translation<sup>2</sup>. Meanwhile, according to the Hearing Care Industry Association<sup>3</sup>, as of June 2015, one in six Australians had hearing loss affecting them and this proportion is expected to increase to one in four by 2050. Due to the societal inclusion and the regional nature of sign languages, Australian sign language (Auslan) datasets are inevitably and urgently needed in order to investigate automatic translation.
Moreover, existing sign language corpora either are captured in controlled laboratory environments [1, 3, 8, 9, 10, 11] or only contain a single person (the signer) at a certain position in each video clip [2, 4, 12, 13]. In [14], Nunez-Marcos et al. point out that the effectiveness of sign language translation can be compromised in multi-individual situations due to the presence of non-signers. Additionally, Yin et al. [15] emphasise that translating sign language in the wild is even more challenging. Thus, the existing datasets lack sufficient diversity and may not fully reflect the complexity in the wild.
In this work, we aim to construct a large-scale Auslan translation dataset in the wild which contains sufficient high-quality Auslan videos and their English transcriptions. We adopt the first Australian educational TV series for deaf children "Sally and Possum", "ABC News with Auslan" and several online domain-specific Auslan corpora as our data source. Firstly, "Sally and Possum" is aired to help deaf children and their parents to learn Auslan, and it covers various topics of daily life, such as drawing pictures with colourful pigments and asking for advice in diverse environments, including indoor and outdoor scenes (Figure 1, left). Secondly, "ABC News with Auslan" starts broadcasting form 2022 and provides the latest news and information from ABC News every week with Auslan interpreted (Figure 1, middle). Lastly, we aggregate additional content-specific Auslan data from
Sign videos are segmented and annotated at the sentence level.
<sup>3</sup>https://www.hcia.com.au/resources/HCIA.pdf
<sup>4</sup>https://earlychildhood.qld.gov.au/early-years/sally-possum
<sup>5</sup>https://iview.abc.net.au/show/abc-news-with-auslan
$^{6}$ https://www.youtube.com/user/InfoVicdeaf
Table 1: Comparison between Auslan-Daily and existing SLT datasets. PPC.: persons per clip.
Dataset	SL	Video	Vocab.	#Signer	Source	Background	PPC.	Signer Position	Expert Check
PHOENIX-2014T[4]	DGS	8K	3K	9	TV	Fixed	1	Fixed	ALL
SIGNUM[10]	DGS	33K	450	25	Lab	Fixed	1	Fixed	ALL
Content4All[12]	DSGS+VGT	15k	-	-	TV	Fixed	1	Fixed	ALL
KETI[9]	KSL	15K	0.5K	14	Lab	Fixed	1	Fixed	ALL
CSL-Daily[3]	CSL	21K	2K	10	Lab	Fixed	1	Fixed	ALL
How2Sign[1]	ASL	35K	16K	11	Lab	Fixed	1	Fixed	ALL
DGS Corpus[8]	DGS	63K	23k	327	Lab	Fixed	1	Fixed	ALL
OpenASL[13]	ASL	99k	33k	~200	Web	Diverse	1	Fixed	PARTIAL
BOBSL[2]	BSL	1.2M	7.8k	39	TV	Diverse	1	Fixed	PARTIAL
Auslan Corpus[19]	Auslan	-	-	300	Lab	Fixed	1	Fixed	ALL
Auslan-Daily(ours)	Auslan	25K	14K	67	TV&Web	Diverse	1-10	Diverse	ALL
online Auslan corpora, including natural disaster reports and interviews (Figure 1, right). We collect these high-quality video materials, which capture various environments and include diverse daily life topics, namely Auslan-Daily.
To better translate sign videos in Auslan-Daily, we devise a two-stage data annotation labelling process, i.e., (1) aligning video clips and transcriptions and (2) detecting the signer in each aligned video clip. The rationale behind these two labelling stages is as follows: in (1), despite the original whole sign videos accompanying English subtitles, misalignments often exist between each sign video clip and its corresponding complete subtitle; In (2), explicitly labelling the signer's position helps simplify multi-person scenes and reduce translation complexity. In the first stage, we invite five Auslan experts to perform multi-gained annotations, including the temporal boundaries of fingerspellings, long-tailed isolated glosses, and sentences. In the second stage, we apply Alphapose [16, 17, 18], a tool for keypoints estimation and person tracking, to obtain each person's unique ID and pose sequences in each sign video clip. Then, for each sign clip, we manually modify the error of the Alphapose results and labelled the signer. After 300 working hours, we complete the fine-grained annotation labelling processing for our collected more than 45 hours of high-quality Auslan videos. Finally, in Auslan-Daily, there are around 2k video  $\leftrightarrow$  fingerspelling, 3k video  $\leftrightarrow$  gloss, and 25k video  $\leftrightarrow$  sentence pairs $^{7}$ .
Based on these multi-grained annotations, we are able to investigate various sign language-related tasks in multi-person scenarios, including sign language translation, signer detection, fingerspelling detection, sign spotting, isolated sign language recognition and alignment. Specifically, we apply publicly available state-of-the-art models for each task and then report the performance with corresponding evaluation metrics. Experimental results demonstrate the challenges of Auslan-Daily due to its rich diversity and complexity. Overall, the contributions of this work are threefold:
- We construct the first large-scale Auslan dataset, dubbed Auslan-Daily, which contains multi-person sign videos on various topics in diverse environments.
- Auslan-Daily provides multi-grained annotations, enabling researchers to investigate a variety of tasks, including signer detection, fingerspelling detection, sign spotting, isolated sign language recognition, sign language alignment, and sign language translation.
- We establish a leaderboard and an evaluation benchmark to promote Auslan SLT research.
2.1 Sign Language Translation Datasets
Since current models for sign language translation are significantly data-driven and based on deep learning, assembling extensive sign language datasets becomes a pivotal component for advancing sign language translation across various nations. Datasets constructed for SLT in recent years are shown in Table 1, where DGS, KSL, CSL, ASL, BSL, and Auslan represent German, Korean, Chinese, American, British, and Australian Sign Languages, respectively. Due to the limited lexical variety and sentence complexity within the SIGNUM [10] and KETI [9] datasets, they are unsuitable for SLT tasks. PHOENIX-2014T [4] is the first sign translation dataset used to translate German Sign Language (DGS). However, since this dataset only consists of weather forecast videos, the topic coverage is limited and may need to be increased for daily communication.
7All the gloss and fingerspelling video clips can be found in sign sentence video clips. While fingerspelling and gloss videos have some overlaps, fingerspelling videos are not the subset of gloss annotations.
BOBSL [2, 20] and OpenASL [13] are large-scale sign language translation datasets. For these two extensive datasets, experts verification has been confined solely to the validation and test sets, whereas the training set has been annotated via a pre-trained BSL sign alignment model [21] or through "self-generated" time boundaries based on ASL News. How2Sign [1] is the largest American Sign Language (ASL) dataset captured in the lab environment. It contains a variety of annotations, including multi-view information, depth information, pose, and speech. CSL-Daily [3] and DGS Corpus [8] are the extensive datasets of Chinese Sign Language (CSL) and German Sign Language (DGS), respectively. How2Sign, CSL-Daily and DGS Corpus cover diverse topics. The datasets pertinent to Auslan are represented by Auslan Signbank [22] and Auslan Corpus [23, 19, 24]. The Auslan Signbank constitutes a dictionary with approximately 5,500 Auslan glosses, while the Auslan Corpus [23, 19, 24] is primarily confined to exploratory research by linguists and is not entirely accessible to the public. More importantly, the above datasets only consider one person (signer) in each sign video clip, and the signer appears at a controlled certain position in each sign video clip. However, in the real world, multiple people may perform sign gestures in a scene, and their positions are diverse. Detecting the actual signer and accurately translating their gestural signs within complex environments presents a novel challenge. Factors such as perspective, illumination and the presence of crowds introduce noise and complexity when translating sign language to spoken language. In contrast, we propose Auslan-Daily, the first publicly available real-world Auslan translation dataset. Meanwhile, we enrich the diversity and complexity by collecting source various data.
2.2 Tasks Associated with Sign Language
Currently, there are several tasks for investigating various granularities of sign language, such as fingerspelling (character), gloss (word) and sentence. All the sign language-related tasks aim to foster better sign language understanding and sign language translation development. The sign language-related task, including: (i) Sign Language Translation (SLT) [4] task is to translate a sign video clip to the corresponding spoken language. The current SLT models can be divided into three categories, Sign2Gloss2Text, Sign2(Gloss+Text), and Sign2Text [25]. The Sign2Gloss2Text [26] model is a two-stage method, Sign2Gloss and Gloss2Text, respectively. The first stage is sign language recognition, which predicts a gloss sequence from a video, and the second stage translates the predicted gloss into the target natural language. Both Sign2(Gloss+Text) [27, 3, 28, 29, 30] and Sign2Text [31, 32, 33, 34] are end-to-end models. The difference is that Sign2(Gloss+Text) jointly trains the sign language recognition and translation tasks, and uses the gloss information as auxiliary supervision to extract features from videos, thereby improving the translation results. Though Sign2(Gloss+Text) models perform well on existing datasets, obtaining large-scale sign language translation data with continuous glosses annotation is costly and time-consuming. The primary reason is that the process of annotating one hour of continuous sign language video [21] requires an expert, who is proficient in sign language, approximately ten to fifteen hours. Thus, it is hard to apply them to SL datasets that do not contain gloss annotations. For Sign2Text models, they aim to directly convert sign language performed by a single person into target natural language. Furthermore, as the existing datasets are not collected in the wild, previous models might fail to tackle complex scenarios and real-world situations with multiple people; (ii) Sign Language Alignment [21, 35] temporally aligns asynchronous subtitles in sign language videos. A proficient alignment model for sign language can mine more sign data for automated translation; (iii) Isolated Sign Language Recognition [36, 37, 20] focuses on identifying and understanding individual gestural signs, independent of any surrounding context or sequence of signs; (iv) Sign Spotting [38, 39, 40] aims to find accurate locations of the given isolated signs in continuous co-articulated sign language videos; (v) Fingerspelling Detection [41, 42, 43] finds the fingerspelling segments' intervals within the clip. Fingerspelling is an important component of Sign Language, in which words are signed letter by letter and (vi) Active Signer Detection, also known as Sign Language Detection [44, 45], is identical to the initial stage of Signer Diarisation [46]. It aims to find the signer in the sign video clip.
3 Auslan-Daily Dataset
In this section, we describe data collection and cleaning, detail the data labelling procedure and provide statistics of the Auslan-Daily train/test split.
Table 2: Key statistics of Auslan-Daily. Auslan-Daily Communication and Auslan-Daily News are two sub-datasets split from Auslan-Daily. OOV: out-of-vocabulary. Singleton: words that only occur once in the training dataset.
Sub-Dataset	Auslan-Daily Communication	Auslan-Daily News
Domain/Topic Video Resolution@FPS	Communication 1920×1080@25	News & Documentary 1280×720/1920×1080@29.97
Split	Train	Dev	Test	Train	Dev	Test	Total
Segments	12,441	800	800	9,665	700	700	25,106
Signers	49	12	9	18	17	17	67
Frames	930,321	45,369	45,171	2,072,475	144,819	142,893	3,381,048
Vocab.	3,064	522	469	12,346	2,872	2,885	13,945
Tot. words	88,167	4,126	4,115	163,268	11,376	11,530	282,582
Tot. OOVs	-	8	10	-	326	304	-
Singletons	1,043	-	-	5,267	-	-	-
Person per clip	1-11	1-8	1-8	1-8	1-8	1-7	1-10
3.1 Data Collection and Cleaning
"Sally and Possum", "ABC News with Auslan" and Auslan corpora from YouTube are public TV programs and open sources $^{8}$ . "Sally and Possum", a premium Australian sign language show for deaf children. To facilitate children learning of Auslan, the production team designs the plots and writes the English scripts. Subsequently, Auslan experts perform sign language based on the script. This show has 6 seasons and 15 episodes per season, which is 22.5 hours long and contains around 20 topics, including daily communication, study skills, and knowledge explanation. Beginning in 2022, "ABC News with Auslan" weekly broadcasts key domestic and international events news and weather forecasts. It is an ongoing TV program, and the current dataset includes 45 news videos as of May 2023. Auslan experts execute a real-time sign language translation (simultaneous interpretation) for deaf or people with hearing loss based on currently broadcast news. However, the English subtitles may leave the following problems [21]: (i) the order of subtitles is not complying between spoken and sign languages, and (ii) the duration of a subtitle varies considerably between signing and speech due to differences in speed and grammar. To enrich the topics of our dataset, we also collect several publicly available high-quality documentaries interpreted with Auslan. These data comprise specific thematic areas, including disaster overviews, preventative measures, and interviews.
All of the original videos are recorded with standard English dubbing and subtitles. We download subtitles of each whole original video, which are arranged with the format "[Start Time] subtitle [End Time]". The time intervals are marked based on dubbing. Upon spot check, we observe that longer subtitles might extend across multiple temporal intervals, whereas several shorter subtitles tend to appear within a time interval. To procure complete sentence-level subtitles, we conduct the data cleaning operations as follows: (1) for incomplete subtitles, e.g., ending with a comma, we connect them with the following subtitles to compose complete sentences and merge their time duration; (2) for several complete subtitles that appear within a time interval, we partition them into multiple independent sentences; (3) for a complete sentence that only contains modal particles, e.g., "Oh!" and "Ha ha ha!", we remove them to avoid meaningless translation. As a result, we acquire approximately 29k complete subtitles that require alignment.
3.2 Two-Stage Data Labelling Procedure
To obtain applicable data for sign language translation, we design two stages of the data labelling procedure: (1) aligning video clips and transcriptions and (2) detecting the signer in each aligned video clip. They are imperative as original subtitles often misalign with sign videos, and the position of the signer varies. By analyzing the frequency of individual tokens across all subtitles, we notice a considerable presence of long-tailed words. Due to their sparse occurrence, these words impose challenges for the model and affect the translation results. To investigate this problem, sign language experts annotate the less commonly used glosses in "Sally and Possum". Experts also label the temporal boundaries of significant fangselling instances, as fangselling is commonly used in the deaf community. Therefore, in the first stage, we engage Auslan experts to synchronise video  $\leftrightarrow$  fangselling, video  $\leftrightarrow$  gloss, and video  $\leftrightarrow$  sentence pairs.
Next, we employ Alphapose [16, 17, 18] to track people in each aligned video clip. Then, we record trajectories and pose sequences along with their corresponding IDs. Considering Alphapose
$^{8}$ Our dataset follows the copyright Creative Commons BY-NC-ND 4.0 license ©.
Figure 3: Distributions of #frames/#words over clips (L) and the topic diversity in Auslan-Daily (R).
may suffer tracking errors sometimes<sup>9</sup>, we thus invite annotators to manually check and modify the tracking results by assigning correct IDs to the signers.
To guarantee the annotation quality of our dataset, we conduct a cross-check verification process during each data labelling procedure stage. Specifically, we ask each Auslan annotator as an examiner to cross-check around  $5\%$  of annotated video clips provided by another annotator. The video clips are chosen randomly. If the examiner finds more than  $10\%$  of annotated videos have obvious errors, a third annotator is invited to review and correct the annotations.
Through the collaborative efforts of five Auslan experts and five annotators, we complete all annotations with approximately 300 work hours. Overall, our Auslan-Daily dataset contains the following annotations: (1) temporal boundaries of sign video clips; (2) temporal boundaries of long-tailed glosses; (3) temporal boundaries of partial fingerspellings; (4) pose sequences of signers and non-signers; (5) bounding-boxes of signers; (6) signer identities and (7) English transcriptions. These multi-grained annotations can be further investigated for Australian sign language-related tasks.
3.3 Data Statistics
During the data labelling, experts discard erroneous data, such as subtitles do not have corresponding sign language. As demonstrated in Table 2, there are in total 25,106 video clips encompassing 67 unique signers, with the vocabulary size of 13,945 words. It should be noted that signers in the validation and test sets appear in the training set. As the number of persons ranges from 1 to 10 in a video clip, the distractions, such as gesture interference of multi-persons, are also involved in sign language translation, thus imposing challenges in this task. To verify the robustness of the various models, we also statistics of the sentences within the test set. As shown in Table 3 (Appendix), over  $80\%$  of video clips in the test set encompass distinct sentences. Therefore, the robustness of the models can be verified by evaluating on the test set.
After examining the domains of the three data sources, "ABC News with Auslan" and the data from online Auslan corpora are found significantly similar. Thus, we combine the data collected from these two video sources and partition the Auslan-Daily dataset into two sub-datasets: Auslan-Daily Communication and Auslan-Daily News. We randomly split the two sub-datasets data into the training, validation and test sets as shown in Table 2. Each sign video clip has 73 frames and 7.8 words on average for the Auslan-Daily Communication sub-dataset. Each sign video clip has 214 frames and 16.9 words on average for the Auslan-Daily News sub-dataset. The distributions of frames and words in different splits of the sub-datasets are shown in Figure 3 (left).
Moreover, Auslan-Daily has 3,000 (600 classes) long-tailed isolated glosses and 2,000 significant fingerspellings. For the isolated gloss, we select words appearing less than ten times after the natural language corpus post-lemmatization<sup>10</sup> in "Sally and Possum". Its main purpose is to enhance translation models to recognise long-tail words through sign language recognition or spotting tasks. Furthermore, considering the pivotal role of fingerspelling in sign language translation, we thus
The overall error rate is less than 20 percent of video clips with multiple signers.
10 Lemmatization facilitates a more fine-grained and accurate analysis and representation of text.