SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 4: Overview of Auslan-Daily tasks and provided annotations.
Annotate Auslan fingerspelling instances. As illustrated in Figure 3 (right), there are 21 types of daily communication and more than 15 types of daily news in our Auslan-Daily dataset.
4 Overview of Auslan-Daily Tasks
4.1 Task Definition
Active Signer Detection (ASD) [44, 45, 46]: Given trajectories of  $N$  people  $\mathcal{P} = \{p_i\}_1^N$  in a sign video clip  $\mathbb{V} = \{f_i\} _i^t$ , the goal of ASD is to find the active signer  $p_i$  in the current video clip.
Sign Spotting (SS) [38, 39]: Given a sign video clip  $\mathbb{V}$  with  $t$  frames and an isolated gloss  $\mathbb{I}$  clip, the goal of SS is to locate the start frame  $f_{m}$  and end frame  $f_{n}$  of  $I$  in  $\mathbb{V}$ .
Isolated Sign Language Recognition (ISLR) [36, 37, 20]: Given an isolated gloss  $\mathbb{I}$  clip, ISLR aims to determine the gloss category  $c$  of  $\mathbb{I}$ , where  $c$  is pre-defined by a Sign Dictionary.
Fingerspelling Detection (FD) [41]: Given a sign video clip  $\mathbb{V}$  with  $t$  frames, the goal of FD is to locate the start frame  $f_{m}$  and end frame  $f_{n}$  of each fingerspelling in  $\mathbb{V}$ .
Sign Language Alignment (SLA) [21]: Given a whole original video  $\mathcal{V} = \{f_i\}_1^T$  with  $T$  frames and its corresponding M subtitles  $S = \{s_i\}_1^M$ , the goal of SLA is to align each subtitle sentence  $s_i$  with the precise begin frame  $f_i$  and end frame  $f_j$ .
Sign Language Translation (SLT) [4]: Given a sign video clip  $\mathbb{V}$  with  $t$  frames, the goal of SLT is to translate  $\mathbb{V}$  to an English sentence  $s_i$ .
4.2 Evaluation Metrics
BLEU and ROUGE: BLEU [47] and ROUGE [48] scores are commonly-used evaluation metrics for Sign Language Translation [4]. BLEU-n measures the n-gram overlap between the generated text and reference text, and ROUGE-L measures the F1 score based on the longest common subsequences between the generated text and reference text.
Top-  $K$  Accuracy: Top-  $K$  classification accuracy measures the number of ground-truth labels within the top  $k$  predicted labels. For Signer Detection,  $K = 1$  is employed. For Isolated Sign Language Recognition, we adopt  $K$  values of 1, 5, and 10.
IoU: Intersection over Union (IoU) is defined as the ratio of the intersection and the union of the predicted and actual time intervals of an action within a video [49, 50]. Similar to [41], we employ AP@IOU as a metric for Fingerspelling Detection. Following [21, 39], we use F@IOU to evaluate Sign Language Alignment and Sign Spotting tasks.
Table 3: Translation results of Single/Multi-Person SLT gloss-free models on Auslan-Daily.
Auslan-Daily Communication	Auslan-Daily News
Single-Per. SLT	Input	R	B1	B2	B3	B4	R	B1	B2	B3	B4
SL-Luong [59]	Pose	37.27	30.15	16.26	11.67	9.45	20.65	19.84	7.81	4.59	2.81
SL-Luong [59]	RGB	13.49	13.54	7.85	5.74	4.66	16.14	16.92	7.44	4.07	2.68
SL-Transf [27]	Pose	35.65	31.31	16.17	11.41	9.20	20.25	21.25	6.57	3.32	2.11
SL-Transf [27]	RGB	14.97	15.25	9.05	6.51	5.20	14.93	17.64	7.41	3.98	2.52
TSPNet-Joint [31]	RGB	26.89	26.07	10.07	5.46	3.76	19.71	18.23	5.97	3.21	2.26
MMTLB [29]	RGB	17.64	18.35	14.27	9.76	6.11	18.90	19.64	5.30	3.26	2.31
GASLT [33]	Pose	35.74	28.19	16.00	11.93	9.95	18.76	15.57	6.06	3.72	2.72
GASLT [33]	RGB	31.46	25.05	10.18	6.25	4.73	22.01	19.54	7.45	4.41	2.56
Multi-Per. SLT	Input	R	B1	B2	B3	B4	R	B1	B2	B3	B4
SL-Luong [59]	RGB	14.21	13.58	6.63	3.83	2.45	14.04	15.53	6.11	3.27	2.05
SL-Transf [27]	RGB	13.53	14.45	7.58	4.48	2.86	13.68	16.58	5.86	2.72	1.55
TSPNet-Joint [31]	RGB	27.86	26.38	10.08	5.00	3.28	14.64	17.33	3.86	1.66	1.89
MMTLB [29]	RGB	20.53	16.54	11.32	6.84	4.52	17.76	16.02	4.81	2.83	1.83
GASLT [33]	RGB	29.33	23.62	9.44	5.69	4.23	19.73	16.99	6.25	3.44	2.26
SD+SLT	Pose	34.28	28.94	14.90	10.49	8.38	19.43	17.22	7.12	4.13	2.53
5 Auslan-Daily Benchmark
5.1 Video Representation
Pose-based video feature representation: Pose-based representations are robust against background clutters, lighting conditions, and occlusions, while explicitly depicting human hand and limb movements [51, 52, 53]. Several recent studies exploit pose information and achieve state-of-the-art performance in sign language translation-related tasks [9, 54, 55, 36, 56]. Hence, we use the key points extracted from Alphapose[16, 17, 18] as video features to provide benchmark results.
RGB-based video feature representation: Several models directly extract features from sign videos, such as CNN-RNN-HMM network [4], S3D [57], and I3D [58]. In the works [20, 36, 31], I3D is used for sign video representation. To better adapt to SL dataset and capture the spatio-temporal information of signs, inspired by [31], we finetune I3D on a word-level sign language recognition dataset and extract sign video features with different window widths and strides.
5.2 Benchmark Results
In this section, we provide benchmark results of sign language translation, alignment, active signer detection, fingerspelling detection, sign spotting and isolated sign recognition tasks on Auslan-Daily.
Sign Language Translation: In this task, we employ publicly available gloss-free SLT models, including (1) the RNN-based language translation model (SL-Luong [59]), (2) the Temporal Semantic Pyramid network (TSPNet [31]), (3) the Sign Language Translation Transformer model without glosses (SL-Transf [27]), (4) the multi-modality transfer learning based model (MMTLB [29]) and (5) transformer-based model with the gloss attention mechanism (GASLT [33]). Note that these models are designed to translate sign videos that only contain one single signer. To meet the input requirement of these SLT models, we crop the signer regions based on the ground-truth bounding-boxes of the signers. Since Auslan-Daily videos are captured in diverse scenes, RGB-based representation models may be affected by background clutter and various camera angles. In our single-person SLT experiments, GASLT performs best on the Auslan-Daily Communication subset, while SL-Luong excels on the Auslan-Daily News subset, both using pose points (hands & body) as input. In addition, to shed some light on how existing SLT models perform on Auslan-Daily with multiple persons in each video clip (without cropping acting signers), we directly feed video clips into SLT models. As indicated by Table 3, we observe significant performance degradation. This implies that existing SLT models do not have attention mechanisms to focus on active signers and non-signers can distract SLT. Therefore, we introduce a paradigm of signer detection followed by a translation model, denoted by  $\mathrm{SD + SLT}$ , in Table 3. It is observed that SD significantly facilitates translation in multi-person scenarios.
Sign Language Alignment: We use Subtitle Aligner Transformer (SAT) [21] model to evaluate the sign language alignment task. It employs Transformer [60] to synchronise subtitles with BSL videos and provides a pre-trained model [2]. The results are shown in Table 4. Leveraging the pre-trained
Table 4: The baseline of Sign Language Alignment (SLA) on Auslan-Daily. Comm., News and Mixed refer to two sub-datasets and the total combined dataset, respectively.
Fine-Tune	Test	F1@.10	F1@.25	F1@.50
News	News	80.59	72.30	50.08
Comm.	Comm.	87.58	77.41	66.33
Mixed	Comm.	81.28	71.81	64.27
News	85.13	77.00	53.05
Mixed	82.49	73.43	60.76
Table 5: The baseline of Signer Detection (SD) and Isolated Sign Language Recognition (ISLR) on Auslan-Daily.  $B$  and  $Hs$  represent Body and Hands keypoints, respectively.
Task	Model + Feature	Top-1	Top-5	Top-10
SD	TGCN + B + HS	88.35	-	-
I3D + Video	89.01	-	-
ISLR	TGCN + B + HS	14.82	25.37	37.32
I3D + Video	11.87	20.10	30.33
model provided by [21] and fine-tuning the sign language alignment model on Auslan enhances alignment performance.
Active Signer Detection: Active Signal Detection can be considered a binary action recognition problem. Inflated 3D ConvNet (I3D) model [58] is employed as the baseline on the RGB-based model. In addition, Pose-based Temporal Graph Convolution Networks (Pose-TGCN) [36, 61] serve as the pose-based model baseline. As shown in Table 5, Pose-TGCN and I3D achieve similar performance.
Isolated Sign Language Recognition: Isolated Sign Recognition is a multi-class action classification task. Similar to ASD, I3D and Pose-TGCN are adopted for the RGB-based and pose-based, respectively. Table 5 illustrates that the pose-based model performs better in complex scenes.
Fingerspelling Detection: We evaluate Auslan-Daily fingerspelling detection using the publicly available state-of-the-art fingerspelling detection model [41]. It is a multi-task model that combines pose estimation, recognition and detection tasks to improve the detection results jointly. The performance for the fingerspelling detector achieves 0.33/0.28/0.21 for AP@IoU(0.1/0.3/0.5). Comparing fingerspelling in Auslan-Daily Communication with Auslan-Daily News reveals more complexity and faster speed of fingerspelling in the latter.
Sign Spotting: We employ the sign spotting model [39] to evaluate the sign spotting task on the Auslan-Daily sign potting task. This method fusion multi-modal features, including RGB and poses, to obtain sign clip representations. Meanwhile, it introduces an innovative top-k transferring technique during testing to reduce the domain gap between isolated signs and continuous sign language. The baseline of this task is an F1 score of 0.27.
6 Discussion and Limitation
Reasons for Low Performance on Auslan-Daily News Translation: (1) Table 2 indicates that the dictionary size of Auslan-Daily News is much larger than that of Auslan-Daily Communication and Auslan-Daily News is long-tailed. These impose challenges in translating Auslan from videos to English [12, 2, 13]. (2) The experts commonly use abbreviations for named entities in videos, such as person and organization names. However, the corresponding words in the subtitles are in the full form. (3) Since "ABC News with Auslan" is a real-time TV program, the experts may summarise and interpret the simultaneous broadcasting news to Auslan, which could lead to missing words. (4) Fingerspelling recognition can be significantly affected by different camera angles and signing speeds, and thus current models struggle to identify each word, especially in news.
Reasons for Low Performance on Long-Tailed Isolated Sign Language Recognition: Since we annotate isolated signs which are distributed in the long tail in the vocabulary, the average number of sign videos for each sign is much smaller than that of normal ISLR dataset [36, 37]. Moreover, as our sign videos include diverse signers, environments and camera perspectives, our ISLR videos are more challenging. This scenario is also practical in SLT tasks since not every sign has a large number of corresponding videos. As a result, the ISLR accuracy in Table 5 is low than existing benchmarking results. As demonstrated in Table 5 (Appendix), we also adopt more methods to evaluate ISLR.
Coreference Resolution: Coreference occurs when two or more expressions refer to the same person or thing [62]. It is observed that translating Auslan faces the challenge of coreference resolution [15, 63]. Signers frequently employ coreference to nouns to reduce the signing complexity. Consequently, we suggest exploiting context information to address coreference resolution in sign language translation [64, 65, 66, 67].
Cross-Domain Investigation: The Auslan-Daily dataset is subdivided into Auslan-Daily Communication and Auslan-Daily News. The differences in sources and topics naturally create a domain gap. Auslan-Daily provides a practical dataset for investigating the cross-domain issue inherent in the sign language translation task [68, 69, 70, 55].
The Volume of Datasets: Our project aims at an ongoing exploration of machine sign language translation for Auslan. Unlike ASL and BSL, the high-quality data corpora of Auslan are relatively small. Therefore, we will continue incorporating content from "ABC News with Auslan" and other Auslan corpora to enrich Auslan-Daily. We intend to leverage existing Auslan data to provide additional annotations, similar to BOBSL [2]. In other words, we can employ a trained Auslan alignment model for preliminary annotations [21] and then conduct manual verification by experts.
In this work, we propose the first pubic available large-scale multi-person Auslan translation dataset with multi-grained annotation, named Auslan-Daily. Moreover, Auslan-Daily includes diverse topics and multiple signers performing in various environments. More importantly, the collected sign conversations are captured in the wild, significantly increasing the challenges of Auslan translation. Extensive experiments demonstrate the validity and challenges of our Auslan-Daily. Thanks to the multi-grained annotations, our dataset can be used for other sign language-related tasks. Furthermore, the presented benchmark results can act as strong baselines for future research. Although Auslan-Daily currently only has English transcriptions, we intend to provide gloss annotations and the benchmark on Continuous Sign Language Recognition (CSLR) in the future to further promote research on Auslan translation.
This research is funded in part by ARC-Discovery grant (DP220100800 to XY), ARC-DECRA grant (DE230100477 to XY) and Google Research Scholar Program. We gratefully thank all the anonymous reviewers and ACs for their constructive comments.
Auslan, like many other sign languages, has its distinctive features in semantics and pragmatics. The complexity and diversity of the expressions of Auslan present a significant hurdle in designing vision-language models. In this paper, the challenges posed by Auslan-Daily help stimulate the development of the vision-language community.
Moreover, as a visual language that unfolds in a three-dimensional space, sign language poses perspective-related complexities, such as capturing signs from a side-view perspective, that differ from written and spoken languages. Incorporating sign language into deep learning networks entails addressing these specific challenges. Releasing this dataset is part of open science. This dataset can encourage and support other researchers to conduct research on Auslan. By utilising this dataset, researchers can develop and improve algorithms and applications that understand and generate Auslan. For example, real-time sign language translation systems can be developed, making it easier for deaf individuals to communicate with others and enhancing their social participation and quality of life. This will also contribute to increased awareness and understanding of the Australian deaf community and foster broader social engagement.
Apart from computer science, our dataset will provide new opportunities for interdisciplinary research. For instance, linguists can use it to study the linguistic features and variations of Auslan, social scientists can investigate the culture and social interactions of the Australian deaf community, and psychologists and neuroscientists can explore the cognitive and neural mechanisms underlying sign language processing and learning. More broadly, the deaf community and sign language users are often overlooked in dataset and technological advancements. This oversight can lead to biases and imbalances within AI models as well. Our releasing this dataset will help bridge that gap and provide necessary data resources for creating more equitable and inclusive AI systems. In the rapidly advancing era of AI, it is of paramount importance to ensure that the needs and inclusion of the deaf community are not overlooked.
[62] D. Crystal. A dictionary of linguistics and phonetics. 1997.
[63] Gabrielle Hodge. Patterns from a signed language corpus: Clause-like units in Auslan (Australian sign language). PhD thesis, Macquarie University, 2014.
[64] Sameen Maruf, Fahimeh Saleh, and Gholamreza Haffari. A survey on document-level neural machine translation: Methods and evaluation. ACM Comput. Surv., 54(2):45:1-45:36, 2022.
[65] Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. In-context examples selection for machine translation. CoRR, abs/2212.02437, 2022.
[66] Lei Shen, Haolan Zhan, Xin Shen, and Yang Feng. Learning to select context in a hierarchical and global perspective for open-domain dialogue generation. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021, Toronto, ON, Canada, June 6-11, 2021, pages 7438-7442. IEEE, 2021.
[67] Lei Shen, Haolan Zhan, Xin Shen, Yonghao Song, and Xiaofang Zhao. Text is NOT enough: Integrating visual impressions into open-domain dialogue generation. In Heng Tao Shen, Yueting Zhuang, John R. Smith, Yang Yang, Pablo César, Florian Metze, and Balakrishnan Prabhakaran, editors, MM '21: ACM Multimedia Conference, Virtual Event, China, October 20 - 24, 2021, pages 4287-4296. ACM, 2021.
[68] Dongxu Li, Xin Yu, Chenchen Xu, Lars Petersson, and Hongdong Li. Transferring cross-domain knowledge for video sign language recognition. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 6204-6213. Computer Vision Foundation / IEEE, 2020.
[69] Roman Töngi. Application of transfer learning to sign language recognition using an inflated 3d deep convolutional neural network. CoRR, abs/2103.05111, 2021.
[70] Md. Monirul Islam, Md. Rasel Uddin, Md. Nasim AKhtar, and K.M. Rafiqul Alam. Recognizing multiclass static sign language words for deaf and dumb people of bangladesh based on transfer learning techniques. Informatics in Medicine Unlocked, 33:101077, 2022.