Auslan-Daily: Australian Sign Language Translation for Daily Communication and News
Authors: Xin Shen
Shaozu Yuan
Hongwei Sheng
Heming Du
Xin Yu
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Austrlain Sign Language
Auslan Dataset
TL;DR: We introduce an extensive Australian Sign Language (Auslan) dataset for advancing various sign language-related tasks.

Abstract: Sign language translation (SLT) aims to convert a continuous sign language video clip into a spoken language. Considering different geographic regions generally have their own native sign languages, it is valuable to establish corresponding SLT datasets to support related communication and research. Auslan, as a sign language specific to Australia, still lacks a dedicated large-scale dataset for SLT.
To fill this gap, we curate an Australian Sign Language translation dataset, dubbed Auslan-Daily, which is collected from the Auslan educational TV series and Auslan TV programs. The former involves daily communications among multiple signers in the wild, while the latter comprises sign language videos for up-to-date news, weather forecasts, and documentaries. In particular, Auslan-Daily has two main features: (1) the topics are diverse and signed by multiple signers, and (2) the scenes in our dataset are more complex, e.g., captured in various environments, gesture interference during multi-signers' interactions and various camera positions. With a collection of more than 45 hours of high-quality Auslan video materials, we invite Auslan experts to align different fine-grained visual and language pairs, including video $\leftrightarrow$ fingerspelling, video $\leftrightarrow$ gloss, and video $\leftrightarrow$ sentence. As a result, Auslan-Daily contains multi-grained annotations that can be utilized to accomplish various fundamental sign language tasks, such as signer detection, sign spotting, fingerspelling detection, isolated sign language recognition, sign language translation and alignment. Moreover, we benchmark results with state-of-the-art models for each task in Auslan-Daily. Experiments indicate that Auslan-Daily is a highly challenging SLT dataset, and we hope this dataset will contribute to the development of Auslan and the advancement of sign languages worldwide in a broader context. All datasets and benchmarks are available at Auslan-Daily.

Auslan-Daily: Australian Sign Language Translation for Daily Communication and News
Xin Shen
Shaozu Yuan² Hongwei Sheng'
Heming Du
Xin Yu!
'he University of Queensland
JD AI, Beijing, China
s.shen3ducconnect. edu
Sally Possum
ABC News with Auslan
Web
Auslan Daily Communication
Auslan Daily News
Diversity of our curated Auslan-Da -Dilly dataset. Auslan-Daily involves various topics, signers and diverse environments. In particular, there are multiple persons scene and signers appear in different areas of the scene.
Figure I:
Abstract
Sign language translation (SLT) aims to convert continuous sign language video clip into spoken language. Considering different geographic regions generally have their own native sign languages, is valuable to establish corresponding SLT datasets to support related communication and research. Auslan, as a sign language specific to Australia, still lacks dedicated large-scale dataset for SLT. To fill this gap. we curate an Australian Sign Language translation dataset, dubbed Auslan-Daily, which is collected from the Auslan educational TV series and Auslan TV programs. The former involves daily communications among multiple signers in the wild, while the latter comprises sign language videos for weather forecasts, and documentaries. In particular, Auslan- -aily has two main features: (I) the topics are diverse and signed by multiple signers, and (2) the scenes in our dataset are more complex, e.g., captured in various environments, gesture interference during multi-signers' interactions and various camera positions. With a collection of more than 45 hours of nigh-quality Auslan video materials, we invite Auslan experts to align different fine- tne-grained visual and language pairs, including video ++ fingerspelling, video ++ gloss, and video sentence. As result, Auslan-Daily contains multi multi-grained annotations that can utilized accomplish various fundamental sign language tasks, such as signer detection, sign spotting. fingerspelling detection, isolated sign language recognition, sign language translation and alignment. Moreover, models for each task in Auslan-Daily Daily. Experiments indicate that Auslan-Daily is highly challenging SLT dataset, and we hope this dataset will contribute to the development of Auslan and the advancement of sign languages worldwide in a broader context. All datasets and benchmarks available at ? Auslan-Daily.
news,
benchmark results with state-of-the-an art
*Conresponding author.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

(+) Amaniaa Enplish: bells my ramn fetreox
(b) Engalah: Eoer-ggring
qpring, sut planei hatelor
(6) CXC Chinnse: & T
(61 DGS -+ denmwry: Scrne ficcr be zehn good sundosii auch
Austalian English: Fossum,
Loma
Ausian Communication
Aucann Daily News
Figure 2: Comparisons of large-scale sign language translation datasets across diverse geographic regions. (a) How2Sign [1] (American), (b) BOBSL [2] (British), (c) CSL-Daily [3] (Chinese), (d) RWTH-PHOENIX--Wen Weather 2014T [4] (Germany), (e) Our proposed Auslan- -Dall (the yellow and black hounding-boxet indicate the signer and non-signer in the sign video clip).
1 Introduction
Sign language (SL) is the primary way for deaf or people with hearing loss to express themselves. Similar to various spoken languages, sign languages have their vocabularies and grammar [5, 6, 7]. More importantly, diverse geographic regions usually have their native sign languages even though these regions share a commonly spoken language, such as America, Australia and the UK. To eliminate the communication barriers between the deaf and hearing communities, sign language translation (SLT) has been proposed to convert signs into spoken languages [1, 2, 4,
With the emerging deep learning techniques and large-scale sign language datasets, SLT has achieved promising progress recently. As shown Figure 2. researchers from various countries have con- structed their sign language datasets and thus thrust SLT in their respective sign languages, such as American sign language (ASL) [1]. British sign language (BSL) [2]. Chinese sign language (CSL) [3] and Germany sign language (DGS) [4]. However, to the best of our knowledge, there is no publicly available large-scale Auslan dataset for continuous sign translation Meanwhile, according to the Hearing Care Industry Association", as of June 2015, one in six Australians had hearing loss affecting them and this proportion expected to increase to one in four by 2050. Due the societal inclusion and the regional nature of sign languages, Australian sign language (Auslan) datasets are inevitably and urgently needed order to investigate automatic translation.
Moreover, existing sign language corpora either are captured in controlled laboratory environments [1, 3, 8, 9, 10, 11] or only contain a single person (the signer) at a certain position in each video clip [2, 4, 12, 13J. In [14]. Núñez-Marcos et al. point out that the effectiveness of sign language translation can be compromised in multi-individual situations due to the presence of non-signers. Additionally, Yin er al. [15] emphasise that translating sign language in the wild is even more challenging. Thus, the existing datasets lack sufficient diversity and may not fully reflect the complexity the wild.
In this work, we aim to construct a large-scale Auslan translation dataset in the wild which contains sufficient high-quality Auslan videos and their English transcriptions. We adopt the first Australian educational TVV series for deaf children "Sally and Possum " "ABC News with Auslan" and several online domain-specific Auslan corpora our data source. Firstly, "Sally and Possum" is aired to help deaf children and their parents to learn Auslan, and it covers various topics of daily life. such as drawing pictures with colourful pigments and asking for advice in diverse environments, including indoor and outdoor scenes (Figure 1, left). Secondly, "ABC News with Auslan" starts broadcasting form 2022 and provides the latest news and information from ABC News every week with Auslan interpreted (Figure 1, middle). Lastly, we aggregate additional -ontent-specific Auslan data from
Siign videos segmented and annotated at the sentence level. 'nttps://www.brr hcia. com. wtt.aaa/w..eeeerree///////// pdf "nctps://earlychii www.hho.........e.oooooooollo..e gld. wtt...gg........a.llra/.... waw.rrlyyyaaararalllllaaaa
'nttpa://iview. .acc. wtw.eiee..ab..eet.aa....eeeeee.e www..t....ao/.abaaaoaaaaaaaa www.-ewwwwwwwwtteeeeeeeeeeee with auslan

---

Table 1: Comparison between Auslan Dailly and existing SLT datasets. PPC.: persons per clip.
Dataset
SL
Video
Yocah.
Vsigner
Source
Background
PFC.
Signer Position
Expert Check
PHOENIX-
DGS
8K
3K
TV
Fixed
Faed
A11
SIGNUM[LO
33K
ALL
ComienAAA1(1a]
DGGS-VGT
15:
Fined
A11
KGT1[9]
KSL
15K
0.5K
14
Lab
Fixed
1
Fixed
ALL
CSI-Duily{3]
CSL
21K
2K
Lab
Fixed
Fixed
ALL
How2Sign[1)
35K
Fised
Fixed
DGS Corpus[8]
DGS
63K
23k
327
Lab
Fixed
Fixed
ALL
Op
ASL
99g
33K
<-200
Web
Divens
Fixed
PARTIAL
ROBSI [3]
BSI.
1.2M
7.8K
39
TV
Diverse
Fixed
PARTIAL
Ausian Corpost [9]
Auslas
Fixed
Fixed
(isiaan-allliourss)
Ausdan
25K
14K
67
Web
Diverse
1-10
Diverse
ALL
online Auslan corpora, including natural disaster reports and interviews (Figure 1, right). We collect these high-quality video materials, which capture various environments and include diverse daily life topics, namely Auslan-Daily.
To better translate sign videos in Auslan-Daily, we devise a two-stage data annotation labelling process, i.e., (1) aligning video clips and transcriptions and (2) detecting the signer in each aligned video clip. The rationale behind these two labelling stages is as follows: in (1), despite the original whole sign videos accompanying English subtitles, misalignments often exist between each sign video clip and its corresponding complete subtitle; In (2), explicitly labelling the signer's position helps simplify multi-person scenes and reduce translation complexity. In the first stage, we invite five Auslan experts to perform multi-gained annotations, including the temporal boundaries of fingerspellings, long-tailed isolated glosses, and sentences. In the second stage, we apply Alphapose [16, 17, 18]. a tool for keypoints estimation and person tracking, to obtain each person's 's unique ID and pose sequences in each sign video clip. Then, for each sign clip, we manually modify the error of the Alphapose results and labelled the signer. After 300 working hours, we complete the fine- ane-ggained annotation labelling processing for our collected more than 45 hours of nigh-quality Auslan videos. Finally, in Auslan-Daas -aily, there are around 2k video < fingerspelling, 3k video ++ gloss, and 25k video ++ sentence pairs'.
Based on these multi multi-graiind annotations, we are able to investigate various sign language-related tasks in multi-person scenarios, including sign language translation, signer detection, fingerspelling detection, sign spotting, isolated sign language recognition and alignment. Specifically, we apply publicly available state-of-the-arr models for each task and then report the performance with corre- sponding evaluation metrics. Experimental results demonstrate the challenges of Auslan-Daily due to its rich diversity and complexity. Overall, the contributions of this work threefold:
We construct the first large-scale Auslan dataset, dubbed Auslan-Daily, which contains multi-person sign videos on various topics in diverse environments.
Auslan-Daily provides multi-grained annotations, enabling researchers to investigate a variety of tasks, including signer detection, fingerspelling detection, sign spotting, isolated sign language recognition, sign language alignment, and sign language translation.
We establish a leaderboard and an evaluation benchmark to promote Auslan SLT research.
2
Related Work
2.1
Sign Language Translation Datasets
Since current models for sign language translation are significantly data-driven and based on deep learning, assembling extensive sign language datasets becomes a pivotal component for advancing sign language translation across various nations. Datasets constructed for SLT in recent years are shown in Table 1, where DGS, KSL, CSL, ASL, BSL, and Auslan represent German, Korean, Chinese, American, British, and Australian Sign L Languages, respectively. Due to the limited lexical variety and sentence complexity within the SIGNUM [10] and KETI [9] datasets, they are unsuitable for [4] the first sign translation dataset used to translate German Sign
SLT tasks. PHOENIX-2014 Language (DGS). However, since this dataset only consists of weather forecast vidcos, the topic coverage is limited and may need to be increased for daily communication.
7 All the gloss and fingerspelling video clips can be found in sign sentence video clips. While fingerspelling and gloss videos have some overlaps, fingerspelling videos are not the subset of gloss annotations.
3

---

BOBSL [2. 20] and OpenASL [13] are large-scale sign language translation datasets. For these two extensive datasets, experts verification has been confined solely to the validation and test sets, whereas the training set has been annotated via a pre-trained BSL. sign alignment model [21] or through "self-generraed" time boundaries based on ASI. News. How2Sign [1] the largest American Sign Language (ASL) dataset captured in the lab environment. It contains a variety of annotations, including multi-view information, depth information, pose, and speech. CSL-Daily [3] and DGS Corpus [8] are the extensive datasets of Chinese Sign Language (CSL) and German Sign Language (DGS), respectively. How2Sign, CSL-Daily and DGS Corpus cover diverse topics. The datasets pertinent to Auslan represented by Auslan Signbank [22] and Auslan Corpus [23, 19, 24]. The Auslan Signbank constitutes a dictionary with approximately 5,500 Auslan glosses, while the Auslan Corpus [23, 19, 24] is primarily confined exploratory research by linguists and is not entirely accessible to the public. More importantly, the above datasets only consider one person (signer) in each sign video clip, and the signer appears controlled certain position in each sign video clip. However, in the real world, multiple people may perform sign gestures in a scene, and their positions diverse. Detecting the actual signer and accurately translating their gestural signs within complex environment presents a novel challenge. Factors such as perspective, illumination and the presence of crowds introduce noise and complexity when translating sign language spoken language. In contrast, we propose Auslan-Daily, the first publicly available real-world Auslan translation dataset. Meanwhile, we enrich the diversity and complexity by collecting source various data.
2.2 Tasks Associated with Sign Language
Currently, there are several tasks for investigating various granularities of sign language, such as fingerspelling (character), gloss (word) and sentence. All the sign language-related tasks aim to foster better sign language understanding and sign language translation development. The sign language- related task, including: (i) Sign Language Translation (SLT) [4] task is to translate sign video clip to the corresponding spoken language. The current SLT models can be divided into three categories, Sign2Gloss2Text, Slgn2(Gloss+Text, and Sign2Text [25]. The Sign2Gloss2Text [26] model is two- stage method, Sign2Gloss and Gloss2Text. respectively. The first stage is sign language recognition, which predicts a gloss sequence from a video, and the second stage translates the predicted gloss into the target natural language. Both (ign@(Gloss+Teex)) [27, 3, 28, 29, 30] and Sign2Texxt [31, 32, 33, 34] are end-to-end models. The difference that Sign2(Gloss+ Text) jointly trains the sign language recognition and translation tasks, and uses the gloss information as auxiliary supervision extract features from videos, thereby improving the translation results. Though Signs(Gioss+Texx) models perform well on existing datasets, obtaining large-scale sign language translation data with continuous glosses annotation is costly and time- consuminn The primary reason is that the process of annotating one hour of continuous sign language video [21] requires an expert, who is proficient in sign language, approximately ten to fifteen hours. Thus, it is hard to apply them to SL datasets that do not contain gloss annotations. For Sign2Text models, they aim to directly convert sign language performed by a single person into target natural language. Furthermore, as the existing datasets are not collected in the wild, previous models might fail to tackle complex scenarios and real-world situations with multiple people; (ii) Sign Language Alignment [21, 35) temporally aligns asynchronous subtitles in sign language videos. A proficient alignment model for sign language can mine more sign data for automated translation; (iii) Isolated Sign Language Recognition [36, 37, 20] focuses on identifying and understanding individual gestural signs, independent of any surrounding context or sequence of signs; (iv) Sign Spotting [38. 39, 40] aims to find accurate locations of the given isolated signs in continuous co-articulated sign language videos; (v) Fingerspellingg Detection [41. 42, 43] finds the fingerspelling segments' intervals within the clip. Fingerspelling is an important component of Sign Language, in which words are signed letter by letter and (vi) Active Signer Detection, also known as Sign Language Detection [44, 45], is identical to the initial stage of Signer Diarisation [46]. aims to find the signer in the sign video clip.
3 Auslan-Daily Dataset
In this section, we describe data collection and cleaning, detail the data labelling procedure and provide statistics of the Auslan-Daily train/test split.
4

---

Table 2: Key statistics of Auslan Auslan-Daily, Auslan- Daily Communication and Auslan-Daily News are two sub-datasets split from Auslan-Daily Daily. OOV: out- voocabulaa....y...... Singleton: words that only occur once the training dataset.
Sub Dutaset
Auslan-Dally Communication
Auslan Daily News
Domain Topii Video Resolutiion 68 FPS
Comounication 1920x 10806 25
Documentary
1280 73011900 1080 97
Splitt
Train
Dev
Test
Train
Day
Test
Toral
Segmeats
12,441
800
800
9,665
7000
700
25,106
Signers
49
12
9
18
17
17
67
Frimes
930,321
45,369
45.171
2,072,475
144,819
142,893
3,181,048
Vocals
3,064
522
469
12,346
2,872
2,885
13,945
Tol. words
88,167
4,126
4,115
163,268
11,376
11,330
282,582
Tol. DOVss
8
10
326
304
Singlctons
1,045
5,267
Person per clip
1-10
3.1 Data Collection and Cleaning
"SSaly and Possum", "ABC News with Auslan" and Auslan corpora from YouTube are public TV programs and open sources "Sally and Possum", a premium Australian sign language show for deaf children. To facilitate children learning of Auslan, the production team designs the plots and writes the English scripts. Subsequently. Auslan experts perform sign language based on the script. This show has 6 scasons and 15 episodes per season, which is 22.5 hours long and contains around 20 topics, including daily communication, study skills, and knowledge explanation. Beginning in 2022, "ABC News with Auslan" weekly broadcasts key domestic and international events news and weather forecasts. It is an ongoing TV program, and the current dataset includes 45 news videos as of May 2023. Auslan experts execute real-time sign language translation (simultaneous interpretation) for deaf or people with hearing loss based on currently broadcast news. However, the English subtitles may leave the following problems [21]: (i) the order of subtitles is not complying between spoken and sign languages, and (ii) the duration of a subtitle varies considerably between signing and speech due to differences in speed and grammar. To enrich the topics of our dataset, also collect several publicly available high-quality documentaries interpreted with Auslan. These data comprise specific thematic areas, including disaster overviews, preventative measures, and interviews.
All of the original videos are recorded with standard English dubbing and subtitles. We download subtitles of each whole original video, which are arranged with the format "[Start Time] subtitle [End Time]". The time intervals are marked based on dubbing. Upon spot check, we observe that longer subtitles might extend across multiple temporal intervals, whereas several shorter subtitles tend to appear within a time interval. To procure complete sentence-level subtitles, we conduct the data cleaning operations as follows: (1) for incomplete subtitles, egg., ending with comma, we connect them with the following subtitles to compose complete sentences and merge their time duration; (2) for several complete subtitles that appear within time interval, we partition them into multiple independent sentences; (3) for a complete sentence that only contains modal particles, e.g., "Oh!" and "Ha ha ha!". we remove them avoid meaningless translation. As a result, we acquire approximately 29k complete subtitles that require alignment.
3.2 Two-Stage Data Labelling Procedure
To obtain applicable data for sign language translation, design two stages of the data labelling procedure: (1) aligning video clips and transcriptions and (2) detecting the signer in each aligned video clip. They are imperative as original subtitles often misalign with sign videos, and the position of the signer varies. By analyzing the frequency of individual tokens across all subtitles, we notice a considerable presence of long-tailed words. Due their sparse occurrence, these words impose challenges for the model and affect the translation results. To investigate this problem, sign language experts annotate the less commonly used glosses in "Sally and Possum".. Experts also label the temporal boundaries of significant fingerspelling instances, as fingerspelling is commonly used in the deaf community. Therefore, in the first stage, we engage Auslan experts to synchronise video <>> fingerspelling, video <+ gloss, and video <>> sentence pairs.
Next, we employ Alphapose [16, 17, 18] to track people in each aligned video clip. Then, we record trajectories and pose sequences along with their corresponding IDs. Considering Alphapose
*Our dataset follows the copyright Creative Commons BY-NC-ND 4.0 license @.
5

---

Ausan-Daill Communication
Ausan-Dally News
Ausan-Dally Overall
Clip
Number of Video Clips
Formort
Cool
Sporte
Show
16
Podto
Rav
10
203
303
20
5i nx
230
150
Boly Prosert
Number Frames
Number Frames
Teckeology
Gatt
thems Peele
Nf Video Clpe
Numoe of Vidco Cit
2019
Sedtl Probder
1800
2013
Disases Prosection
Econenie
Law
Paint
Sudt
Intervine
ane Tometal
Profeator
Mearth
Rostino
ove
Number Word
Number of Word
Number Word
Figure 3: Distributions of #frames/#words over clips (L) and the topic diversity in Auslan-Daily (R).
may suffer tracking errors sometimes', we thus invite annotators to manually check and modify the tracking results by assigning correct IDs the signers.
To guarantee the annotation quality of our dataset, we conduct a cross-check verification process during each data labelling procedure stage. Specifically, ask each Auslan annotator as examiner to cross-check around 5% of annotated video clips provided by another annotator. The video clips chosen randomly. If the examiner finds more than 10% of annotated videos have obvious errors, third annotator is invited to review and correct the annotations.
Through the collaborative efforts of five Auslan experts and five annotators, we complete all annota- tions with approximately 300 work hours. Overall, our Auslan-Daile -Dily dataset contains the following annotations: (I) temporal boundaries of sign video clips; (2) temporal boundaries of long-tailed glosses; (3) temporal boundaries of partial fingerspellings: (4) pose sequences of signers and non- signers; (5) bounding boxes of signers; (6) signer identities and (7) English transcriptions. These multi-grained annotations can be further investigated for Australian sign language-related tasks.
3.3 Data Statistics
During the data labelling, experts discard erroneous data, such subtitles do not have corresponding sign language. As demonstrated in Table 2, there are in total 25, 106 video clips encompassing 67 unique signers, with the vocabulary size of 13,945 words. It should be noted that signers in the validation and test sets appear in the training set. As the number of persons ranges from I to 10 in video clip, the distractions, such as gesture interference of multi-persons, are also involved in sign language translation, thus imposing challenges in this task. To verify the robustness of the various models, we also statisties of the sentences within the test set. As shown in Table 3 (Appendix), over 80% of video clips in the test set encompass distinct sentences. Therefore, the robustness of the models can be verified by evaluating on the test set.
After examining the domains of the three data sources, "ABC News with Auslan" and the data from online Auslan corpora are found significantly similar. Thus, we combine the data collected from these two video sources and partition the Auslan-Daily dataset into two sub-datasets: Auslan-Dail -aily Communication and Auslan-Dail training, validation and test sets as shown in Table 2. Each sign video clip has 73 frames and 7.8 words on average for the Auslan-Daily Communication sub-dataset. Each sign video clip has 214 frames and 16.9 words on average for the Auslan Auslan-Daily News sub-dataset. The distributions of frames and words in different splits of the sub-datasets are shown in Figure 3 (left).
News. We randomly split the two sub-datasets data into the
Moreover, Auslan-Daily Daily has 3,000 (600 classes) long-tailed isolated glosses and 2,000 significant fingerspellings. For the isolated gloss, we select words appearing less than ten times after the natural language corpus post-lemmatization" 10 in "Sally and Possum". Its main purpose is to enhance translation models recognise long-tail words through sign language recognition or spotting tasks. Furthermore, considering the pivotal role of fingerspelling in sign language translation, we thus
Thhe overall error rate is less than 20 percent video clips with multiple signers. 10 "Lemmatization facilitates more fine-graine fine-grained and accurate analysis and representation of text.

---

Sign Language Aligrmmet
Possum.
SALLY
Possum.
Temd
Signer Datection
Fingerspling Detectioo
Sign Spotting
POSSUM
Isolated SLR
Sign Language Translation:
am Sally.
am Possum.
Figure 4: Overview of Auslan-Daily tasks and provided annotations.
annotate Auslan fingerspelling instances. As illustrated in Figure 3 (right), there are 21 types of daily communication and more than 15 types of daily news in our Auslan- Aoilan-Daily dataset.
Overview of Auslan-Daily Tasks
4.1 Task Definition
Active Signer Detection (ASD) [44, 45, 46]: Given trajectories of N people P = {pi H in sign video clip V [fili. the goal of ASD find the active signer pe in the current video clip.
Sign Spotting (SS) [38, 39]: Given sign video clip V with frames and an isolated gloss II clip, the goal of SS is to locate the start frame Sn and end frame of in V.
Isolated Sign Language Recognition (ISLR) [36, 37, 20]: Given an isolated gloss T clip, ISLR aims to determine the gloss category c of II, where is pre-defined by Sign Dictionary.
Fingerspelling Detection (FD) [41]: Given sign video clip V with frames, the goal of FD is to locate the start frame fm and end frame fn of each fingerspelling in V.
Sign Language Alignment (SLA) [21]: Given whole original video V {fi]T with T frames and its corresponding M subtitles = {M the goal of SLA is to align each subtitle sentence s; with the precise begin frame f; and end frame fj-
Sign Language Translation (SL.)) [4]: Given sign video clip V with frames, the goal of SLT to translate V to an English sentence 84-
4.2 Evaluation Metrics
BLEU [47] and ROUGE [48] scores are commonly-used evaluation metrics for Sign Language Translation [4]. BLEU-n measures the n-gram overlap between the generated text and reference text, and ROUGE-L measures the Fl score based the longest common subsequences between the generated text and reference text.
BLEU and ROUGE:
Top-K Accuracy: Top-K classification accuracy measures the number of ground-truth labels within the top i predicted labels. For Signer Detection, K = is employed. For Isolated Sign Language Recognition, adopt K values of 5, and 10.
IoU: Intersection over Union (loU) is defined as the ratio the intersection and union of predicted and actual time intervals of an action within video [49, 50). Similar to [41], we employ AP@IOU) as a metric for Fingerspelling Detection. Following [21, 39), we use F@IQU IOU evaluate Sign Language Alignment and Sign Spotting tasks.
7

---

Table 3: Translation results of Single/Multi-Perssee SLT gloss-free models on Auslan- Daily.
Auslan- Auslan-Daily Communication
Austan Dally News
Siagle-Per. SLT
B1
B12
B3
R
B1
B4
SL-Loung [59]
Pose
37.27
30.15
16.26
11.67
9.45
20.65
19.84
7.81
4.59
2.81
5.74
16.92
2.65
SL-Transf [27)
Pase
35.65
31.31
16.17
11.41
9.20
20.25
21.25
6.57
3.32
2.11
SL-Tranff [27]
RGB
14.97
15.25
9.05
6.51
5.20
14.93
17.64
7.41
3.98
2.52
TSPNat-Joit [31]
RGB
26
26.07
10.07
5.46
3.76
19.71
18.23
5.97
3.21
2.26
GASLT [33]
Pose
35.74
28.
16,00
11.93
9.95
18.
13.57
6.06
3.72
2.72
CASLT [33]
RGB
31.46
25.05
10.18
6.25
4.73
22.01
19.54
7.45
4,41
2.56
Multi-Per. SLT
Inpui
B1
B3
B3
R
B1
mong (59)
6.63
3.83
15:53
2.05
St--Tandd [27]
RGB
13.53
14.45
7.58
4.48
2.86
13.68
16.58
5.86
2.72
1.55
TSPNat-oin [3I]
RGB
27.86
26.38
10.08
5.00
3.78
14.64
17.33
3.86
1.65
1.89
16.54
16.02
GASLT [33]
RGB
29.33
23.62
9,44
5.69
4.23
19.73
16.99
6.25
3.44
2.26
17.22
Auslan-Daily Benchmark
5.1 Video Representation
Pose-based video feature representation: Pose-based representations are robust against back- ground clutters, lighting conditions, and occlusions, while explicitly depicting human hand and limb movements [51, 52, 53]. Several recent studies exploit pose information and achieve state-of-the-ar performance in sign language iosllation-relatee tasks [9, 54, 55, 36. 56]. Hence, points extracted from Alphapose[16, 17, 18] as video features to provide benchmark results.
the key
RGB-based video feature representation: Several models directly extract features from sign videos, such as CNN-RNN-HMM network [4], S3D [57], and 13D [58]. In the works [20, 36, 31], J, I3D used for sign video representation. To better adapt to SL. dataset and capture the spatio-temporal information of signs, inspired by [31], we finetune I3D on a word-level sign language recognition dataset and extract sign video features with different window widths and strides.
5.2 Benchmark Results
In this section, we provide benchmark results of sign language translation, alignment, active signer detection, fingerspelling detection, sign spotting and isolated sign recognition tasks on Auslan-Daiiy
Sign Language Translation: In this task, we employ publicly available gloss-free SLT models, including (1) the RNN-based language translation model (SL-Luong [59]). (2) the Temporal Semantic Pyramid network (TSPNet [31]). (3) the Sign Language Translation Transformer model without glosses (SL-Transf [27]), (4) the multi-modality transfer learning based model (MMTLB [29]) and (5) transformer-based model with the gloss attention mechanism (GASLT [33]). Note that these models are designed to translate sign videos that only contain one single signer. To meet the input requirement of these SLT models, we crop the signer regions based on the ground-truth bounding- boxes of the signers. Since Auslan-Daily videos are captured in diverse scenes, RGB-based representation models may be affected by background clutter and various camera angles. In our single-person SLT experiments, GASLT performs best on the Auslan-Daily Communication subset, while SLo-Lunn excels on the Auslan-Daily News subset, both using pose points (hands & body) as input. addition, to shed some light on how existing SLT models perform on Auslan-Daily with multiple persons in each vidco clip (without cropping acting signers), we directly feed video clips into SLT models. As indicated by Table 3, we observe significant performance degradation. This implies that existing SLT models do not have attention mechanisms to focus active signers and non-signers can distract SLT. Therefore, we introduce a paradigm of signer detection followed by translation model, denoted by SD+SLT, in Table 3. It is observed that SD significantly facilitates translation in multi-person scenarios.
Sign Language Alignment:
We use Subtitle Aligner Transformer (SAT) [21] model to evaluate the sign language alignment task. employs Transformer [60] synchronise subtitles with BSL. videos and provides pre-trainn pre-trained model [2]. The results are shown in Table 4. Leveraging the pre-trained

---

Table 4: The baseline of Sign Language Align- ment (SLA) on Auslan-Daily. Comm., News and Mixed refer to two sub-datasets and the total com- bined dataset, respectively.
Table The baseline of Signer Detection (SD) and Isolated Sign Language Recognition (ISLR) on Auslan-Daily. and Hs represent Body and Hands keypoints, respectively.
Fine Tune
Text
F1G.10
F16.25
F16.50
Tisk
Model Feature
Top-1
Top-5
Top-10
News
News
80.59
72.30
50,08
TGCN + B + IIS
86.35
Comm.
Comm.
87.58
77.41
66.33
13D + Video
89.01
Comm.
81.28
71.81
64.27
Mixed
News
85.13
77.00
53.05
ISLR
TGCN R HS B3D Video
25.37
37.32
Mixed
$2.49
73:43
60.76
20.10
30.33
model provided by [21] and fine-tuning the sign language alignment model on Auslan enhances alignment performance.
Active Signal Detection can be considered binary action recognition problem. Inflated 3D ConvNet (I3D) model [58] is employed as the baseline on the RGB-baser model. In addition, Pose-based Temporal Graph Convolution Networks (Pose-TGees TGCN) [36, 61] serve as the pose-based model baseline. As shown in Table 5, Pose-TGCN and I3D achieve similar performance.
Active Signer Detection:
Isolated Sign Language Recognition: Isolated Sign Recognition is a multi-class action classifi- cation task. Similar to ASD, I3D and Pose-TGCN are adopted for the RGB-based and pose-based, respectively. Table 5 illustrates that the pose-hased model performs better in complex scenes.
Fingerspelling Detection:
We evaluate Auslan-Daill fingerspelling detection using the publicly available state-of-the-art fingerspelling detection model [41]. It is multi-task model that combines pose estimation, recognition and detection tasks to improve the detection results jointly. The perfor- Comparing
mance for the fingerspelling detector achieves 0.33/0.28/0. for AP@ fingerspelling in Auslan -aily Communication with Muslan-Daily News reveals more complexity and faster speed of fingerspelling in the latter.
Sign Spotting: We employ the sign spotting model [39] to evaluate the sign spotting task on the Auslan-Daily sign potting task. This method fusion multi-modal features, including RGB and poses, to obtain sign clip representations. Meanwhile, it introduces an innovative top-k transferring technique during testing reduce the domain gap between isolated signs and continuous sign language. The baseline of this task is an Fl score of 0.27.
6 Discussion and Limitation
Reasons for Low Performance on Auslan-Daily News Translation: (1) Table 2 indicates that the dictionary size of Auslan-Daily News is much larger than that of Auslan-Daily Communication and Auslan-Daily News is long-tailed. These impose challenges in translating Auslan from videos to English [12. 2, 13]. (2) The experts commonly use abbreviations for named entities in videos, such as person and organization names. However, the corresponding words in the subtitles are in the full form. (3) Since "ABC News with Auslan" is real-time TV program, the experts may summarise and interpret the simultaneous broadcasting news to Auslan, which could lead to missing words. (4) Fingerspelling recognition can be significantly affected by different camera angles and signing speeds, and thus current models struggle to identify each word, especially in news.
Reasons for Low Performance on Long-Tailed Isolated Sign Language Recognition: Since we annotate isolated signs which are distributed in the long tail in the vocabulary, the average number of sign videos for each sign is much smaller than that of normal ISLR dataset [36, 37J. Morcover, as our sign videos include diverse signers, environments and camera perspectives, our ISLR videos are more challenging. This scenario is also practical in SLT tasks since not every sign has a large number of corresponding videos. As result, the ISLR accuracy in Table 5 is low than existing benchmarking results. As demonstrated in Table 5 (Appendix), we also adopt more methods to evaluate ISLR.
Coreference Resolution: Coreference occurs when two or more expressions refer to the same person or thing [62]. It is observed that translating Auslan faces the challenge of coreference resolution [15, 63]. Signers frequently employ coreference to nouns to reduce the signing complexity. Consequently, we suggest exploiting context information to address coreference resolution in sign language translation [64, 65, 66, 67].

---

Cross- Cross-Domain Investigation: The Auslan-Dail -aily dataset is subdivided into Auslan-Dail -aily Communi- cation and Auslan-Daily News. The differences in sources and topics naturally create domain gap. Auslan-Daily provides practical dataset for investigating the cross-domain issue inherent in the sign language translation task [68, 69, 70, 55].
The Volume of Datasets:
Our project aims at an ongoing exploration of machine sign language translation for Auslan. Unlike ASL and BSL, the high-quality data corpora of Auslan are relatively small. Therefore, we will continue incorporating content from "ABC News with Auslan" and other Auslan corpora to enrich Auslan-Dall We intend leverage existing Auslan data to provide additional annotations, similar to BOBSL [2]. In other words, we can employ a trained Auslan alignment model for preliminary annotations [21] and then conduct manual verification by experts.
7 Conclusion
In this work, we propose the first pubic available large-scale multi-person Auslan translation dataset with multi-grained annotation, named Auslan-Dail Moreover, Auslan- Daily includes diverse topies and multiple signers performing in various environments, More importantly, the collected sign conversations are captured in the wild, significantly increasing the challenges Auslan translation. Extensive experiments demonstrate the validity and challenges of our Auslan-Daily. Thanks to the multi-grained annotations, our dataset can be used for other sign language-related tasks. Furthermore, the presented benchmark results can act as strong baselines for future research. Although Auslan- Daily currently only has English transcriptions, we intend to provide gloss annotations and the benchmark on Continuous Sign Language Recognition (CSLR) in the future to further promote research on Auslan translation.
Acknowledgement
This research is funded in part by ARC-Discovery grant (DP220100800 to XY), ARC-DECRA grant (DE230100477 to XY) and Google Research Scholar Program. We gratefully thank all the anonymous reviewers and ACs for their constructive comments.
Broader Impact
Auslan, like many other sign languages, has its distinctive features in semantics and pragmatics. The complexity and diversity of the expressions of Auslan present significant hurdle in designing vision-language models. In this paper, the challenges posed by Auslan-Daily help stimulate the development of the vision-language community.
Moreover, a visual language that unfolds in diree-dimensional space, sign language poses perspective-reltted complexities, such as capturing signs from a side-view perspective, that differ from written and spoken languages. Incorporating sign language into deep learning networks entails addressing these specific challenges. Releasing this dataset is part of open science. This dataset can encourage and support other researchers to conduct research on Auslan. By utilising this dataset, researchers can develop and improve algorithms and applications that understand and generate Auslan. For example, real-time sign language translation systems can be developed, making easier for deaf individuals to communicate with others and enhancing their social participation and quality of life. This will also contribute to increased awareness and understanding of the Australian deaf community and foster broader social engagement.
Apart from computer science, our dataset will provide new opportunities for interdisciplinary research. For instance, linguists can use it to study the linguistic features and variations of Auslan, social scientists can investigate the culture and social interactions of the Australian deaf community, and psychologists and neuroscientists can explore the cognitive and neural mechanisms underlying sign language processing and learning. More broaderly, the deaf community and sign language users are often overlooked in dataset and technological advancements. This oversight can lead to biases and imbalances within Al models as well. Our releasing this dataset will help bridge that gap and provide necessary data resources for creating more equitable and inclusive Al systems. In the rapidly advancing era of AI, is of paramount importance to ensure that the needs and inclusion of the deaf community are not overlooked.
10

---

References
[1] Amanda Cardoso Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram. Kenneth DeHaann Florian Metze, Jordi Torres, and Xavier Giró-i-Nicto. How2sign: large-scale multimodal dataset for continuous american sign language. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021,
virtual, June 19-25, 2021, pages 2735-2744 Computer Vision Foundation/ / JEEE, 2021.
[2] Samuel Albanie, Gül Varol, Liliane Momeni, Hannah Bull, Triantafyllos Afouras, Himel Chowdhury, Neil Fox, Bencie Woll, Rob Cooper, Andrew McParland, and Andrew Zisserman. Bbc-oxford british sign language dataset. CoRR, abs/211 1.03635, 2021.
[3] Hao Zhou, Wengang Zhou, Weizhen Qi. Junfu Pu, and Houqiang Li. Improving sign language translation with monolingual data by sign back- translation. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 1316 1325. Computer Vision Foundation IEEE, 2021.
[4] Necati Cihan Camgöz. Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. Neural sign language translation. In 2018 IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 7784-7793. Computer Vision Foundation / IEEE Computer Society, 2018
[5] Natasha Abner, Carlo Geraci. Shi Yu, Jessica Lettieri, Justine Mertz. and Anah Salgat. Getting the upper hand sign language families: Historical analysis and annotation methods. FEAST. Formal and Experimental Advances in Sign language Theory, 3:17-29, 2020.
[6] William C Stokoe Jr. Sign language structure: An outline of the visual communication systems of the american deaf. Journal of deaf studies and deaf education, 10(1):3-37, 2005.
William C Stokoe. Sign language structure. Annual review of anthropology, 91):365-330, 1980.
[8] Thomas Hanke, Mare Schulder, Reiner Konrad, and Elena Jahn. Extending the Public DGS Corpus in size and depth. In Proceedings of the LREC2020 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives, pages 75-82, Marseille, France, May 2020. Furopean Language Resources Association (ELRA).
[9] Sang- Ki Ko, Chang Jo Kim, Hyedong Jung, and Choong Sang Cho. Neural sign language translation based human keypoint estimation. CoRR, 1/5/1811.11453 2018
[10] Urrich von Agris, Moritz Knorr. and Karl-Friede Friedrich Kraiss. The significance of facial features for automatic sign language recognition. In 8th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2008), Amsterdam, The Netherlands, 17-19 September 2008, pages 1-6. IEEE Computer Society, 2008
[11] Aoxiong Yin, Zhou Zhao, Weike Jin, Meng Zhang, Xingshan Zeng, and Xiaofei MLSLT: towards mul- tilingual sign language translation. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, CVPR 2022. New Orleans, LA, USA, June 18-24, 2022, pages 1109. IEEE, 2022.
[12] Necati Cihan Camgöz, Ben Saunders, Guillaume Rochette, Marco Giovanelli, Giacomo Inches, Robin Nachtrab-Ribback, and Richard Bowden. Content4all open research sign language translation datasets. In 16th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2021, Jodhpur, India, December 15-18, 2021, pages 1-5, JEEE, 2021.
[13] Bowen Shi, Diane Brentari, Gregory Shakhnarovich, and Karen Livescu. Open-domain sign language translation learned from online video. In Yoav Goldberg, Zomitsa Kozareva, and Yue Zhang, editors. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 6365-6379 Association for Computational Linguistics, 2022.
[14] Adrián Nárez-Marros, Olatz Perez-de- Viiiiiisee and Gorka Labaka. A survey on sign language machine translation. Expert Syst. Appl., 213(Part): 118933, 2023.
[15] Kayo Yin. Amit Moryossef, Julie Hochgesang, Yoav Goldberg, and Malihe Alikhani. Including signed languages in natural language processing. In Chengqing Zong, Fei Xia. Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL//CNLP 2021. (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages -3600 Association for Computational Linguistics. 2021.

---

[16] Hao-Shu Fang. Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Li, and Cewu Lu. Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2022.
[17] Hao-Shu Fang, Shuqin Xie, Yu- Vu-Wing Tai, and Cewu Lu. RMPE: Regional multi pelliipprsonnnee pose estimation. In ICCV, 2017.
[18] Jiefeng Li, Can Wang, Hao Zhu, Yihuan Mao, Han-Shu Fang, and Cewu Lu, Crowdpose: Efficient crowded scenes pose estimation and new benchmark. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10863-10872, 2019.
[19] River Tae Smith, Louisa Willoughby, and Trevor Johnston. Integrating Auslan resources into the language data commons of Australia. In Proceedings of the LREC2022 10th Workshop on the Representation and Processing of Sign Languages: Multilingual Sign Language Resources, pages 181-186, Marseille, France, June 2022. European Language Resources Association.
[20] Samuel Albanie, Gul Varol, Liliane Momeni, Triantafyllos Afouras, Joon Son Chung, Neil Fox, and Andrew Zisserman. BsI-Ik: lk: Scaling up co-articulated sign language recognition using mouthing cues. In European conference computer vision, pages 55-53. Springer, 2020.
[21] Hannah Bull, Triantafyllos Afouras, Gül Varol, Samuel Albanie, Liliane Momeni, and Andrew Zisserman. Aligning subtitles in sign language vidcos. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canuda, October 10-17, 2021, pages 11532-11541. IEEE, 2021.
[22] Steve Cassidy, Onno Crashom, Henri Nieminen, Wessel Stoop, Micha Hulsbosch, Susan Even, Erwin Komen, and Trevor Johnson. Signbank: Software support web based dictionaries of sign language. In Nicoletta Calzolari. Khalid Choukri, Christopher Cieri. Thierry Declerck, Sara Goggi, Kôiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asunción Moreno, Jan Odijk. Stelios Piperidis, and Takenobu Tokunaga, editors, Proceedings of the Fleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018. European Language Resources Association (ELRA), 2018.
[23] Trevor Johnston. From archive to corpus: transcription and annotation in the creation of signed language corpora. In Rachel E. O. Roxas, editor, Proceedings of the 22nd Pacific Asia Conference on Language, Information and Computation, PACLIC 22, Cebu City, Philippines, November 20-22, 2008, pages 16-29. De La Salle University (DLSU). Manila, Philippines, 2008
[24] Trevor Johnston. From archive to corpus: Transcription and annotation the creation of signed language corpora. International journal of corpus linguistics, 15(1):106-13)) 2010.
[25] Yutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, and Stephen Lin. A simple multi-modality transfer learning baseline for sign language translation. CoRR, abs/2203 04287, 2022.
[26] Kayo Yin and Jesse Read. Better sign language translation with stme- trae-transfommer. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5975-5989, 2020.
[27] Necati Cihan Camgöz, Oscar Koller, Simon Hadfield, and Richard Bowden. Sign language transformers: Joint end-to-end sign language recognition and translation. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition. CVPR 2020, Seattle, WA, USA. June 13-19, 2020, pages 10020-10030. Computer Vision Foundation IEEE, 2020.
[28] Hao Zhou, Wengang Zhou, Yun Zhou, and Houqiang Li. Spatial-temporal multi-cue network for sign language recognition and translation. IEEE Transactions on Multimedia, 24:768 779, 2021.
[29] Yutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, and Stephen Lin. A simple multi modality transfer learning baseline for sign language translation. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 51 10-5120. IEFF, 2022.
[30] Yutong Chen, Ronglai Zuo Fangyun Wei, Yu Wu, Shujic Liu, and Brian Mak. Two-stream network for sign language recognition and translation. In NewrlPS, 2022.
[31] Dongxu Li, Chenchen Xu, Xin Yu, Kaihao Zhang, Benjamin Swift, Hanna Suominen, and Hongdong Tspnet: Hicrarchical feature learning via temporal semantic pyramid for sign language translation. In Hugo Larochelle, Mare Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
12

---

[32] Jian Zhao. Weizhen Qi. Wengang Zhou, Nan Duan, Ming Zhou, and Houqiang Li. Conditional sentence generation and cross-modal reranking for sign language translation. IEEE Transactions on Multimedia, 24:2662-2672, 2021.
[33] Aoxiong Yin, Tianyun Zhong, Li Tang, Weike Jin, Tao Jin, and Zhou Zhao. Gloss attention for gloss free sign language translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2551-2562, 2023.
[34] Jiangbin Zheng, Yidong Chen, Chong Wu, Xiaodong Shi, and Suhail Muhammad Kamal. Enhancing neural sign language translation by highlighting the facial expression information. Neurocomputing, 464:462-472
2021.
[35] Liliane Momeni, Hannah Bull, K. R. Prajwal, Samuel Albanie, Gül Varol, and Andrew Zisserman. Automatic dense annotation of vogeblocay sign language vidcos. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision ECCV 2022- 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXV, volume 13695 of Lecture Notes in Computer Science, pages 671-690. Springer, 2022.
[36] Dongxu Li, Cristian Rodriguez, Xin Yu, and Hongdong Li. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. In The IEEE Winter Conference on Applications of Computer Vision, pages 1459-1469, 2020.
[37] Hamid Reza Vaezi Joze and Oscar Koller. MS-ASL: A large-scale data set and benchmark for understanding american sign language. In 30th British Machine Vision Conference 2019, BMVC 2019, Cardiff. UK, September 9-12, 2019, page 100. BMVA Press, 2019.
[38] Liliane Momeni, Gül Varol, Samuel Alhanie, Triantafyllos Afouras, and Andrew Zisserman. Watch, read and lookup: I Learning to spot signs from multiple supervisors. Hiroshi Ishikawa, Cheng-Lin Liu, Tomás Pajdla, and Jianbo Shi, editors, Computer Vision ACCV 2020 - 15th Asian Conference on Computer Vision, Kyoto, Japan November 30 December 4, 2020, Revised Selected Papers, Purt VI, volume 12627 of Lecture Notex in Computer Science, pages 291-308. Springer, 2020.
[39] Hongyu Fu, Chen Liu, Xingqun Qi. Beibei Lin, Lincheng Li, Zhang, and Xin Yu. Sign spotting via multi-modal fusion and testing time transferring. In Leonid Karlinsky, Tomer Michaeli, and Ko Nishino, editors, Computer Vision ECCV 2022 Workshops- Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VIII, volume 13808 of Lecture Notes in Computer Science, pages 271 287. Springer, 2022.
[40] Güi Varol, Liliane Momeni, Samuel Albanic, Triantafyllos Afouras, and Andrew Zisserman. Scaling up sign spotting through sign language dictionaries. Int. J. Comput. Vis., 13066):4416-4439, 2022.
[41] Bowen Shi, Diane Brentari, Greg Shakhnarovich, and Karen Livescu. Fingerspelling detection in american sign language. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 4166 4175. Computer Vision Foundation / IEEE, 2021.
[42] K. R. Prajwal, Hannah Bull, Liliane Momeni, Samuel Albanie, Gul Varol, and Andrew Zisserman. Weakly- supervised fingerspelling recognition in british sign language videos. In 33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November 21-24, 2022, page 609. BMVA Press, 2022.
[43] Bowen Shi, Aurora Martinez Del Rio, Jonathan Keane, Jonathan Michaux, Diane Brentari, Greg Shakhnarovich, and Karen Livescu. American sign language fingerspelling recognition in the wild. In 2018 IEEE Spoken Language Technology Workshop, SLT 2018, Athens, Greece, December 18-21, 2018, pages 145-152. IEEE, 2018.
[44] Mark Borg and Kenneth P Camilleri. Sign language detection "in the wild" with recurrent neural net- works. In ICASSP 2019-2019 JEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). pages 1647-1641. JEEE, 2019.
[45] Amit Moryossef, Joannis Tsochantaridis, Roce Aharoni, Sarah Ebling, and Srini Narayanan. Real-time sign language detection using human pose estimation. In European Conference on Computer Vision, pages 237-248. Springer, 2020.
[46] Samuel Albanie, Gül Varol, Liliane Momeni, Triantafyllos Afouras, Andrew Brown, Chuhan Zhang, Ernesto Coto, Necati Cihan Camgöz, Ben Saunders, Abhishek Dutta, Neil Fox, Richard Bowden, Bencie Woll, and Andrew Zisserman. Sechear: Signer diarisation and a new dataset. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021, Toronto, ON, Canada, June 6-11. 2021, pages 2280 2284. IEEE, 2021.
13

---

[47] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
[48] Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81. 2004.
[49] Haroon Idrces, Amir R. Zamir, Yu- Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. The THUMOS challenge on action recognition for videos "in the wild". Comput. Vis. Image Underst. 155:1-23 2017.
[50] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: large- scale video benchmark for human activity understanding. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 961-970. IEEE Computer Society, 2015.
[51] Philippe Weinzaepfel, 7aid Harchaoui, and Cordelia Schmid. l.caning to track for spatio-temporal action localization. In 2015 IEEE International Conference Computer Vision. ICCV 2015, Santiago, Chile, December 7-13, 2015. pages 3164-3172. IEEE Computer Society, 2015.
[52] Chenyang Si, Ya Jing. Wei Wang. Liang Wang, and Tieniu Tan, Skeleton based action recognition with spatial reasoning and temporal stack learning. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu. and Yair Weiss, editors, Computer Vision- ECCV 2018 15th European Conference. Munich, Germany, September 8- 8-14, 2018 Proceedings, Part I, volume 1205 of Lecture Notes in Computer Science, pages 106-121, Springer, 2018.
[53] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton- based action recognition. In Sheila A. Mcllraith and Kilian Q. Weinberger, editors, Proceedings of the Thiroy-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018 pages 1444-7452, AAAI Press. 2018.
[54] Hezhen Hu, Weichao Zhao, Wengang Zhou, Yuechen Wang. and Houqiang Li. Signbert: Pre-training of hand- model-aware representation for sign language recognition. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 11067-
11076. IEEE, 2021.
[55] Gokul NC, Manideep Ladi, Sumit Negi, Prem Selvaraj. Pratyush Kumar, and Mitesh Khapra. Addressing resource scarcity across sign languages with multilingual pretraining and unified-vocabule datasets. In NeurlPS, 2022.
[56] Shiwei Gan, Yafeng Yin, Zhiwei Jiang, Lei Xie, and Sanglu Lu. Skeleton-aware neural sign language translation. In Heng Tao Shen, Yueting Zhuang, John R. Smith, Yang Yang. Pablo César, Florian Metze, and Balakrishnan Prabhakaran, editors. MM 21: ACM Multimedia Conference, Virtual Event, China. October 20- 24, 2021, pages 4353-4361. ACM, 2021.
[57] Yutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, and Stephen Lin. A simple multi modality transfer learning baseline for sign language translation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages
IEEE, 2022.
[58] João Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 4724-4733 IEEE Computer Society, 2017.
[59] Thang Luong, Hicu Pham, and Christopher D. Manning. Effective approaches to attention-based neu- ral machine translation. In Lluis Mårquez, Chris Callison Burch, Jian Su, Danicle Pighin, and Yuval Marton, editors, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Pro- cessing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 1412-1421. The Association for Computational Linguistics, 2015.
[60] Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[61] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton- based action recognition. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
14

---

[62] D. Crystal. A dictionary of linguistics and phonetics. 1997.
[63] Gabrielle Hodge. Patterns from signed language corpus: Clause-like units in Auslan (Australian sign language). PhD thesis, Macquarie University, 2014.
[64] Sameen Maruf, Fahimeh Saleh, and Gholamreza Haffari. A survey on document-level neural machine translation: Methods and evaluation. ACM Comput. Surv, 54(2):5::144:: 2022.
[65] Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. In-context examples selection for machine translation. CoRR, abs/2212.02437, 2022.
[66] Lei Shen, Haolan Zhan, Xin Shen, and Yang Feng. Learning to select context in hierarchical and global perspective for open-domain dialogue generation. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021, Toronto, ON, Canada, June 6-11, 2021, pages 7438-7442. IEEE, 2021.
[67] Lei Shen, Haolan Zhan, Xin Shen, Yonghao Song, and Xiaofang Zhao. Text is NOT enough: Integrating visual impressions into open-domain dialogue generation. Heng Tao Shen, Yueting Zhuang, John Smith, Yang Yang, Pablo César, Florian Metze, and Balakrishnan Prabhakaran, editors, MM '21: ACM Multimedia Conference, Virtual Event, China, October 20 24, 2021, pages 02287-4290 ACM, 2021.
[68] Dongxu Li, Xin Yu, Chenchen Xu, Lars Petersson, and Hongdong Li. Transferring cross-domain knowledge for video sign language recognition. In 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 6204-6213 Computer Vision Foundation IEEE, 2020.
[69] Roman Töngi. Application of transfer learning to sign language recognition using inflated 3d deep convolutional neural network. CoRR, abs/2103.00 101.111111..111 2021.
[70] Md. Monirul Islam, Md. Rasel Uddin, Md. Nasim A AKhtar, and K.M. Rafiqul Alam. Recognizing multiclass static sign language words for deaf and dumb people of bangladesh based on transfer learning techniques. Informatiss in Medicine Unlocked, 33:101077, 2022.
15


Reviews:
[{"review_id": "7hWCAVybvj", "reviewer": "Reviewer_fC6g", "summary": "", "questions": "", "limitations": "W1. The authors should try to present better baselines where the performance on the dataset is more accurate. Currently, the scores in the case of SLT are pretty low, and thus it leaves some doubt about whether the dataset is correctly cleaned or not.\n\nW2. The authors can present some baseline results on Sign Language Production too on their dataset.\n\nW3. The authors can present some ablation studies and try out different representations of the visual stream to get a better understanding of their dataset.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "S1. The dataset is large and consists of significant variation among the subjects covered. \n\nS2. The dataset has multiple levels of annotations, including fingerspellings. This is a great initiative.\n\nS3. The total number of signers covered is also decently large, which adds to the variation present in the dataset.\n\nS4. The dataset collection strategy and cleaning strategies are presented well.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "BB7dTBOgAr", "reviewer": "Reviewer_MEP1", "summary": "", "questions": "", "limitations": "The authors have adequately discussed limitations.", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. It is the first large-scale Australian sign language dataset, which is not only helpful to the research on Auslan but also to the community of sign language understanding.\n2. The vocabulary size is large, and the background and topics are diverse.\n3. A high resolution of 1080p.\n4. The dataset can support multiple sign language understanding tasks, e.g., sign language translation, sign spotting, and isolated sign recognition.\n5. The dataset consists of conversational sign videos, which are deficient in existing sign language datasets.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Tsnfrxis23", "reviewer": "Reviewer_4ztv", "summary": "", "questions": "", "limitations": "In the section 6 of this paper, the authors discuss some special experimental phenomena and the dataset's limitations from four points, which I think is sufficient.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "Overall, the paper is well written and understandable. The dataset is well annotated and usable for multiple tasks. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "3PjB5kwSoZ", "reviewer": "Reviewer_vHjw", "summary": "", "questions": "", "limitations": "Not really.", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "In general, the paper is well written. The design of the dataset is well thought. The major strengths of the paper are:\n1. obviously a new relatively large dataset for Auslan as there are not many large SL datasets even in other languages. Here, the dataset is \"large\" in terms of number of signers which 67 and vocabulary which is 14K.\n2. it covers both in-the-wild scenarios and more predictable scenarios such as news report, weather forecast and documentaries with possibly diverse backgrounds. Few current SLT datasets contain in-the-wild scenarios.\n3. it covers situations that there can be more than 1 signer in a scene, and thus SLT will require signer tracking and perhaps requires dealing with signer interference. Again, few current SLT datasets contain multi-signer situations.\n4. multi-grained annotations are provided at the sentence level for the whole dataset, and at the gloss level and fingerspelling for some subsets of the dataset.\n5. train/dev/test data splits are given and benchmark evaluation results are given using publicly available recent SLT models so that future research on the new dataset can be properly gauged.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Prs3L2SrXn", "reviewer": "Reviewer_qbZ3", "summary": "", "questions": "", "limitations": " The authors adequately addressed the limitations of their work in Section 6 Discussion and Limitation.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The proposed dataset is with large-scale, fully annotated, and can be used for various tasks.\n\nFrom the experimental results, it can be seen that Auslan-Daily is a highly challenging SLT dataset. \nThis makes the proposed dataset promising to contribute to the development of Auslan, even the advancement of sign languages worldwide in a broader context. \n\nAll datasets and benchmarks have been available to the public.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: All the reviewers express positive opinions on the paper, which creates a new sign language dataset on Australian Sign Language with benchmark results provided. The dataset is well annotated (by experts at various level of details) and usable for multiple tasks. Also one leaderboard is established for promoting Auslan SLT research.
Therefore, the paper should be accepted.
Decision: Accept (Poster)