BANZ-FS: BANZSL Fingerspelling Dataset
Authors: Xin Shen
Yan Ke
Xinyu Wang
Xin Yu
Venue: ICLR 2026 Poster (2026)
Keywords: Sign Language
BANZSL
Fingerspelling
Primary area: datasets and benchmarks

Abstract: Fingerspelling plays a vital role in sign languages, particularly for conveying names, technical terms, and words not found in the standard lexicon. However, evaluation of two-handed fingerspelling detection and recognition is rarely addressed in existing sign language datasets—particularly for BANZSL (British, Australian, and New Zealand Sign Language), which share a common two-handed manual alphabet. To bridge this gap, we curate a large-scale dataset, dubbed BANZ-FS, focused on BANZSL fingerspelling in both controlled and real-world environments. Our dataset is compiled from three distinct sources: (1) live sign language interpretation in news broadcasts, (2) controlled laboratory recordings, and (3) diary vlogs from online platforms and social media. This composition enables BANZ-FS to capture variations in signing tempos and fluency across diverse signers and contents. Each instance in BANZ-FS is carefully annotated with multi-level alignment: video ↔ subtitles, video ↔ fingerspelled letters, and video ↔ target lexicons. In total, BANZ-FS includes over 35,000 video-aligned fingerspelling instances. Importantly, BANZ-FS highlights the unique linguistic and visual challenges posed by two-handed fingerspelling, including handshape coarticulation, self-occlusion, intra-letter variation, and rapid inter-letter transitions. We benchmark state-of-the-art models on the key tasks, including fingerspelling detection, isolated fingerspelling recognition, and fingerspelling recognition in context. Experimental results show that BANZ-FS presents substantial challenges while offering rich opportunities for BANZSL understanding and broader sign language technology. The dataset and benchmarks are available at BANZ-FS.

Published as a conference paper at ICLR 2026
BANZ-FS: BANZSL FINGERSPELLING DATASET
Xin Shen', Yan Ke', Xinyu Wang', Xin Yu?* Thee University of Queensland Australian Institute for Machine Learning, Adelaide University x. s @uqconnect edu.au
Y
Zero
Onl
Two
Three
Four
Five
Six
Sevin
Eight
Nüse
ABC News with
Lab Recordings
Web Videos
Figure 1: Overview of BANZ-FS dataset sources and coverage. The figure shows fingerspelling
instances for all 26 letters (A-Z) and 10 digits (0-9) from three sources: ABC News, Lab Recordings, and Web Videos. News instances reflect formal, live interpretations by professional signers. Lab recordings offer clean, controlled settings ideal for analysis. Web videos capture diverse, in-the-wild signing styles across various environments.
ABSTRACT
Fingerspelling plays a vital role in sign languages, particularly for conveying names, technical terms, and words not found in the standard lexicon. However, evaluation of two-handed fingerspelling detection and recognition is rarely addressed in existing sign language datasets caausstt-pprtticlaatt for BANZSL (British, Australian, and New Zealand Sign Language), which share common two-handed manual alphabet. To bridge this gap, we curate large-scale dataset, dubbed BANZ-FS, focused on BANZSL fingerspelling dataset is compiled from three distinct sources: (1) live sign language interpretation in news broadcasts, (2) controlled laboratory recordings, and (3) diary vlogs from online platforms and social media. This composition enables BANZ-FS capture variations in signing tempos and fluency across diverse signers and contents. Each instance in BANZ-FS is carefully annotated with multi-level alignment: video ++ subtitles, video ++ fingerspelled letters, and video + target lexicons. In total, BANZ-FS includes over 35,000 video-aligned fingerspelling instances. Importantly, BANZ-FS highlights the unique linguistic and visual challenges posed by two- handed fingerspelling. including handshape coarticulation, self-occlusion, intra- letter variation, and rapid inter-letter transitions. We benchmark state-of-the-art models on the key tasks, including fingerspelling detection, isolated fingerspelling recognition, and fingerspelling recognition in context. Experimental results show that BANZ-FS presents substantial challenges while offering rich opportunities for BANZSL understanding and broader sign language technology. The dataset and benchmarks are available at O BANZ-FS.
both controlled and real-world environments. Our
"Corresponding author.

---

Published as a conference paper at ICLR 2026
Exacl Match
SaF-ommake Dihin
goork grin boddy leady Maken noriig pe piaceimmse
Spdlig
32%
RP%
Arrgga
Abbreviation
Spelling Error
krdoeenus beading
Acronym m Use
Self-Correction
The Werme Reklip boste Grote Spley (CWSs
Challonge: Self-Ocelasion
Challenge: Intro-I Varlation
Challenge: Transitions
GENDER
Figure 2: Overview of typical fingerspelling phenomena and visual challenges captured by the BANZ-FS dataset. The pie chart (top-left) illustrates the proportion different fingerspelling phenomena annotated within the dataset. Representative examples below highlight diverse wool-world cases, such as exact matches ("equity"), lexical abbreviations ("equipment" + "EQ"), spelling errors ("Maguire" misspelled "Maquire"), acronym ("Greater Western Sydney" "GWS"), and inline corrections ("miimiles" corrected to "miles"). The bottom row (green boxes) highlights key visual challenges specific two-handed fingerspelling systems, such self-occlusion, intra- letter variation, and rapid inter-letter transitions, further underscoring the complexity of accurate fingerspelling recognition and translation BANZSL.
INTRODUCTION
Sign languages (SL) are natural languages that serve as primary modes communication for Deaf and hard-of-hearing individuals, enabling rich self-expression and full participation in society. Like spoken languages, sign languages possess their own grammars and lexicons, and they vary widely across regions-evee in places that share common spoken language. For example, American Sign Language (ASL) (Duarte et al., 2021; Shi et al., 2022; Uthus et al., 2023; Tanzer & Zhang, 2024; Li al.. 2020a) and Australian Sign Language (Auslan) (Shen et al., 2023; 2024a; Sheng al., 2024) are linguistically distinct, each with unique phonological, lexical, and syntactic features. To bridge communication between Deaf and hearing communities, sign language translation (SLT) (Shen et al., 2025a) systems have been developed to automatically translate sign videos into spoken languages.
Among sign languages, fingerspelling (FS), the manual representation of alphabets and numbers. plays a critical role in SLT (Shen et al., 2023; Tanzer, 2024b; Georg et al., 2024; Kim et al., 2017; Papadimitriou al., 2024; Sheng et al., 2026: Shen et al., 2024b), particularly for expressing proper nouns, technical terms, and items not represented in the standard sign lexicon. Unlike single-handed systems, such as ASL (Shi et al., 2021; Padden., 1998; Tanzer, 2024a), BANZSL employs a distinc-
G BANZSL refers to sign language family which encompasses BSL, Auslan and NZSL. These sign languages he considered as dialects of BANZSI. due to their shared manual alphabet, grammatical structure, and substantial lexical overlap.
2

---

Published as a conference paper at ICLR 2026
tive two-handed fingerspelling system. This two-handed system introduces significant challenges for machine translation (see the bottom row of Figure 2), such as frequent self-occlusion. high intra-letter variations, and rapid handshape transitions. Hence, accurate recognition of fingerspelling is crucial, as it frequently conveys essential semantic content, such as named entities, numerical data, and domain-specific vocabulary that lack conventional sign equivalents.
Despite significant progress in sign language research (Huang et al., 2018; Zhou et al., 2021; Shi al., 2022; Duarte et al., 2021; Camgöz et al., 2018; Uthus et al., 2023; Tanzer & Zhang, 2024; Shen et al., 2023; 2024a), most publicly available datasets focus on single-handed fingerspelling understanding task, such as those used in ASL (Shi et al., 2021; Padden., 1998: Tanzer, 2024a: Georg et al.. 2024) and GSL (Papadimitriou et al.. 2024), leaving the two-handed fingerspelling system of BANZSI (Shen et al., 2024a; Prajwal et al., 2022) comparatively underexplored. Moreover, existing datasets often lack the scale and linguistic realism required for fingerspelling research. In particular, they rarely capture naturally occurring phenomena, such as spelling errors, lexical abbreviations, acronyms, and inline corrections, which are commonly encountered in practical scenarios. This highlights a critical gap: the need for a large-scale, real-world BANZSL fingerspelling dataset to facilitate the study BSL, Auslan and NZSL.
To address this gap, we introduce BANZ-FS, a large-scale dataset dedicated to BANZSL finger- spelling, collected from both real-wort world and controlled environments. As shown in Figure I, BANZ- FS integrates multiple sources to reflect diverse and authentic usage scenarios: (1) professional live Auslan interpretations from ABC News with Auslan broadcasts (capturing formal, high-register discourse); (2) controlled laboratory recordings (offering clean, high-quality reference data); and (3) user-generated vlog content from online platforms and social media (representing casual, daily communication). This diverse composition allows BANZ-FS to capture a broad spectrum of signing tempos and registers, from formal broadcast interpretation everyday interaction.
Specifically. BANZ-FS comprises more than 35,000 aligned fingerspelling instances, During an- notating fingerspelling, we additionally align 40 hours of Auslan news footage, which not only substantially extends the prior benchmark Auslan-Dail News (Shen et al., 2023) but also allows us to investigate recognition accuracy of fingerspelling within contexts. Our annotation protocol includes fine-grained alignment across video ++ subtitles, video ++ fingerspelled letters, and video ++ target lexicons. As illustrated in Figure 2, we explicitly annotate and categorize key linguistic phenomena prevalent in fingerspelling, including abbreviations, acronyms, misspellings, and inline corrections. Furthermore, our proposed dataset captures the visual and articulatory complexities inherent in two-handed fingerspelling systems, underscoring the challenges of accurate fingerspelling recognition in BANZSL..
With BANZ-FS, we investigate range of fingerspelling-relatee tasks, including fingerspelling detec- tion, isolated fingerspelling recognition and fingerspelling recognition in context. We benchmark publicly available state-of-the-ard models on each task and then report the performance using corre- sponding evaluation metrics. Experimental results demonstrate that the complexity and realism of BANZ-FS pose significant challenge to existing methods, highlighting its potential drive progress in two-handed fingerspelling understanding. Overall, the contributions of this work are threefold:
We curate the first large-scale fingerspelling dataset specifically for the BANZSL system, capturing real-world complexities across diverse contexts.
We provide comprehensive multi-level annotations to support fingerspelling-rehe related tasks, includ- ing fingerspelling detection, isolated fingerspelling recognition, and fingerspelling recognition within continuous sign language sentences.
We benchmark state-of- the-art methods to highlight the unique challenges posed by BANZ-FS. and establish an ideal platform to evaluate fingerspelling recognition capabilities.
2 RELATED WORK
2.1 FINGERSPELLING DATASETS
Early research in sign language recognition primarily addressed isolated sign language recogni- tion (Shen et al., 2024a; Li et al., 2020a; Desai et al., 2023; Starner et al., 2023; Shen et al., 2025c;b), but recent trends have progressively emphasized continuous sign language recognition (Chen et al.,
3

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of the proposed BANZ-FS dataset with existing datasets widely used for "FSD ", and "IFSR" represent Fingerspelling Recogni-
fingerspelling-relate tasks. tion in Context, Fingerspelling Detection and Isolated Fingerspelling Recognition, respectively.
Dalave
SL
Video
Vocal.
FS Segs
4Signer
Source
FSR-Cuntest
FSD
IFSR
Flour-ASI 2024cch)
ASI.
1.7K
1db
SL-ReDu-Fingg (Pirddimitrion 2024)
GSL
1.5K
24
1.5K
21
Lab
BOBSL-ES (Prajoaa 2012)
BSL.
Web
ChicggoFVVid (Kim ol., 2017)
AST.
4K
26
4K
4
Lab
FShoord (Gooeg et al., 2024)
ASL
15IK
147
Smartphose
Chill (Shi al. 2018)
ASI.
7K
76
7K
160
Wib
ChicaooS Wilke (Shi al., 2019)
ASL
55K
26
55K
260
Web
Auslan Daily Comam (Shen et al. 2023)
Auslan
14K
3K
IK
49
Teb
Ausian-Daily News (Sben ol., 2023)
Audlan
11K
13K
18
TV&Wab
BANZ-FS (Ours)
BANZSL
35K
36
35K
116
Leb
2022c; Min et al., 2021) and fingerspelling recognition (Shen al., 2023; Kim et al., 2017; Shi et al., 2018; 2019; Fayyazsanavi et al., 2024). Despite growing interest, as shown in Table I, existing American Sign Language (ASL) and single-handed
fingerspelling datasets largely concentrate signing systems (Papadimitriou et al., 2024), such as ChicagoFSWild+ (Shi et al., 2019). ChicagoF- SWild (Shi al., 2018), FSBoard (Georg al., 2024) and Fleurs-ASL-FSS (Tanzer, 2024a:b). Among these, FSBoard (Georg et al., 2024) is currently the largest dataset, containing approximately 151K fingerspelling sequences collected from 147 signers, captured uniquely via smartphone in single- handed manner. However, FSBoard is limited to recognition tasks due to the absence of segment-level annotations, which restricts its application in fingerspelling detection. Datasets capturing finger- spelling "in-the- thtdd""""d", such as ChicagoFSWild (Shi et al., 2018) and ChicagoF'S Wild+ (Shi et al., 2019), have improved realism by sourcing content from online platforms, encompassing diverse signer appearances and envirommental variations. It is worth noting that Fleurs-ASL-rS (Tanzer, 2024a;b) only provides sentence-level annotations indicating the presence of fingerspelling, without corresponding temporal boundaries. As result, it can only be used for fingerspelling in context task, but not for fingerspelling detection or localization.
Regarding BANZSL-related resources, prior datasets such Auslan-Daily (Shen et al., 2023) and BOBSL-FS (Prajwal et al., 2022) contain only a limited number of fingerspelling instances, and consequently provide insufficient support for developing and evaluating fingerspelling-spedde tasks. Our proposed BANZ-FS dataset addresses these limitations by introducing over 35K aligned fingerspelling instances with comprehensive annotations suitable for fingerspelling detection and recognition tasks. In parallel with the annotation of BANZ-FS, we also engaged Auslan experts to extend the Auslan-Daily (Shen et al., 2023) News subset through additional annotation and alignment, resulting in threefold increase in scale.
2.2 FINGERSPELLING DETECTION AND RECOGNITION METHODS
Early fingerspelling detection (Shi al., 2021) methods utilized visual features such optical flow predefined hand keypoints (Yang & Lee, 2010; Yanovich et al., 2016). However, such methods have primarily been evaluated in controlled environments, with limited effectiveness in unconstrained, real- world settings due to unreliable pose estimations (Tsechpenakis et al., 2006b;a). Recent approaches have favored recurrent neural networks (RNNs) (Luong et al., 2015) and transformer-based (Vaswani et al., 2017) architectures to enhance temporal modeling capabilities and robustness (Li et al., 2020c; Moryossef et al., 2020; Zuo et al., 2023; Pugeault & Bowden, 2011; Li al., 2020a).
For fingerspelling recognition, convolutional neural networks (CNNs) combined with RNNs or Long Short-Term Memory (LSTTM) networks have been widely employed (Schuster & Paliwal, 1997: Shi et al., 2021). Transformer-based models have recently emerged as powerful alternatives, effectively capturing long-range temporal dependencies and contextual information crucial for recognizing fingerspelled sequences (Boháček & Hrúz, 2022; Hu et al., 2024; Prajwal et al., 2022). Several studies have explored multimodal fusion approaches, integrating RGB frames, optical flow, and pose estimation features enhance recognition accuracy (Jiang et al., 2021a;b; Zuo al., 2023). While significant progress has been made, challenges remain, particularly regarding ambiguity due to similar handshapes across distinct letters and digits. Methods like those proposed in (Li et al., 2023;
4

---

Published as a conference paper at ICLR 2026
Fayyazsanavi et al., 2024) modify Transformer encoder-decoder architectures explicitly to mitigate ambiguities arising from visually similar fingerspelling representations.
BANZ-FS DATASET
In this section, we describe data collection for web-based fingerspelling data, as well as the recording protocol for the lab-collected instances?. We provide detailed statistics of the BANZ-FS dataset.
3.1 COLLECTION, CLEANING AND LABELLING PROCEDURE FOR WEB DATA
Collection. "ABC News with Auslan" and Youeube sources are open sources. Beginning in 2022, "ABC News with Auslan" has provided weekly broadcasts covering key domestic and international news events, as well as weather forecasts. It is an ongoing program, and previous work (Shen et al., 2023) aligned 45 videos collected up to May 2023, along with small number of fingerspelling annotations. In this work, we extend the collection by acquiring an additional 80 videos spanning from May 2023 to April 2025. These broadcasts feature live sign language translation (simultaneous interpretation) by Auslan experts, intended for deaf and hard-of-hearing viewers. News content inherently includes a rich set of fingerspelling scenarios, such as personal names, place names, organization names, phone numbers, and other proper nouns, making it an ideal source for studying fingerspelling phenomena. To further diversify our dataset, we also include several high-quality publicly available documentaries and educational videos interpreted with BSL and NZSL, primarily sourced from Yubeub.. These videos typically feature daily conversations, learning activities, or introductions to specific topies. All online videos are included only via their official public URLs, without redistribution of any copyrighted material, as detailed in the Ethics Statement.
Cleaning. All original videos are accompanied by standard English dubbing and subtitles. We retrieve subtitles for each complete video, formatted as "/Start Time] subtitle [End TimeJ", with timing aligned the spoken dubbing. Following the subtitle cleaning strategy proposed in Auslan- Daily (Shen et al., 2023), we perform simple cleaning operations: merge fragments ending in commas with their subsequent lines, split overlapping subtitles into separate sentences, and discard entries with only non-semantic fillers. As a result, we obtain approximately 30K complete and cleaned subtitles requiring further alignment with fingerspelling segments.
Labelling. To annotate fingerspelling instances, we invited experienced Auslan experts to assist in the labelling process. In particular, for news videos, we additionally perform temporal alignment to ensure segment-level consistency. For each video, we first employ AlphaPose (Fang al., 2022; 2017; et al., 2019) to track all individuals in the scene. The annotation process proceeds in several steps: (1) verify and refine video-subtitle alignment; (2) identify the signer ID based on pose trajectories. AlphaPose tracks all people in the scene and generates adentity-consister pose sequences. Annotators manually review these trajectories and select the signer based on spatial position and continuous signing motion; (3) if fingerspelling is present, annotate the corresponding temporal segment; and (4) retrieve the associated target lexicon from the subtitle, if exists. To guarantee the annotation quality of our dataset, we conduct a cross-check verification process during each data labelling procedure stage. Specifically, we employed "recognition-basee verification" protocol where each Auslan annotator (examiner) cross cross-checks random 5% sample of clips provided by another annotator. practice, we observed high consistency, with approximately 95% of the sampled batches passing verification in the first round. If the examiner finds more than 10% of annotated videos have obvious errors, the entire batch is rejected, and third annotator is invited to review and correct the annotations. Through the collaborative efforts of five Auslan experts and five annotators, we complete all annotations with approximately 500 work hours. Overall, our dataset contains the following annotations: (1) temporal boundaries of sign video clips; (2) temporal boundaries of fingerspellings; (3) lexical forms of fingerspellings; and (4) English transcriptions. These annotations can be further investigated for fingerspelling- haappelllin---llteeddde related tasks.
"The original Auslan News data is provided by Auslan-Daily (Shen al., 2023). Both the web-hased and lah-collected portions of dataset will he released under the Creative Commons BY-NC-SA 4.0 license O
5

---

Published as a conference paper at ICLR 2026
Table Key statistics of the BANZ-FS dataset across three data sources: ABC News with Auslan, Lab Capture, and YouTube. OOS lout-of-trainigg Signers) and OOFS (out-of-training FS strings) are signers and FS sequences that never appear in the training set, while FS Singletons occur only once in training.
Dota Source
ABC News with Auslan
Capture
YouTube
Source Language
Domiin//opie
Neus
Daily Used
Communkcation Variou
Video Resolation
7200x7208225
1930 10800 30/1280 7206600
Split
Train
Dev
Test
Train
Dev
Test
Train
Dev
Test
Total
Video Segmens
18,894
1,608
1,896
6,828
1,952
1,952
1,498
300
300
35,028
Signers
22
52
116
Avg. FS Segments
2,4
2.0
2.5
1.0
1.8
1.95
Tot. FS Chass
144,090
12.383
15,217
11,748
3,360
3,350
12,034
2,154
2,446
207,292
PS Speed (cham/sencund)
4.59
4.39
4.17
1.30
1,31
1.30
1.99
1.45
1.68
3.41
ODFS:
FS Singletons
1,201
1,191
8000
(b)
Numbez of detoption clige
100000
(c)
Number detectiion
7000
8000
5000
Deaf
Male
5000
Individual
2000
1000
38.5 ())
(1.5.40)
Number of segmeats per dotection slip
duration I'S desection clip (scconds)
Number of occurrencss
(d)
1000
6
D
W
P
11
Unique charactens
Figure 3: (a) Distribution of the number of fingerspelling (FS) segments per clip. (b) Distribution of FS clip durations. (c) Distribution of signer demographics categorized by Auslan proficiency gender. (d) Character frequency distribution across all FS clips.
3.2 COLLECTION FOR LAB DATA
To complement web data, followed by (Shen al., 2024a; Ying et al., 2024), record lab- controlled videos using a multi-camera RGB-D setup. The recording studio is equipped with a green screen and includes three Kinect-V2 cameras positioned at left-front, front, and right-front angles, along with centrally placed RealSense camera. We invite participants with diverse Auslan experience, including deaf individuals, Auslan experts, and sign language learners. Participants instructed to perform fireqeenly used fingerspelling words and expressions commonly encountered in daily communication contexts. Each sign instance is verified by at least one expert ensure expression accuracy, while the inclusion of volunteers enhances signer diversity and realism. This setup facilitates the study of caosea robustness and supports high-quality benchmarking under controlled conditions. Full details regarding ethical compliance, participant consent, and fair compensation are provided in the Section Ethics Statement.
'In this dataset, the label "Auslan Expert" refers specifically to hearing professional interpreters, whereas "Deaf individual" denotes Deaf participants. These labels reflect only the demographic grouping used in this study and do not imply that the categories mutually exclusive.

---

Published as a conference paper at ICLR 2026
Ground Truth
IFSR
C
E ECETLVITNE
FSD
FSD-R
8
CA
R
FSR-Context
Hello, welcome ABC News.
Celina.
Croand Trakk Hundlry Pediow/
01:08.40
01:0-30
01.00
01:3400
01:1000
01 01.00.00
01:10.00
01:1220
01:1.60
01:12.00
Figure 4: Overview of fingerspelling-rrlated tasks in our BANZ-FS dataset.
3.3 DATA STATISTICS
As shown in Table 2, we present key statistics of BANZ-FS to highlight the diversity and complexity of the dataset. BANZ-FS consists over 35,000 annotated video segments sourced from news broadcasts, lab recordings, and online videos, covering 1 116 unique signers. The dataset is split into training, development, and test sets to facilitate fair evaluation fingerspellingreehlee rellaieg tasks. We segment each video by applying a 10-second sliding window around any detected FS segment. As a result, each detection clip may contain multiple FS instances. As shown in Figure 3, most clips contain only 1-2 FS segments and last less than 1.5 seconds, indicating that F$ often embedded briefly within continuous signing. Furthermore, the signer population includes balanced mix of Auslan experts, deaf individuals, and volunteers, offering wide range of signing styles and linguistic competence. The FS character distribution reveals a long-tail pattern: common letters such as A, E, and N appear frequently, while rare characters (e.g., numerals and less frequent letters) occur sparsely. This imbalance poses additional challenges for generalization and open- coca--oy recognition, especially in low-resource conditions. addition, we report the number of out-of-training (OOFS) and FS singletons in Table 2, which quantify the presence of unseen or rare FS sequences and further reflect the open-set nature the task. To evaluate generalization to unseen users, we explicitly report the number of out-of-traiie oft-of-training signers in Table which refers signers that never appear in the training set. Further statistics and discussions can be found in Appendix Section B, Section C.6, Section C.7 and Section H.1.
strings
OVERVIEW OF BANZ-FS TASKS AND EVALUATION METRICS
In this section, we provide an overview the BANZ-FS benchmark tasks and their corresponding evaluation metrics, as illustrated in Figure 4.
Isolated Fingerspelling Recognition (IFSR) (Shi et al., 2018; 2019): Given a segmented fin- gerspelling clip fs (T,,, Ty.). the goal of IFSR is transcribe into the corresponding letter sequence L (l,...,4 ). Evaluation Metric for IFSR Letter Accuracy Defined as 1,"
1- EdinDistance L , where L* is the ground-truth letter sequence and É is the predicted sequence.
This edit-distance-based metric captures the correctness of the full predicted sequence, accounting for insertions, deletions, and substitutions.
Fingerspelling Detection (FSD) (Shi et al., 2021): Given an untrimmed sign language video V= (1, /2...,,T7 /T) with T frames, the goal of FSD is to identify temporal segments (fa, fe) that localize fingerspelling intervals within V. Each predicted segment corresponds to a time span where fingerspelling occurs. The evaluation metric for FSD is AP@ IoU, specifically AP@ IoU It represents the Average Precision calculated based on a temporal Intersection-over Union (IoU) threshold of 0.5 between predicted and ground-truth segments.
7

---

Published as a conference paper at ICLR 2026
Fingerspelling Detection followed by Recognition (FSD-R) (Shi et al., 2021): FSD-R is tto-stage approach where an FSD model first predicts temporal segments from an untrimmed sign language video V. and each predicted segment is subsequently processed by fingerspelling recognizer (IFSR) to generate the corresponding letter sequence. The evaluation metric for FSD-R is AP@ Acc, specifically AP@ Acc It represents the Average Precision where predicted segment is considered a True Positive only if the character-level accuracy of the downstream recognizer exceeds a threshold of 50%.
Fingerspelling Recognition in Context (FSR-Context) (Tanzer, 2024b); Given a full sentence-level sign language video V and its predicted spoken language translation T', the task of FSR-Context is to evaluate how accurately the model transcribes fingerspelled terms embedded within the sentence. Specifically, fingerspelled spans annotated in the video are aligned with corresponding spans in the predicted translation, and character-level accuracy is measured. Evaluation Metrics for FSR-Context is Letter Accuracy.
5
BANZ-FS BENCHMARK
5.1 VIDEO REPRESENTATION
Pose-based video feature representation: Pose-based representations are robust against background clutters, lighting conditions, and occlusions, while explicitly depicting human hand and limb move- ments (Weinzaepfel et al., 2015; Si et al., 2018; Yan et al., 2018). Several recent studies exploit pose information and achieve state-of-the-art performance in fingerspelling-relan Fayyazsanavi et al., 2024; Moryossef et al., 2023). Hence, DWPose (Yang et al., 2023) as video features to provide benchmark results.
tasks (Tanzer, 2024b; the key points extracted from
RGB-based video feature representation: Several models directly extract features from sign videos, such as CON-RNN-HMI network (Camgöz et al., 2018), S3D (Chen et al., 2022b) and 13D (Carreira & Zisserman, 2017). In the works (Albanie et al., 2020; Li et al., 2020a;b), I3D is used for sign video representation. To better adapt to SL dataset and capture the spatio-tempore information of signs, inspired by (Li et al., 2020b), we finetune I3D on word-level sign language recognition dataset (Shen et al., 2024a) and extract sign video features with different window widths and strides. Recent studies have shown that feeding raw RGB video directly into end-to-end models yields strong performance on various sign language tasks (Chen et al., 2022c; Min et al., 2021). Following this trend, we adopt raw video frames as input for our end-to-end models.
5.2 BENCHMARK RESULTS
In this section, we provide benchmark results of isolated fingerspelling recognition (IFSR), finger- spelling detection (FSD), fingerspelling detection followed by recognition (FSD-R) and fingerspelling recognition in context (FSR-Contexl) tasks on BANZ-FS.
Isolated Fingerspelling Recognition (IFSR): Table presents a cross-domain evaluation of state-of- the-art models on the IFSR task. The models trained separately on four different subsets-News, Lab, Web, and the union of all (Full)-and evaluated across each domain. This setup allows for detailed analysis of both in-domain performance (training and testing the same source) and out-of-domen generalization (training on one domain, testing on another). Among all the models, HandReader Korotaev et al. (2025) consistently demonstrates the strongest cross-domain robustness. When trained on the Full set, it achieves the highest overall accuracy (75.4%), and outperforms other models by large margin on challenging domains such as Web (40.2%). Notably, HandReader (Ko- rotaev et al., 2025) trained on News alone generalizes well to Lab data (48.5%) and achieves the best News-to-News performance (68.3%) reflecting the benefit of its multimodal (RGB + 3D pose) architecture in capturing signer-invariant features. In contrast, Iterative-Att (Shi et al., 2019) show limited generalization ability. The performance drops significantly when evaluated on unseen do- mains, especially on Web videos where visual variability is high. MiCT-RANet (Mahoudeau, 2020) and TS-FS-Reg (Chen et al., 2022c) achieve stronger generalization than early models, particularly when trained on the Full set, with accuracies of 68.6% and 69.7% respectively. TS-FS-Reg (Chen et al., 2022c) benefits from dual modality inputs, which help mitigate overfitting to domain- sppciffc appearance. Overall, the results reveal that while most models perform well the domain they
8

---

Published as a conference paper at ICLR 2026
Table 3: Performance comparison (Let- ter Accuracy) on Fingerspelling Recog- nition (IFSR) across three data domains: News, Lab, and Web. "Full" refer to the combined dataset.
Table 4: Performance comparison on Fingerspelling De- tection (FSD) and FSD-R tasks across three data domains. Average Precision (AP) at 0.5 IoU threshold for FSD (AP@ (oUn.s)) and AP at 0.5 recognition accuracy thresh- old for FSD-R (AP@ Acco.5).
Method
Train
Latter Accumecy (9)
Method
Train
AP810U
APG Acti.
Nows
Full
News
Lah
Full
News
Lib
Full
Full
58.6
Full
31.1
560
15.5
39.
14.8
26,9
Itenstve-AIt (Shi et al., 2019)
News
46.7
Bi-LSTM (Huung et ol., 2015)
News
43.4
29.0
35.2
21.6
17.8
Lab
30.2
87.7
30.3
57.3
Lab
10.0
72.9
15.1
40.0
5.0
71.1
7.5
36.3
Wib
21.0
35.8
36.3
29.1
Wibs
13.1
37.
26.1
25.5
6.5
13.2
13,3
10.1
Pull
56.4
81.8
660.1
68.6
Full
35.2
64.6
32.2
48.8
19.6
43.3
15.6
30.5
MICT-RANet (Mahoudeau, 2020)
News
53.3
Modified R-C3D (Nu 2017)
News
47.9
33.0
25.0
29.2
23.2
162
18.9
Lab
60.9
Lab
11.9
75.5
42.2
6.1
77.3
10.8
39.9
Web
22.9
38.6
42.5
31.7
Wab
15.9
40.9
30.1
28.7
7.0
14.5
14.4
11.1
Full
57.2
82.9
62.4
69.7
Full
41.3
69.0
37.4
54.1
23.5
64.0
22.1
42.5
TS PS Rey
News
59.2
54.3
50.4
56.2
TS F8 Det
News
53.6
41.8
34,0
46.6
29.3
26.5
19.3
27.3
(Chen al., 20022g
Lab
64.1
(Chon al., 2M22e)
79.6
44.4
7.4
77.9
40.7
Web
31.6
44.3
22.7
9.9
23.9
17.4
Full
74.7
48.6
79.7
25.4
68.7
No-PoseNet
News
66.2
52.9
48.2
58.6
MT-FS-Dat (Shi et al.. 2021)
News
57.8
44.3
34.2
49.7
32.9
29.1
20.9
30.2
(Fayyazsanavi al., 2024)
Lab
36.2
92.8
29,4
62.3
Lab
18.1
82.3
19.0
48.4
10.4
81.7
13.7
44.2
Web
26.5
47.2
51.4
38.0
Web
20.8
44.2
41.5
33.3
10.1
25.5
26.9
18.6
Fell
64.4
86.7
71.8
75.4
Full
53.9
82.7
47.3
66.9
33.7
76.3
30.2
53.5
HandReader
News
60.8
S1-Seg (Maryossaf 2023)
45.
35.9
50.3
(Kocotaev al.,, 2025)
Lab
37.1
93.1
63.1
Lab
86.3
50.7
10.2
85.0
45.7
Wob
29.8
48.1
55.0
40.2
Web
22.5
46.1
40.4
34,9
11.8
28.2
29.9
20.8
are trained on en-espccially Lab, which offers controlled recording eralization remains significant challenge. FS-PoseNet (Fayyazsanavi al., 2024) stands out by consistently maintaining strong performance across all domains, making it particularly promising for deployment in real-world scenarios with diverse video sources.
gen-
Fingerspelling Detection (FSD): The FSD results in Table show notable variation across domains. SL-Seg (Moryossef et al., 2023) achieves the best overall detection performance, particularly excelling on the Web domain (47.3%), where other models generally struggle. This suggests that frame-level BIO tagging with bose- cues provides more robust temporal boundary modeling than proposal- based or prrression-basede methods. While MT-FS-Det (Shi et al., 2021) and TS-FS-De (Chen et al., 2022c) perform competitively on News and Lab subsets, their generalization to noisy Web data limited. In contrast, performance of earlier approaches, such as Modified R-C3D (Xu et al., 2017) and Bi-LSTM CTC (Huang et al., 2015), is much inferior, highlighting the importance of structured temporal representations and boundary-aware learning for reliable fingerspelling localization.
Fingerspelling Detection followed by Recognition (FSD-R): The FSD-R results evaluate the quality of detected segments by measuring whether they are both correctly localized and correctly recognized. In our setup, each predicted segment is fed into a pre-trained FS-PoseNet (Fayyazsanavi al., 2024) recognizer, and is counted as correct only if the recognition accuracy exceeds 50%. As shown in Table 4, we observe a clear performance gap between detection and FSD-R. Even well-localized segments often fail to reach the required recognition threshold, especially in the Web domain. SL-Seg (Moryossef et al., 2023) achieves the highest overall AP@ Acc (53.5%), yet still struggles on the Web subset (20.8%), underscoring the difficulty of maintaining segment quality under noisy conditions. These results emphasize that effective detection must also consider recognizability. motivating future work on recognition-awee aware detection or joint optimization approaches.
Fingerspelling Recognition in Context (FSR-Context)) Following the protocol introduced in (Tanzer, 2024b), we extract fingerspelled spans from predicted translations and compute character- level Letter Accuracy based on alignment with annotated ground-truti phrases. We conduct our evaluation on expanded Auslan News dataset, which contains 18,604 sentence-level sign language videos, 13,208 of which include fingerspelled terms. We first evaluate state-of-the-art gloss-free SLT model (Zhou et al., 2023) on sentence-level sign language videos that contain fingerspelled content. The model achieves Letter Accuracy of 16.4% on the two Transformer based translation models: T5 (Raffel et al., 2020) (subword tokenization) and ByT5 (Xue et al., 2022) (character-level tokenization). Results show that ByT5 T5 outperforms T5,
task. We then compare

---

Published as conference paper at ICLR 2026
Ground Truth
6 V V 6 + + P D
IFSR
9
FSD
FSD-R
6
Figure 5: Case study comparing IFSR, FSD, and FSD-R on fingerspelling sequences.
achieving Letter Accuracy of 25.8% compared to just 10.2% for T5, These findings are consistent with previous work (Tanzer, 2024b) and suggest that character-level tokenization offers substantial benefits for preserving the spelling of out- out-of-vocabulary words in translation output. The details of the expanded Auslan News dataset are provided in the Appendix Section C.3.
5.3 CASE STUDY
In Figure 5, we present several case studies across IFSR, FSD, and FSD-R settings. Some segments are correctly detected and recognized. However, rapid fingerspelling challenges the recognizer (e.g., two P's within 8 frames), and loose detection boundaries cause unrelated gestures be included, resulting spurious predictions like "1" These errors highlight the compound difficulty of accurate detection and recognition under natural signing conditions. Additional case studies provided in the Appendix Scction I.
CONCLUSION
In this work, we introduce BANZ-FS, a large-scale and richly annotated dataset dedicated to finger- spelling in the BANZSL. Our dataset is constructed from three diverse sources that cover broad spectrum of signing tempos and registers, from formal broadcast interpretation to everyday interac- tion. BANZ-FS features over 35,000 aligned fingerspelling instances with multi-level annotations, including temporal segments, character sequences, lexical forms, and full-sentence transcriptions. We benchmark several fingerspelling-relann isolated fingerspelling recognition, and fingerspelling recognition in context. To support contextual FS evaluation, we also extend the Auslan-Daily News subset with three times more aligned content. Through analysis and experiments, we demonstrate the challenges and opportunities posed by fin- gerspelling in BANZSL, particularly in the context of two-handed systems, self-occlusion, rapid transitions, and lexical variability. We hope that BANZ-FS will serve valuable resource for advancing sign language understanding, and encourage further research on fingerspelling phenomena across diverse linguistic and visual contexts.
tasks BANZ-FS, including fingerspelling detection,
ACKNOWLEDGEMENT
This research is funded in part by ARC-Discovery grant (DP220100800 XY), ARC-DECRA grant (DE230100477 to XY) and Google Research Scholar Program. We also gratefully thank all anonymous reviewers and ACs for their constructive comments.
ETHICS STATEMENT
This work involves the curation of BANZ-FS, a large-scale dataset for BANZSL fingerspelling. We have carefully considered the ethical implications data collection, annotation, and release, line with the ICLR Code of Ethics.
Human Subjects Consent. All data collected the Lab Recordings subset are conducted under ethical oversight in safe, supervised laboratory environment. Each participant (or guardian, in the case of minors) signs detailed consent form (Appendix Section D) stating that their facial expres- sions and hand gestures may be recorded for research only, without commercial redistribution.
10

---

Published as a conference paper at ICLR 2026
Participation is voluntary, and participants can withdraw any time. The study protocol, consent procedure, and data handling plan are reviewed by the University's 's Research Ethics Committee, which classifies the study ethically exempt under its guidelines. We maintain signed consent records and perform post-recording verification, ensuring adherence to consent terms.
Compensation. All contributors fairly compensated: general volunteers at AUD $40/hour, and Deaf signers and Auslan experts at AUD $100/hour, in line with institutional and regional standards. Approximately 500 hours of paid annotation work are carried out under formal contract.
Privacy & Anonymization. All participants explicitly consent to the public release of their (un- blurred) recordings for academic apply face-blic bice-blurring (using de face) upon participant request, and delete data entirely if consent is withdrawn. Future releases also provide 2D/3D pose annotations to support privacy-preserving re- search. No personally identifiable information (PII) or sensitive data (health or financial information) collected.
A withdrawal and anonymization protocol is in place:
Copyright & Licensing. For the web-sourced and broadcast portions of the dataset, we strictly comply with platform Terms of Service. We not download, re-host, redistribute, or store any copyrighted audiovisual content from YouTube or other platforms. Only our derived annotations and the official public URLs of the original videos are released, and all video playback remains hosted under the control of the original content creator. Users of BANZ-FS corresponding platform ToS. Where videos are distributed under Creative Commons licenses (c.g., CC BY-NC-SA 4.0) (Shen et al., 2023) explicit permissions are available, follow those terms accordingly. The full dataset release is governed by a CC BY-NC-SA 4.0 license and an End-User User License Agreement (EULA) that prohibits commercial exploitation, re-identification, or surveillance use. A takedown-request email is provided for removal or anonymization requests.
required to adhere to the
Fairness & Representattivenss. We report signer demographics (gender, age, region, race) across all three data sources (Appendix Section C.7). While Auslan data dominates, the BANZSL alphabet shared across dialects, and cross-dialect evaluation shows good generalization. We acknowledge remaining imbalances and plan to mine additional BSL/NZSL data and incorporate community feedback in future releases.
Responsible Use. We include Responsible Use Statement with the dataset that explicitly prohibits its deployment surveillance, biometric identification, or other sensitive without further ethical review. Our release aims to support inclusive, equitable research benefiting the BANZSI. and Deaf communities.
contexts
REPRODUCIBILITY STATEMENT
We take reproducibility seriously, and due to the current stage of the review process, we provide website that includes some data samples (due storage limitations), data storage structure, and annotation files. Our main dataset repository and Google Drive link will be released after the review process. This ensures that anyone can regenerate exactly the dataset version used in our experiments.
For benchmarking, we exclusively use publicly available models and follow their original hyperpa- rameter settings without any modification. This guarantees fair and consistent comparison across methods. Detailed dataset statistics, preprocessing steps, and quality control procedures are pro- vided in Section 3. For more experiment details, please refer to Appendix Section F and our website ? BANZ-FS.
REFERENCES
Samuel Albanie, Glil Varol, Liliane Momeni, Triantafyllos Afouras, Joon Son Chung, Neil Fox, and Andrew Zisserman. Bsl-1k: Scaling up co-articulated sign language recognition using mouthing cues. In European conference on computer vision, pp. 35-53. Springer, 2020.
Matyáš Boháček and Marek Hrúz. Sign base- transformer for word-level sign language recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, pp. 182-191, January 2022.

---

Published as a conference paper at ICLR 2026
Necati Cihan Camgöz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. Neural sign language translation. In 2018 IEEE Conference on Computer Vision and Pat- tern Recognition, CVPR 2018. Salt Lake City, UT, USA. June 18-22, 2018, pp. 7784- 7793. Computer Vision Foundation IEEE Computer Society, 2018. doi: 10.1 10.1109/CVPR. 2018.00812. 12. URL http: hwwp//ccpnccccccccoooooocc theevf..e.... wttvv/ccco////o wwt.c///ooottttoooooooo Sergoo_Neural__iignnanngee
201 8/html/
Necati Cihan Camgöz, Oscar Koller, Simon Hadfield, and Richard Bowden. Sign language transform- ers: Joint end-to-end sign language recognition and translation. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp.
10020-10030, Computer Vision Foundation IEEE, 2020. doi: 10. R0.110/CCPR2200000o 200.0000...200000000 URL https: /owp/accpscccscepeoepoepcee.eese thectf. com content Seteeentt_______e CVPR Btt____CVV_____-0 sign
/Camgoz_ Sign__anguage........aa ptm.p..eee............ html.
Trans coaerrss___itt___eeeereeeetss htw.raaffooorreeeeoooooooee--o-o
João Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 4724-4733, IEEE Computer Society, 2017. doi: 20.1100/CVPR.2017..500 URL https: /tt./iiiooooo org/ 10. 1109 Ctt.///CCRRR../ 2017.
Yutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, and Stephen Lin. A simple multi multi-modality transfer learning baseline for sign language translation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp- 5110- 5120. IEEE, 2022a. doi: 10. (0009/CVPR2288.22222220 1109 00999/CVR52888 2 C22 00506.
URL https: http://ooooooooooo org/10.
Yutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, and Stephen Lin. A simple multi-modality transfer learning baseline for sign language translation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 5110- 5120. IEEE, 2022b. doi: 10. 1w..109//VP522688..2 201...50008822200050000 URL https: /rt//..oo........//////// 1109/CVPR5268 20...00220.00 00506.
Yutong Chen, Ronglai Zuo, Fangyun Wei, Yu Wu, Shujie Liu, and Brian Mak. Two-stream network for sign language recognition and translation. 2022c. URL http://papers: nips oc Con.recncccccoeeeeee.eee.ee.ee html. 6cd3ac24cdb1 (eee77/eeaa9971457748.....
NeurIPS, /
Aashaka Desai, Lauren Berger, Fyodor Minakov, Nessa Milano, Chinmay Singh, Kriston Pumphrey, Richard E. Ladner, Hal Daumé III, Alex X. Lu, Naomi Caselli, and Danielle Bragg. ASL citizen: community-sourced dataset for advancing isolated sign language recognition. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL TtLp:tp/// /papers/ mips. cc papappr_files/ppppper 2023/haase
(tw.ee/e66a444446665e333. wat.raagaaaaaatttttttt-tt-ttee and Jeencharrks.ht..
Amanda Cardoso Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth De- Haan, Florian Metze, Jordi Torres, and Xavier Giró-i-Nieto. How2sign: A large-scale multimodal dataset for continuous american sign language. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 2735- 2744. Computer Vision Foundation / IEEE, 2021. doi: 10. 0.1109/CCPR44433.222...... URL https: ottpps://ppencccessstttt thecff..sseee.....o com contenntt////ccoocooococooooo Dat.aaastoooo_ooonnininette Cnnous.
Largee Ho Scw.arggeeeaaaeeMMaaaaeeeee pttmr....ppp....mll html.
Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. RMPE: Regional multi-person pose
estimation. In ICCV, 2017.
Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Lu Li, and Cewu Lu. Alphapose: Whole-bod. regional multi celti--peroonn pose estimation and tracking in real-time. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
12

---

Published as a conference paper at ICLR 2026
Pooya Fayyazsanavi. Negar Nejatishahidin, and Jana Košecká. Fingerspelling posenet: Enhancing fingerspelling translation with bose-based transformer models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Viston pp. 1 1120-112 1130, 2024.
Manfred Georg, Garrett Tanzer, Saad Hassan, Maximus Shengelia, Esha Uboweja, Sam Sepah, Sean Forbes, and Thad Starner. Fsboard: Over 3 million characters of asl fingerspelling collected via smartphones. arXiv preprint arXiv:2407. 158660....1,,,, 2024.
Lianyu Hu, Liqing Gao. Zekang Liu, and Wei Feng. Dynamic spatial-temporal aggregation for skeleton-aware sign language recognition. In Nicoletta Calzolari, Min- Kan, Véronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguisties, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, pp. 5450-5460. ELRA and ICCL, 2024. org/ 2024 wree-reeeeee maain... 484.
URL httpt:tt////em http:///claaathhoooco
Jie Huang, Wengang Zhou, Houqiang Li, and Weiping Li. Attention- 3d-cnns for large- vocabulary sign language recognition. IEEE Transactions Circuits and Systems for Video Technology, 29(9):2822-2832. 2018.
Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional LSTM models for sequence tagging. CoRR, Abs//508.0199) 2015. URL http: http::/arxiv..oo http://aaaii...orrr/////0//00 abaaaabss/////////////// 01 991.
Songyao Jiang, Bin Sun, Lichen Wang, Yue Bai, Kumpeng Li, and Yun Fu. Skeleton aware multi- modal sign language recogmition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR) Workshops, 2021a.
Songyao Jiang, Bin Sun, Lichen Wang, Yue Bai, Kunpeng Li, and Yun Fu. Sign language recognition skeleton-aware multi-model ensemble. CoRR, abs/21 10.06161, 2021b. URL https://
arxivv.rg/aai wtt.orrgg/bbb///aee /2100 066661.
Tachwan Kim, Jonathan Keane, Weiran Wang, Hao Tang, Jason Riggle, Gregory Shakhnarovich, Diane Brentari, and Karen Liveseu. Lexicon-free fingerspelling recognition from video: Data, models, and signer adaptation. Computer Speech & Language, 102209-232. 2017.
Pavel Korotaev, Petr Surovtsev, Alexander Kapitanov, Karina Kvanchiani, and Aleksandr Na- gaev. Handreader: Advanced techniques for efficient fingerspelling recognition. arXiv preprint arXiv:2505. 10267, 2025.
Dongxu Li, Cristian Rodriguez, Xin Yu, and Hongdong Li. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. In The IEEE Winter Conference on Applications of Computer Vision, pp. 1459-1469, 2020a.
Dongxu Li, Chenchen Xu, Xin Yu, Kaihao Zhang, Benjamin Swift, Hanna Suominen, and Hongdong Li. Tspnet: Hierarchical feature learning via temporal semantic pyramid for sign language translation. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiees Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Newral Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020b. URL httpttt:::///ooooettetee ngs neurips 8c00dee 448 wtt.:c/777.eee000000000000000000 aawp:aaeooo.ttoaaaaaaaaaaaaaeeer cetprrps.ccpappeeee www.cc/ppp...cccccccc 2020/has2 html.
Dongxu Li, Xin Yu, Chenchen Xu, Lars Petersson, and Hongdong Li. Transferring cross-domain knowledge for video sign language recognition. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 6204 6213. Computer Vision Foundation IEEE, 2020c. doi: 10. 10.1009/VV4422000.200 001.R44260002000066666 2020.00624. URL https: http::::///peeaceessssss
hambrfrrrrrnngcooosesoooonnnnnii theccf..cc.cc com/ wwtp///tttetttteecoeeeooooee stdeeeeooooiieeee__s_eeeeeeeeeee Saaeddeo_Siin__annneeeeee Recognition_CvPeR 2w20 httpe_2000ppppppppe____ htmi.
wwtperttccc___2___ Cotp:///////ttttee___/_/20220000 /html/Li.
Jiefeng Li. Can Wang, Hao Zhu, Yihuan Mao, Hao-Shu Fang. and Cewu Lu. Crowdpose: Efficient crowded scenes pose estimation and a new benchmark. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10863-10872, 2019.
13

---

Published as a conference paper at ICLR 2026
Linjun Li, Tao Jin, Xize Cheng, Ye Wang. Wang Lin, Rongjie Huang, and Zhou Zhao. Contrastive token-wiss wise meta- veta-learning for unseen performer visual temporal-alige translation, In Findings of the Association for Computational Linguistics: ACL 2023, pp. 10993-11007, 2023.
Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention- based neural machine translation. In Lluís Marquez, Chris Callison-Burch, Jian Su, Daniele Pighin, and Yuval Marton (eds.), Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pp. 1412-1421. The Association for Computational Linguisties, 2015. doi: 10.18653/v1/d15-1 166. URL https:
/ //doi. www.ooggg/....o.oooooooooo.oe 18w..v1166vvv.. 116-1166..
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9NNov)2557--2655 2008
Florent Mahoudeau. Mict-ranet for real-time asl fingerspelling video recognition. https://
github. com/ Emahoudeau/MicT RANeR- ASLL- SOLL-Fingee Fingernppeliiiiieeeeeeeeeee ling, 2020.
Yuecong Min, Aiming Hao, Xiujuan Chai, and Xilin Chen. Visual alignment constraint for continuous sign language recognition. In proceedings of the IEEE/CVF international conference on computer vision, pp. 1 11542-11551, 2021.
Amit Moryossef, Ioannis Tsochantaridis, Roee Aharoni, Sarah Ebling, and Srini Narayanan. Real- time sign language detection using human pose estimation. In European Conference on Computer Vision, pp. 237-248. Springer, 2020.
Amit Moryossef, Zifan Jiang, Mathias Müller, Sarah Ebling, and Yoav Goldberg. Linguistically motivated sign language segmentation. arXiv preprint arXiv:2310. 2023.
Carol A. Padden. The asl lexicon. pp. 1:33-51, 1998.
Katerina Papadimitriou, Galini Sapountzaki, Kyriaki Vasilaki, Eleni Efthimiou, Stavroula-Evita Fotinea, and Gerasimos Potamianos. A large corpus for the recognition of greek sign language gestures. Computer Vision and Image Understanding, 249:104212, 2024.
KR Prajwal, Hannah Bull, Liliane Momeni, Samuel Albanie, Güil Varol, and Andrew Zisserman. Weakly-supervinn fingerspelling recognition in british sign language videos. arXiv preprint arXiv:2211.0895.. 2022.
Nicolas Pugeault and Richard Bowden. Spelling it out: Real-time asl fingerspelling recognition. In 2011 IEEE International conference on computer vision workshops (ICCV workshops), pp. 1119-1119. Ieee, 2011.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-60 1-6,, 2020.
Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, 45(1 1):2673-2661 1997.
Xin Shen, Shaozu Yuan, Hongwei Sheng, Heming Du, and Xin Yu. Auslan-daily: Australian sign language translation for daily communication and news. Alice Oh, Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 pacppppprrllleeee files/ieee/ee 16. 2023, 2023. URL http: http::/ppppess nips, cc/
Benchmarks html.
Xin Shen, Heming Du, Hongwei Sheng, Shuyun Wang, Hui Chen, Huiqiang Chen, Zhuojie Wu, Xiaobiao Du, Jiaying Ying. Ruihan Lu, al. Mm-wlauslan Multi-view multi-modal word-level australian sign language recognition dataset. arXiv preprint arXiv:2410. 19488, 2024a.
Xin Shen, Lei Shen, Shaozu Yuan, Heming Du, Haiyang Sun, and Xin Yu. Diverse sign language translation. arXiv preprint 140 19586, 2024b.
14

---

Published as a conference paper at ICLR 2026
Xin Shen, Heming Du, Hongwei Sheng, Lincheng Li, and Kaihao Zhang. Auslanweb: A scalable web- based australian sign language communication system for deaf and hearing individuals. In Guodong Long, Michale Blumestein, Yi Chang, Liane Eewin,Eytan, Zi Helen Huang, and Elad Yom-Top (eds.), Proceedings of the ACM on Web Conference 2025, WWW 2025, Sydney, NSW, Australia, 28 April 2025- 2 May 2025, pp. 5212-5223, ACM, 2025a. doi: 10.1145/6696410.374455 URL https: http/ii...// org 10 0tte://...11/339966644 0111111///49111.1 3714525
Xin Shen, Heming Du, Miao Xu, Miaomiao Liu, and Xin Yu. Cross-view isolated sign language recognition challenge: Design, results and future research. In Companion Proceedings of the ACM on Web Conference 2025, pp. 2444-2447, 2025b.
Xin Shen, Xinyu Wang, Lei Shen, Kaihao Zhang, and Xin Yu. Cross-view isolated sign language recognition via view synthesis and feature disentanglement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20647,20657, 2025c.
Hongwei Sheng. Xin Shen, Heming Du, Hu Zhang, Zi Huang, and Xin Yu. Ai empowered auslan learning for parents of deaf children and children of deaf adults. Al and Ethics, pp. 1-11, 2024.
Hongwei Sheng, Xin Shen, Heming Du, and Xin Yu. Mobile auslan: A multimodal dialogue-centered sign language learning system. Computer Vision and Image Understanding, pp. 104646, 2026.
Bowen Shi, Aurora Martinez Del Rio, Jonathan Keane, Jonathan Michaux, Diane Brentari, Greg Shakhnarovich, and Karen Livescu. American sign language fingerspelling recognition in the wild. In 2018 IEEE Spoken Language Technology Workshop, SLT 2018, Athens, Greece, December 18-21, 20/8, pp. 145-152. IEEE, 2018. doi: 10. 10.1109/SLT.2218.6600 86639663, org/ 1109/STSA 201 8639639.
URL http::/ /dooi.
Bowen Shi, Aurora Martinez Del Rio, Jonathan Keane, Diane Brentari, Greg Shakhnarovich, and Karen Livescu. Fingerspelling recognition in the wild with iterative visual attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5400-5409, 2019.
Bowen Shi, Diane Brentari, Greg Shakhnarovich, and Karen Livescu. Fingerspelling detection in american sign language. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 4166-4175 Computer Vision Foundation IEEE, 2021. doi: 10. 00.110990CPP44447.20000000000 1.0415. URL https: http:///ppaaccccseeeeesssess Srirtent//TrR2222///tt////Si/i/p www.rrrepeeeininnggg
cttrcessss.hetee. thecvf. com/
html.
Bowen Shi, Diane Brentari, Gregory Shakhnarovich, and Karen Livescu. Open-domain sign language translation learned from online video. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 6365-6379, Asso- ciation for Computational Linguistics, 2022. URL https: http:///alaattthho 2wwp://ammttoollloooo//022222222 org/2022. emn.p-mainn 100-mainn 427.
Chenyang Si, Ya Jing, Wei Wang, Liang Wang, and Tieniu Tan. Skeleton- action recognition with spatial reasoning and temporal stack learning. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018 15th European Conference, Munich. Germany. September 8-14, 2018, Proceedings, Part I, volume 11205 of Lecture Notes in Computer Science, pp. 121. Springer, 2018. doi: 10.1107/978-3-000-20 htttss://dof orgi 10. 1007 978 030- 01 246-127..22
URL
Thad Starner, Sean Forbes, Matthew So. David Martin, Rohit Sridhar, Gururaj Deshpande, and et al. Popsign ASL v1.0: An isolated american sign language dataset collected via smartphones. In Alice Oh. Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URL http: /pap:r://ppp/aaaee/eee .ni cc. halpppr__fllssppeee 2023 h12/3//aah/oee 6080000daada00000000000000 (ttp2222ea729992222662222 and Bed_heenhmarks .tt.al.......m........m..m.m
Dhstract-Datasees
Garrett Tanzer. Fleurs-asl: Including american sign language in massively multilingual multitask evaluation. arXiv preprint 2rX8v:2408./333 /3585, 2024a.
15

---

Published as a conference paper at ICLR 2026
Garrett Tanzer. Fingerspelling within sign language translation. arXiv preprint arXiv:2408.07065, 2024b.
Garrett Tanzer and Biao Zhang. Youtube-sl -l-2u: A large-scale, open-domain multilingual sign language parallel corpus. arXiv preprint urXiv:2407. 11144, 2024.
Gabriel Tsechpenakis, Dimitris N. Metaxas, and Carol Neidle. Learning-based dynamic coupling of discrete and continuous trackers. Comput. Vis. Image Underst., 104(2-3):140- 156, 2006a. doi: 10. 1016/J. CVI..2006.0..0000 .7.09. URL https: /tttps://cc... org/ 10. 10.11101110 6/ ieviu.2000 07 009.
Gabriel Tsechpenakis, Dimitris N Metaxas, Carol Neidle, and Olympia Hadjiliadis. Robust online change-point detection in video sequences. In Proceedings of the 2006 Conference on Computer Vision and Putlern Recognition Workshop, pp. 155-161, 2006b.
Dave Uthus, Garrett Tanzer, and Manfred Georg. Youtube-asl: A large-scale, open-domain american sign linguage-english parallel corpus. Advances in Neural Information Processing Systems, 36: 190299290447 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Philippe
Weinzaepfel, Zaïd Harchaoui, and Cordelia Schmid. Learning to track for spatio-temports action localization. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 3164-3172. IEEE Computer Society, 2015. doi:
10.1 100100//CCV.2015..3.6 URL https: /ooiiiiiio/// org/ /00 wtt./////..././////////./. 1109/10CCY/C wtt.ccc......... 2015.362.
Huijuan Xu, Abir Das, and Kate Saenko. R-C3D: region convolutional 3d network for temporal activity detection. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 5794-5803. IEFE Computer Society, 2017, doi: 10.1 10.1109/CCCV.2017...7 www.org/ooooooooooo............. .rgg 10. 11 09/ / ICVV. 2017. 61
URL wwt.thtpp//.oo. / io
Qingzheng Xu, Ru Cao, Xin Shen, Heming Du, Sen Wang, and Xin Yu. M3gmm: A large-scale multimodal multi-view multi-person pose dataset for fitness activity understanding in real-world settings. In Proceedings of the Computer Vision and Pottern Recognition Conference, pp. 12289- 12300, 2025.
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. Byt5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291-306, 2022.
Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton- based action recognition. In Sheila A. Mcllraith and Kilian Q Weinherger (eds.), Proceedings of the Thirty-Secoon AAAI Conference on Artificial Intelligence, pp. 1444-7452. AAAI Press, 2018. URL https: www. aaai. org/ ocs index.ndx. phpp tatp/AAA/ /At//AAAI/AAAA AAAIIAAAIIII paper wwwppe/vvwwwwwe/////
Hee-Deok Yang and Scong- Bcong-Whan Lee. Simultancous spotting of signs and fingerspellings based on hierarchical conditional random fields and boostmap embeddings. Pattern Recognit., 43(8)22858 2870, 2010. doi: 10. patcog. 2010 03. 007.
URL attpps:/ii /to.. wtt.dd....g/0..11/////.33
Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. Effective whole-body pose estimation with two-stages distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4210-4220, 2023.
Polina Yanovich, Carol Neidle, and Dimitris N. Metaxas. Detection of major ASL sign types in continuous signing for ASL recognition. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck. Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asunción Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Tenth Interna- tional Conference
Language Resources and Evaluation LREC 2016, Portorož, Slovenia, May 23-28, 2016. European Language Resources Association (ELRA), 2016. URL http: summa r es 714 4.hml.
rec201 w/ww.. 1 rrec- conf. worg/pprccee.. rec21 2emne2000mmmmmaaaa
16

---

Published as a conference paper at ICLR 2026
Aoxiong Yin, Tianyun Zhong, Li Tang. Weike Jin, Tao Jin, and Zhou Zhao. Gloss attention for gloss-free sign language translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2551 2023.
Jiaying Ying, Xin Shen, and Xin Yu. Vision-based abnormal action dataset for recognising body motion disorders. In Australasian Joint Conference on Artificial Intelligence, pp. 443-455. Springer, 2024.
Benjia Zhou, Zhigang Chen, Albert Clapés, Jun Wan, Yanyan Liang, Sergio Escalera, Zhen Lei, and Du Zhang. Gloss-free sign language translation: Improving from visual-language pretraining. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20871-20881, 2023.
Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li. Improving sign lan- guage translation with monolingual data by sign back-translation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 1316-1325. Computer Vision Foundation / IEEE, 2021. doi: 10.11000CVP44444.2222000000 2011.0. URL httpss//ppeaaccsss.hhheeee
22ou theccf..ssse..ccs.....e http:////cooeeee/////////////aa/ ottp://aaaa.aaaaaaoooooooooooooa
Ronglai Zuo, Fangyun Wei, and Brian Mak. Natural language-assisted sign language recognition. In
IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC,
Canada, June 17-24, 2023, pp. 14890-14900, IEFE, 2023. doi: 1..1099CV952229..222..00
URL wwt.tttpp//.oo.....m /ooi wwrpoi.orgg/ooe 1109/ 1ttp//////...1//////////7772 722...222222.22 2 2023001330......
17

---

Published as a conference paper at ICLR 2026
This appendix is organized as follows:
BROADER IMPACT (Section A).
LIMITATION AND FUTURE WORK (Section B).
BUILDING BANZ-FS (Section C).
CONSENT FORM FOR BANZ-FS RECORDING (Section D).
MORE DETAILS FOR VIDEO REPRESENTATION (Section E).
EXPERIMENTAL SETTINGS (Section F).
THE BASELINE OF AUSLAN-DAILY NEWS V2 (Section G).
ADDITIONAL. DISCUSSION (Section H).
CASE STUDY FOR BANZ-FS LINGERSPELLING DETECTION AND RECOGNITION (Section I).
CASE STUDY FOR AUSLAN-DAILY NEWS SIGN LANGUAGE TRANSLATION (Sec-
tion J).
..LM USAGE STATEMENT (Section K).
A BROADER IMPACT
The BANZ-FS dataset addresses critical gap in the sign language research community by focusing on the underrepresented two-handed fingerspelling systems of BANZSL (British, Australian, and New Zealand Sign Languages). This work has the potential to significantly improve accessibility technologies for Deaf and hard-of- heehiinn communities across multiple English-speaking regions. By supporting robust research on fingerspelling detection and recognition, BANZ-FS can contribute to the development of real-time translation systems, assistive educational tools for sign language learners, and inclusive communication platforms. Importantly, the dataset includes real-world scenarios drawn from news, vlogs, and lab settings, thus promoting domain generalization in practical applications. However, as with any dataset involving human subjects, privacy, representation, and consent are essential considerations. All included video data are sourced from publicly available content or recorded with informed consent. Nevertheless, we acknowledge that biases may still exist, such as overrepresentation of Auslan compared to BSL or NZSL, and uneven distribution of fingerspelled letters. Future work should address these issues through targeted data collection and model adaptation techniques. Finally, while the goal to aid accessibility, there also a risk misuse such as surveillance unauthorized profiling using sign language recognition systems. We strongly encourage researchers and practitioners to follow ethical guidelines and collaborate closely with Deaf communities when deploying models trained on BANZ-FS.
B LIMITATION AND FUTURE WORK
Our work has two primary limitations. Letter Frequency Imbalance. Letter imbalance exists in the dataset- dataset-frequent characters like "N", "S", "E", and "W" dominate due to their linguistic role in directional terms, while rarer letters such "X" and "I" are significantly underrepresented. Such skew is common in natural language corpora and other fingerspelling datasets; for example, in ChicagoFSWild+ (Shi et al., 2019), the most frequent letters "A", "E", and "O" each appear over 20K times, whereas the rarest letters "Z" and "Q" appear only 494 and 311 times, respectively. To mitigate this effect, we carefully balanced the validation and test splits ensure fair representation of low- low-frequency letters. As shown in Table 5, several characters and digits occur fewer than 1,000 times, but we ensure their presence across all splits. Our dataset design further supports alleviation of long-tail issues: the Lab Recordings subset enables targeted collection of underrepresented characters, and our BANZ-FS-trained detector can be used to mine additional candidate instances from unlabeled videos. In future work, we plan to explore sign language generation techniques to synthetically augment low-frequee frequencyyny letters and improve overall letter coverage. Dialect Imbalance. There is clear dialect imbalance within the BANZSL subset: the data is heavily skewed toward Auslan, with BSL and NZSL contributions being relatively limited. Nevertheless. all three dialects share the same two-handed fingerspelling alphabet, and shown in Table 3 and Table 4, models trained on the
18

---

Published as a conference paper at ICLR 2026
Table Frequency of underrepresented letters and digits (total count 1,000) across train/ vaiin//aid/test splits in BANZ-FS.
Split
FS-XX
FS-1J"
Valid
98
84
56
Tist
114
107
85
79
66
54
39
Total
808
734
664
027 470 420
339
Auslan subset generalize well BSL and NZSL web data. In future work, will our trained BANZ-FS detector to mine additional BSL/NZSL clips from broadcast and web sources, thereby improving dialectal diversity and representation.
Beyond data, we also plan to investigate recognition-awer -aware detection models that jointly optimize for temporal localization and fingerspelling accuracy. Our current benchmark focuses RGB front-view videos to ensure comparability with web data and existing open-source models. Although the lab setup records synchronized multi-vier RGB-D streams, these modalities are not used in the reported experiments. In future work, we plan to explore depth-based modeling, multi-view fusion, and 3D hand reconstruction to further leverage the full potential of the released recordings. In addition, we aim to explore context-conditioned models that leverage sentence-level semantics to improve recognition of ambiguous or incomplete fingerspelling segments.
C BUILDING BANZ-FS
In this section, we explain the various stages of data processing and labelling in detail for preparing BANZ-FS, from collecting sources to storing final data.
C.I DATA PROCESSING LABELLING
Although the original Auslan videos accompanied by English subtitles, the sentence boundaries in the subtitles are often misaligned with the actual signing segments due to differences grammar, timing, and expression modalities between sign and spoken languages. To address this, perform a sentence-level alignment procedure. First, we clean the raw subtitles by merging incomplete fragments (e.g., those ending with commas), splitting multiple complete sentences within single time interval, and removing non-informative expressions such as interjections.
As delineated in Section 3.1, employ three distinct operations for subtitle cleaning (Shen et al., 2023). Here, we present a few representative examples:
Incomplete subtitles:
(00:000077,480)-(000000) 10, 160] Today, a flood emergency warning issued for (00:00.10.220)-00000011,,,, Tasmania's 's River Derwent.
Revise:
Today, flood emergency warning issued for Tasmania's 's
River Derwent.
Several complete subtitles that appear within a time interval:
(00:00..4.5200) Hello and welcome ABC News. I'm Gemma Veness.
Revise:
[000:00.54.2200--0::0005..3. Hello and welcome ABC News. I'm Gemma Veness.
[00.000.54.520]-000000:338
Complete sentence that only contains modal particles:
(0023::0.22400-0002334440,000 Ha-Ha.
Revise: Remove this subtitle.
After preprocessing, we obtain clean sentences. Then, Auslan experts manually align each sentence with its corresponding video segment. As illustrated in Figure 6, this alignment is conducted the sentence level, taking into account both audio-aligned and sign-aligned timelines. The signer is first tracked, followed by precise segmentation of each sentence's temporal span within the video.
19

---

Published as a conference paper at ICLR 2026
Signer Track
www. www. www. www.
welcome News.
am
Subtitle Aligament
News.
Celina.
Time
Vingerspdling Alignment
ABB
Lesicous Aligument
ABC
News
Celina
Figure 6: Illustration of sentence-level alignment. The signer is first tracked, followed by audio and sign alignment of the subtitle. Sentence segments, fingerspelling intervals, and lexical boundaries then precisely annotated.
This results in temporally grounded Mideo-sentent pairs, which are essential for training robust and accurate sign language translation (SLT) models.
addition to sentence-level alignment, we also annotate fine-grained elements such as fingerspelling segments and lexical boundaries. Fingerspelling alignment identifies the exact start and end times of letter-by-letter signing, which often corresponds to proper nouns or unseen words. Lexicon alignment further breaks down the sentence into semantically significant units such named entities or domain-speciffe terms. These multi-level annotations enable a richer understanding signed content and facilitate downstream tasks such as fingerspelling recognition and detection.
C.2 ADDITIONAL DETAILS DATASET STRATIFICATION
This section provides additional clarification on how the dataset splits are constructed.
Signer stratification. Each domain includes Out-of-Set (OOS) signers in the test partition to prevent identity memorization. News, Lab, and Web respectively contain multiple unseen signers in their test splits, ensuring that evaluations reflect generalization
rather than repeated individuals.
Vocabulary stratification (OOFS). We track Out-of- Out-of-Training Fingerspelling Strings (OOFS), defined character sequences appearing in the test set but not in the training set. All domains include substantial number of OOFS items, preventing inflated performance due to repeated fingerspelling strings across splits.
Visual stratification (Lab). The Lab subset contains synchronized recordings from multiple view- points. Even when lexical items coincide across splits, differences in viewing angle and spatial configuration reduce the risk of pixel-level memorization and encourage viewpoint-robust evaluation.
Character distribution. We report full letter-frequency statistics in Table 5 of the supplementary material. Rare letters (e.g., "X", "J"") appear in both training and test partitions, supporting fair assessment of long-tail character performance without artificially balancing the splits.
C.3
AUSLAN-DATI DAILY NEWS V2
Auslan experts manually annotate sentence-level temporal boundaries within the Auslan-Daily News videos to accurately align signed utterances with their corresponding English subtitles. This aligmment process not only corrected mismatches between spoken captions and signing segments but also ensured each signed sentence was temporally grounded with high precision. As a result of this effort, we substantially expanded the original Auslan-Daily News subset and release it as a new version, termed Auslan-Daily News V2.
Table presents detailed comparison between the Auslan-Daily News VI and V2 sub-datasets in terms of data volume, diversity, and vocabulary statistics. V2 significantly expands upon VI, featuring nearly twice the number of annotated segments (29,669 vs. 11,065). frames (5.6M vs. 2.3M), and total words (492,624 188,774). It also includes larger vocabulary size (15,976 vs. 12,346) and more signers (27 vs. 18), reflecting improved linguistic and signer diversity. The increase in oft-of-vocabulae (OOV) words and singletons further illustrates the dataset's long-tail lexical
20

---

Published as a conference paper at ICLR 2026
Table 6: Key statisties of Auslan-Dail Daily New VI and Auslan-Daile -aill New V2. OOV: out-of-vocabulary. Singleton: words that only occur once in the training dataset.
Sub-Dataset
Ausian-Daily (Shen 2023)
Austan-Dally
Domain/ Topic
Documentary
Video Resolution EPS
x280x 720/1900 29.97
x280x 7206
Späi
Train
Dev
Test
Train
Dev
Text
Total
Segments
9.665
700
16,604
1,000
Signers
18
17
17
22
19
27
Frames
2,072,455
144,819
142,893
2,925,597
157,619
149,984
5,593,387
Vacals
12,346
Tot.
163,268
11,376
11,580
277,699
14,543
Tol. DOVs
326
304
217
224
475
Singletons
5,267
6,110
8,039
Table Examples of fingerspelling (FS) instances and their corresponding aligned content.
Sentence
Fingerspelled Sequence
Aligned Text
The growth going have heavily equity stadents wanting univensity.
EQTITY
equity
We' 've used Variety the post for his equipment and support.
EQ
equipment
The Western Buildogs beaten Greater Western Sydney.
Grester Western Sydney
Irishwomon Looen Maguire has ono-shot lead heading.
M
Maguire
Steven milles did a good job as leader.
MIIMILES
miles
He said he will get job.
BOB
No aligned weed
distribution, which poses challenges but also fosters better generalization in sign language translation and recognition models.
C.4 FINGERSPELLING ANNOTATION GUIDELINES.
To ensure consistency and accuracy in fingerspelling (FS) labeling, adopt structured three-step annotation protocol:
1. Temporal Identification: Annotators first review the entire video and identify all time intervals where fingerspelling occurs. These segments are typically characterized by rapid handshapes corresponding to individual alphabet letters, often used to spell out names, technical terms, or voocabbbyyayy words.
2. Character-Level Transcription: Within each identified FS segment, annotators transcribe the fingerspelled content into a sequence of characters (A-Z), ensuring the character se- quence reflects the exact order and repetition observed in the signing. Ambiguous or occluded handshapes may be annotated with a special token (e.g.. ***) when necessary.
3. English Alignment:
After obtaining the character-level transcription, annotators check whether the transcribed fingerspelling sequence corresponds to any English word or phrase within the aligned sentence. If match is found, the FS sequence is linked to the correspond- ing word or phrase. If no such alignment exists (e.g., due to fingerspelling of foreign names or uncommon entities), the segment is annotated as not aligned with any English content.
As shown in Table 7, present several representative samples from our fingerspelling annotation process. Each example includes the full sentence containing a fingerspelled segment, the transcribed character sequence, and the corresponding aligned English word phrase when available. This protocol ensures that each FS annotation is temporally precise, linguistically grounded, and aligned with surrounding sentence context where possible, enabling reliable training of FS recognition and translation models.
C.5
FINAL DATASET STORAGE
The finalized BANZ-FS dataset, curated and annotated by sign language experts and trained annota- tors, is structured into three task-specific subfolders and is hosted on public cloud repository. The organization of the dataset is illustrated in Figure 7. Each subfolder corresponds to a specific task: Fingerspelling Recognition, Fingerspelling Detection, and Sign Language Translation, respectively.
21

---

Published as a conference paper at ICLR 2026
BANZ-FS
Tmnernpeling_RReegaiinnn Recogaition
Fingerspelling Detection
Sips Lan_jugggg_____ Laaquag_TTamnatiinn
Tramiation
Ansotation
Amnotation
Annotation
Data
Data
Data
Elagrrpiligg Rnoquioadd
Eiprrpptingg Deistinnip
Skp. Lampacy Trankitio
Figure 7: Hierarchical data folders for BANZ-FS on G Google Drive.
Table 8: Statistics of isolated versus continuous fingerspelling clips. "Length" refers total duration in seconds; "Average" is the mean duration per clip.
Source
Isolated Clips
Contionons Clips
AClips
Length
Average
AClips
Length
Average
Newa
22,108
17.483.39
6,471.37
1.76
2,149
5,770.38
2,336,38
Lab Recordings
10,732
14,141.18
131
0
0.00
0.00
Within each task folder, the dataset is further divided into two main components. The Annota- tion subfolder contains three files: Train, Val, and Test, each storing task-specific labels such as sentence-level alignments, fingerspelling boundaries, or character sequences depending on the task. These annotations are derived from expert manual alignment procedures and re- flect high-quality temporal labeling. The Data subfolder contains compressed archive (e.g.. Fingerspel 1 ling Beiinn_eeeee Recognition zip) holding all the corresponding RGB video clips. These videos are already pre pre-processed to focus the signer and are trimmed according to the annotated segment durations.
This storage structure ensures modular access for each task. allowing researchers to independently work on detection, recognition, or translation without ambiguity. Additionally, all annotations are time-aligned with video content, facilitating temporal learning and evaluation. An overview of the folder structure is shown in Figure 7, and recommended splits are discussed in Table 2 and Table 6.
C.6 CONTINUOUS FINGERSPELLING CLIPS
In our annotations, each lexical fingerspelled item is treated the minimal unit. For example, in the utterance "Here are the forecasts for Brisbane, New South Wales, and Sydney", although the FS segments "BB", "NSW". and "SY" appear consecutively, we annotate each of them as a separate fingerspelling segment with its own time interval. This segmentation strategy results in an average clip length of approximately 1.5 seconds across the dataset.
To support research on long-form fingerspelling detection and recognition, we additionally merge temporally adjacent fingerspelling segments and report statistics for these continuous spans. The statistics summarized in Table 8, and both isolated and continuous segment boundaries will be included the public release to enable future benchmarking efforts.
These findings highlight the linguistic and stylistic diversity captured in BANZ-FS: fingerspelling in news broadcasts tends to be fast and compressed, whereas YouTube content is slower and more naturalistic. Although our Lab Recordings subset does not include continuous FS clips, it contains fine ine-ggriied temporal annotations for each lexical item, enabling generation of high-quality synthetic continuous sequences in future work.
22

---

Published as a conference paper at ICLR 2026
Table Signer Demographic Breakdown by Subset.
Subset
67 #Signers
Gender (M/F)
Age Kange
Region
Demographics (Caucasian / Asian Africon)
Lab Recordings
38/ 29
16-75
BANZ
27 /0/00/10
News Clips
25 55
Aushan
23/6/0
BSL, NZSL
C.7 DEMOGRAPHIC REPRESENTATION AND COVERAGE
As shown in Table 9, we provide demographic summary of the signers in our dataset, including gender, age range, collection region, and demographics across the three subsets: Lab Recordings, News Clips, and Web Data. While Web and News subsets rely on public sources and offer limited control over signer demographics, we address this by proactively recruiting a diverse signer pool in the Lab Recordings subset, ensuring broader representation in terms of age and racial background. We also confirm that our collection covers participants from multiple regions within Australia, and the Web subset includes samples from the UK and New Zealand. Although Auslan content makes up the majority of the dataset, the BANZSL fingerspelling alphabet is shared across all dialects, providing solid foundation for cross-dialect generalization. We acknowledge that perfect demographic balance (Section B) is difficult to achieve, but our ongoing efforts in data collection and documentation aim to support transparent and inclusive dataset construction.
D CONSENT FORM FOR BANZ-FS RECORDING
Consent Form for Recording of the Australian Sign Language Dataset
Dear Participant,
Hello!! We dedicated the research sign lingugg. We conducting academic project aiimed at rasodding and mnllyving Austmlian Sign I Languego (Auslan). We invite you to particigate this project. The porpose this project
learning and dissemunation of sign language and enhance understanding
application of Aualan.
Mode Participation:
You will be recorded while using Ausian for comeunication.. These recordings include your facial expression and hand gestures.
Privacy and Data Use:
We commit using recorded solely for academic research purposes any commercial will be anonymized ensure the security of your personal information. The video material may be presented academic conferences, in research papers, educational courses.
Consent Details:
1 have read and understood the information about the reseurch described above.
lagree to participate the recordings Australian Sigs Language
uadeestand that my participation volumtar, withdraw any without any adverse
consequences
lagree that my ficisll expressions and gestures may be rocorded and used for acadomic rosearch.
Please fill out the Sollowing information and sign below to indicate your consent to participate:
Name:
Signature:
Date:
We greatly appreciate purticipation and suppoct!
Should you have any questions require further information, please contact us at:
Contact Person: [Name of Coordinntoe
Email: [Cooedinator's Enail]
[Cocrdinator's Phone]
Figure 8: Consent Form for Recording.
23

---

Published as a conference paper at ICLR 2026
Due to the inclusion of facial information in our dataset, obtain consent from volunteers and have them sign the consent form depicted in Figure 8 before recording data. We do not release personally identifiable information such names, ages, occupations, indications of whether individuals deaf or hard of hearing. It is important to note that our dataset is strictly for academic use and can not be used for commercial purposes.
E MORE DETAILS FOR VIDEO REPRESENTATION
RGB-based: We use the pre-trained I3D model form (Li et al., 2020a) and features with a window width of 16 and a stride of 2 are extracted:
13D(Ft-g ... F, --- @ Fr+g),
(1)
where Se is the representation of the 1-th frame, is the window width, and @ denotes the concatena- tion operation.
Leveraging pose information in action recognition presents significant benefits regard- ing robustness and semantic representation. We flatten the pose array 2 to Ay R (2' where 7 is the number of frames and N is the number of keypoints. Meanwhile, the results of our experiment show that the use of partial body and two hand keypoints will perform better for tasks related to sign language (Xu et al., 2025; Shen et al., 2025c).
Pose-based:
F EXPERIMENTAL SETTINGS
We mention that all models used in this work are publicly available. Each of the models we use is linked below:
Isolated Fingerspelling Recognition:
SL-Transformer (Camgöz al., 2020) 7. Iterative-Att (Shi et al., 2019) O
MiCT-RANet (Mahoudeau, 2020) O TS-FS-Reg (Chen et al., 2022c) o and FS-
PoseNet (Fayyazsanavi et al., 2024) O.
Fingerspellingg Detection:
Bi-LSTM CTC (Huang et al., 2015) Q. Modified R-C3D (Xu et al., 2017) A. TS-FS- Det (Chen et al., 2022c) O MT-FS-Det (Shi et al., 2021) O and SL-Seg (Moryossef et al., 2023) ?
Fingerspelling Recognition in Context and Sign Language Translation: SL-Luong (Luong et al., 2015) Q Trannf (Camgöz et al., 2020) Q. TSPNet (Li et al., 2020b) ? MMTLB (Chen al., 2022a) O GASLT (Yin et al., 2023) O and GFSLT- VIP (Zhou et al., 2023) (
We express profound gratitude to the aforementioned authors for their invaluable contributions.
All the training and fine-tuning experiments are run on a machine with four NVIDIA GeForce RTX 3090 GPUs. We use the default hyperparameters for training the models of fingenspelling-related tasks and sign language translation.
G THE BASELINE OF MASLAN-DAILY NEWS V2
To evaluate the effectiveness of existing sign language translation (SLT) models on our extended dataset, we benchmark several state-of-the-art gloss-free SLT systems on both Auslan-Daily News VI (Shen al., 2023) and our newly constructed Auslan-Daily News V2. We consider settings based on input pre-proeee rrc-proceesing:
Single-Person SLT: The signer is automatically detected and cropped from the original video. This setting eliminates most background noise and visually isolates the signing individual.
Multi-Person SLT: The entire video frame is preserved, including other people and back- ground elements. Although only one person performs sign language in these clips, the presence of scene context and distractors makes translation more challenging.
24

---

Published as a conference paper at ICLR 2026
Table 10: Translation results of Single/Multt--Prsson SLT gloss-free models on Auslan-Dalte Dailan News (Shen et al., 2023) and our newly extend Auslan-Daily News V2.
Ausian Daily News V1 (Shen et al., 2025)
Ausdan Daily News V2
Single-Per. SLT
Inpux
R
B1
B2
B3
B4
R
B1
B2
B3
B4
SL.Luung (Luong el al. 2015)
Pose
20.65
19.84
7.81
4.59
2.81
21.71
21.42
10.55
6.58
4.94
St.-Luong (Loong 2015)
RGB
16.14
16.92
7.44
4.07
2.68
15.47
16.43
7.61
5.19
3.91
Transf (Camgöi
St.-Trand (Campiir al.. 2020)
RGB
14.93
17.64
7,4
3.98
2.52
13.93
7.02
4.33
3,05
TSPNet-Joimi (1. ot 20200)
RGB
19.71
18.23
5.97
3.21
2.26
20.09
20.39
7.66
4.23
2.83
MMTLB (Chen
2022a)
RGB
18.90
19.64
5.30
3,26
2.31
16.80
18.80
8.01
5.15
3,68
GASLT 2023)
GASLT' (Yin et al., 2023)
RGB
22.01
19.54
7,45
4.41
2.56
23.94
20.99
8.22
6.08
3.77
CFSLT VLP Vin et al., 2023)
RGB
27.32
23,00
9.93
6.06
4.43
26.24
22,59
11.50
7.29
5.44
Multi-Per. SLT
Input
B2
B3
B4
SI-LLonn (Luong 2015)
6.11
3.27
StanTe (Camgrie et 2020)
RGB
13.68
16.58
5.86
2.72
1.55
15.05
17.29
7.63
4.49
3.09
TSPNer-Joim ( 20200)
RGB
14.64
17.33
3,86
1/66
1.89
14.68
13.77
6.90
3.69
2.33
2022a)
GASLT (Yin et
RGB
19.73
16.99
6.25
3.44
2.36
20.75
6.91
3.56
GFSLT-VIP (Yin al., 2023)
RGB
20.83
18.93
6.02
4.33
3.05
21.87
18.07
7.66
5.12
4.10
We evaluate models using RGB and pose input modalities, reporting BLEU scores (B1-B4) and ROUGE (R) metrics in Table 10. The results clearly show that Auslan-Daily News V2 more challenging than VI, with slightly lower scores across all models and metrics, especially under the Multi-Person setting. This highlights the increased variability and complexity introduced by our new annotations and broader content coverage.
Among the tested models, GFSLT-VLP (Zhou et al., 2023) consistently achieves the best performance across both datasets and settings, demonstrating the benefit of vision-languaal pretraining. Notably, Single-Person setups tend to outperform Multi-Person ones, confirming that signer isolation reduces visual ambiguity and aids translation. These baselines provide strong references for future research on realistic, scalable, and context-aware SLT in broadcast news environments.
H ADDITIONAL DISCUSSION
H.1 1 EMPIRICAL JUSTIFICATION FOR TEMPORAL SEGMENTATION DESIGN
In the data construction phase, we adopt fixed 10-second sliding window strategy for segmenting fingerspelling sequences. Here, we provide the rationale and empirical validation for this design choice from three perspectives: sequence coverage, training robustness, and performance comparison.
Coverage of Long-tail Durations. Fingerspelling sequences in BANZ.-FS exhibit significant variance in duration across different domains. While many isolated segments in the Lab subset are relatively short (<) 1.5s). naturalistic continuous fingerspelling
the Web subset significantly longer. As detailed in Table 8 8, continuous clips from YouTube have average duration of 6.25 seconds, with some instances significantly exceeding this length. A shorter window (e.g., 5 seconds) would pose a high risk of truncating these naturalistic, long-tai tail sequences, leading the loss of critical start/end tokens and visual context.
Robustness via Temporal Randomness. Our sliding window approach is designed to capture 10-second context around any detected fingerspelling segment without strictly centering it. This design allows the target sequence to appear variable temporal positions within the window. This introduces an implicit form of temporal data augmentation during training. By exposing the model to varying temporal offsets, we encourage the learning of translation invarrian features, thereby improving robustness against the imprecise temporal proposals often encountered in real-world detection scenarios.
Table 11: Ablation study on sliding window size for Fingerspelling Detection (AP@loUg.)..
Window Size
News
Lab
Web
Full
secunds
53.4
82.7
45.9
66.2
10 seconds
53.9
82.7
47.3
66.9
25

---

Published as a conference paper at ICLR 2026
Cross Doman Featare Dearbodon P-SN
IV Contation Natre Janblgun Grogpn
Spord Acceney Andlyds
Proprpoting
Figure 9: Comprehensive Analysis of Domain Variability and Error Sources. (a) Cross-Domain Feature Distribution: t-SNE visualization of video features reveals distinct clusters for News (Blue) and Lab (Green), while Web data (Red) exhibits high variance, confirming the domain gap. (b) Confusion Matrix: Highlights specific visual ambiguities. (c) Speed vs. Accuracy: Empirically quantifies the negative correlation between signing speed and recognition accuracy, identifying rapid transitions in News data key bottleneck.
Ablation Study: 5s vs. 10s Window. To empirically validate the impact of window size, we conducted an ablation study comparing the performance of the baseline detection model (SL-Seg) trained with 5-second window versus our proposed 10-second window. As observed in Table 11, the 10-second window yields consistently superior performance compared to the 5-second setting. The performance degradation observed with the 5-second window confirms that the larger temporal context is essential for handling variable-lengee in-the-wild data, while simultaneously avoiding truncation errors.
H.2 VISUALIZATION OF -DOSS- FEATURE DISTRIBUTION
To qualitatively analyze the domain variability and generalization challenges discussed in Section 5.2, we conducted a t-SNE (Maaten & Hinton, 2008) visualization of the video feature representations extracted from our three data sources: ABC News, Lab Recordings, and Web Data. We utilized the same HandReader (RGB) (Korotaev et al., 2025) backbone used our benchmark experiments to extract high-dimensional features from randomly sampled subset of video clips, which were then projected into a 2D space.
As illustrated in Figure 9(), the feature distributions exhibit clear patterns that corroborate our quantitative findings. Specifically, the ABC News subset (Blue) and Lab Recordings (Green) form distinct, tightly grouped clusters, reflecting the standardized studio environment and professional lighting typical of these controlled settings. stark contrast, the Web Data (Red) exhibits a highly dispersed distribution with significantly larger variance; notably, its manifold spatially surrounds and partially overlaps with the Lab cluster but extends into sparse regions of the feature space. This visualizes the "in-the-wild" nature of the Web subset, encompassing wide spectrum of visual conditions from simple setups to highly complex scenarios. This high variance visually justifies the performance drop observed in cross-domain benchmarks and underscores the necessity of diverse training data for robust fingerspelling recognition.
H.3 QUANTITATIVE ERROR ANALYSIS
To complement the qualitative case studies in Section 5.3 and provide deeper diagnostic insights into model failures, we conducted two quantitative analyses: per-letter confusion analysis and speed-performance correlation, as illustrated in Figure 9.
As shown in Figure 9(b), the confusion matrix verifies distinet error patterns specific to the two- handed BANZSL system. A significant source of error arises within the consonant group, particularly between 'M' and 'N', This misclassification is attributed to the subtle visual difference finger counting (three vs. two fingers), which is easily obscured by motion blur self-occlusion during continuous signing. Additionally, we observe mutual confusion among the vowel group ('A', 'E', 'T', 'O', *U). Since BANZSL vowels involve contacting specific fingertips of the non-dominant hand, rapid articulation often leads to imprecise contact localization, causing the model to struggle in distinguishing these spatially adjacent classes.
26

---

Published as a conference paper at ICLR 2026
To amalyze the impact of temporal dynamics, Figure 9(c) plots Letter Accuracy against Fingerspelling Speed. A clear negative correlation is observed: as signing speed increases, recognition accuracy consistently declines. The data distribution reflects Nomain-specinf challenges. The Lab recordings cluster in the low-speed and high-aceuracy region, serving as an upper bound. contrast, the News domain is characterized by high speed and moderate accuracy, confirming that rapid inter-letter transitions significantly degrade performance for professional signers. Finally, the Web domain suffers from lower accuracy regardless of speed, indicating that visual factors such lighting and background clutter, rather than speed alone, are the dominant sources of error in in-the-wild settings.
H.4 STRATIFIED ANALYSIS OF FACTORS AFFECTING FINGERSPELLING PERFORMANCE
To better understand which factors influence fingerspelling detection and recognition BANZ-FS, we conduct stratified analyses across dialect, signer profile, and letter frequency.
Dialect and context. We first examine performance across the three BANZSL dialects. The BSLNNZSL-dominant Web subset is more challenging than the Auslan-dominant News and Lab subsets, which is consistent with its smaller annotated scale and greater contextual variation. Training on the combined Full dataset improves performance on the Web test set relative to training on Web-only data (Table 3), suggesting that enlarging the multilingual training pool benefits cross-dialect generalization.
Signer profiles. We further analyze performance by signer type and signing speed. As reported in Table 2, News interpreters fingerspell substantially faster (4.59 characters per second) than Lab and Web signers (approximately 1.30 characters per second). Correspondingly, models achieve the highest accuracy on the slower Lab data and the lowest on the rapid, highly fluent News data.
Letter frequency. To analyze how letter frequency affects recognition accuracy, we stratify perfor- mance by individual characters using the HandReader model trained on the Full dataset. Table 5 reports the accuracy of the most and least frequent letters. Frequent characters such as "N" and "A" reach approximately 83% and 80% accuracy, respectively, whereas characters such as "X" and "J" achieve around 54% and 41%. These results demonstrate clear correlation between letter occurrence and recognition accuracy and highlight the long-tail challenge in BANZ-FS. Future extensions of the dataset will explicitly target low-frequency characters to improve model robustness.
I CASE STUDY FOR BANZ-FS FINGERSPELLING
Figure 10 illustrates qualitative examples from our BANZ-FS dataset, comparing ground truth annotations with predictions from three systems. Across most examples, FSD-IFSR demonstrates accurate segmentation and character-level recognition, closely matching the ground truth, especially in clear and isolated contexts. However, recognition becomes more challenging in broadcast news settings, as shown in the top-right example. The model confuses the final "N" with "M". likely due to their similar handshapes and coarticulation under fast signing. This highlights the visual ambiguity of adjacent characters in rapid sequences. Another common еттог arises from temporal proximity of repeated letters. In the bottom-right example, the system fails to distinguish the two or three "0's near the end, merging them into single instance. This suggests the need for improved temporal modeling to separate closely spaced, visually similar gestures.
J CASE STUDY FOR AUSLAN-DAILY NEWS SIGN LANGUAGE TRANSLATION
Table 12 showcases qualitative examples from our Auslan-Daily News V2 translation benchmark. We compare ground-truth sentences with outputs from SL--Luong Pose (Camgöz et al., 2020) and GFSLT-VLP (Zhou et al., 2023) models. Both models perform well on short, common sentences, but longer or more complex utterances reveal clear differences. GFSLT-VLP captures more complete sentence structures and preserves key semantic information better than the baseline.
K LLM USAGE STATEMENT
Large Language Models (LLMs) such as ChatGPT are used as general-purposs tools to improve readability and clarity of the manuscript, e.g., for grammar checking, LaTeX formatting, and restruc-
27

---

Published as a conference paper at ICLR 2026
Ground Truth
IFSR
FSD
ESD-IESR
O'
G
Ground Truch
IFSR
S
S
FSD
ESD-IESR
S
Ground Truth
IFSR
8" 0 8
FSD
FSD-IFSR
Figure 10: Case study comparing IFSR, FSD, and FSD-R on fingerspelling sequences.
Table 12: Case study. We highlight exactly correct translations in red and missing contents in blue.
hello and welcome abe news.
SL-Luonn Pose (Camgüi 2020)
hello and welcome abe
GFSLT VLP (Zhou al., 2023)
hello and welcome to abe news
GT
do you think thing have significantly change in the last year.
SL-Lwong + Pose (Camgöz et al., 2020)
you think there escalate
GESLT-VLP (Zhou al., 2023)
do you think
signlicantly change the last year
mortheast.
St.-Tuong Pose (Camgliz et ol., 2020)
rain and coal in the east to new south
GFSLT-VLP (Zhou al., 2023)
rain and cool northeast
GT
the prime minister have reject suggestion he redefine his
SL-Loong (Camgii 2020)
prime minister new suggestion
GFSLT VL. (Zhon al., 2023)
the prime minister have reject suggestion that he
turing sentences. No parts of the research idea, dataset design, experimental results are generated influenced by LLMss. All technical contributions and conclusions are solely those of the authors.
28


Reviews:
[{"review_id": "QwNC7XG9GL", "reviewer": "Reviewer_DWDk", "summary": "This paper introduces BANZ-FS, a dataset for two-handed BANZSL fingerspelling, combining ABC News Auslan broadcasts, lab multi-camera RGB-D recordings, and web-sourced videos. The dataset contains >35k annotated fingerspelling segments from 116 signers, with multi-level alignments (video ↔ subtitles ↔ letters ↔ lexicon). The authors define and benchmark four tasks (IFSR, FSD, FSD-R, FSR-Context), evaluate pose- and RGB-based baselines, present cross-domain experiments, and report qualitative case studies.\n\n**Overall:** the dataset is useful and fills a real gap; however, the manuscript needs clearer documentation of procedures, metrics, and ethics before the benchmark claims can be fully trusted.", "questions": "Most of my concerns and suggestions are already reflected in the Weaknesses section. I believe that clarifications or additional experiments addressing these issues would substantially strengthen the paper.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "**Presentation:** The paper is clearly structured (Intro → Related Work → Data Collection → Statistics → Tasks → Benchmarks → Case Studies). Tables summarizing dataset comparisons and statistics (Table 1 & Table 2) are informative and well-labeled. The taxonomy of tasks (IFSR, FSD, FSD-R, FSR-Context) is clearly defined and consistent throughout.\n\n**Contribution:** The proposed BANZ-FS dataset addresses a tangible linguistic and practical gap by focusing on two-handed BANZSL fingerspelling, a setting largely underrepresented in current literature. Its annotation depth (multi-level temporal alignment, OOFS/singleton reporting, and FS speed distribution) and multi-domain composition (news, lab multi-view, web) provide meaningful diversity and reusability for the community.\n\n**Techniques:**\n\n1. A well-designed multi-domain acquisition strategy combining professional broadcast data, lab-controlled RGB-D recordings, and in-the-wild web footage, enabling both realism and controlled benchmarking.\n\n2. Rich, fine-grained annotations supported by comprehensive statistics (e.g., long-tail letter frequency, signer diversity, OOFS/singletons, FS speed).\n\n3. A complete benchmark suite spanning isolated recognition, detection, joint detection-recognition, and contextual evaluation, covering both constrained and realistic use cases.\n\n4. Practical qualitative case studies that transparently illustrate typical failure modes such as rapid fingerspelling or loose boundary segmentation, adding diagnostic insight to the quantitative results.", "weaknesses": "**1. Segmentation design lacks empirical justification:** Section 3.3 adopts a fixed 10s sliding window without ablation or rationale. An analysis of alternative window lengths (e.g., 5s, adaptive) is needed to confirm robustness.\n\n\n**2. Annotation reliability not quantified:** Although multi-level verification is mentioned (Section 3.1), no inter-annotator agreement (e.g., κ, α) or adjudication detail is provided. This weakens confidence in temporal boundary quality.\n\n\n**3. Evaluation criteria under-specified:** Metrics such as AP@IoU and AP@Acc (Section 5.2) lack exact definitions and sensitivity checks. Clear reporting of thresholds and pipeline details is essential for reproducibility.\n\n\n**4. Cross-domain variability insufficiently analyzed:** Results in Table 3 - 4 reveal strong domain gaps, yet no normalization or domain ablation is provided. A domain-wise feature analysis (e.g., t-SNE) would clarify generalization behavior.\n\n\n**5. Error analysis remains anecdotal:** Case studies (Section 5.3) illustrate failures qualitatively but lack quantitative breakdowns, e.g., per-letter confusion or temporal precision - which would turn examples into diagnostic insight.\n\n**6. Signer-split clarity:** Section 3.3 does not specify whether the splits are signer-disjoint. Please clarify whether signer identities overlap across splits, and report this information explicitly.\n\n**7. Ethical transparency missing:** No explicit consent, license, or release protocol is reported, though videos include identifiable individuals. An ethics statement is required for dataset release.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9N59wQBMxx", "reviewer": "Reviewer_YuAS", "summary": "The paper introduces BANZ-FS, a new, large-scale dataset focused on two-handed fingerspelling for BANZSL (British, Australian, and New Zealand Sign Language). The authors highlight that existing sign language datasets rarely address the specific challenges of two-handed fingerspelling, focusing instead on single-handed systems like American Sign Language (ASL). The dataset features rich, multi-level annotations, including video-to-subtitle, video-to-letter, and video-to-lexicon alignments. It is designed to capture real-world linguistic phenomena like spelling errors, acronyms, and abbreviations , as well as visual challenges unique to two-handed signing, such as self-occlusion and rapid transitions. The authors benchmark state-of-the-art models on tasks like fingerspelling detection, isolated recognition, and recognition in context, demonstrating that the dataset presents significant challenges for current methods.", "questions": "See above", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Novel and Specific Contribution: This is the first large-scale fingerspelling dataset specifically for the two-handed BANZSL system. It fills a critical gap, as most resources are for single-handed systems.\n\n2. High Data Diversity: By combining news, lab, and web sources, the dataset captures a broad spectrum of signing styles. This includes formal signing from professionals, clean signing in controlled settings, and casual \"in-the-wild\" signing from 116 unique signers.\n\n3. Linguistic Realism: A major strength is the inclusion and annotation of real-world linguistic phenomena often missing from clean datasets. This includes spelling errors (e.g., \"Maguire\" signed as \"Maquire\"), lexical abbreviations (\"EQ\" for \"equipment\"), acronyms (\"GWS\"), and self-corrections.\n\n4. Rich, Multi-Level Annotations: The dataset provides comprehensive annotations beyond simple transcription. This multi-level alignment supports a variety of tasks, including detection (temporal boundaries), isolated recognition (letter sequences), and contextual recognition (alignment with spoken language lexicons).\n\n5. Robust Benchmarking: The authors establish strong baselines by testing existing models on the new data. The results (Tables 3 & 4) show that current methods struggle with cross-domain generalization (e.g., training on Lab data and testing on Web data), proving the dataset is a challenging and valuable resource for advancing the field.", "weaknesses": "1. The methods evaluated in Tab.3 and Tab.4 are a bit old, which are mostly before 2023. For a paper submitted to ICLR 2026, new state-of-the-art methods could be incorporated.\n2. As shown in Tab.2, the dataset is \"heavily skewed toward Auslan\" (Australian Sign Language). The BSL (British) and NZSL (New Zealand) contributions are \"relatively limited\" , primarily coming from the smaller \"Web Videos\" and \"YouTube\" sources.\n3. While the proposed dataset highlight the \"in-the-wild\" characterisitic, this is not fully reflected in the data construction and composition. Most videos are still captured from News or Lab environments.\n4. Some labelling steps remain confusing. For example, how to (2) identify the signer ID based on pose trajectories? Besides, only a little data is cross-checked. How to guarantee the data quality is a question.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CzDFtsEizM", "reviewer": "Reviewer_qoMh", "summary": "The authors introduce BANZ-FS, a new dataset for two-handed fingerspelling detection and recognition. The dataset originates from three sources, including web recordings, news broadcasts, and lab recordings in a controlled setting. Over 35k video-aligned instances are collected, aligned on a video-subtitle, video-fingerspelled letters, and video-target lexicon level.\n\nThis paper tackles a particular shortcoming of other papers in the field of fingerspelling recognition, which is that the datasets used in this field are too simple and not representative of real-world fingerspelling. That is, they are often based on images of individual fingerspelled letters, instead of videos of sequences of fingerspelling. The latter are far more difficult to recognize accurately due to co-articulation and fast spelling (and due to the fact that it's video data, not image data). In contrast, the former is a trivial task.\n\nThe authors detail their data collection, cleaning, and labelling techniques for the web data in sufficient detail. Subtitles, which are noisy annotations, are cleaned by the authors to obtain 30k cleaned subtitles. Then, signing experts verify alignment of subtitles and video data, refining it if necessary. Annotations are cross-checked by other annotators (5%). \n\nLab data is collected with multiple RGB-D cameras, and participants were properly informed of their rights according to an ethical protocol. Signing levels were recorded, though it is not immediately clear if hearing status correlates to signing level.", "questions": "How are spelling errors in labels handled? Should they remain or not?\n\nPage 5. For the web dataset, you identify signer IDs based on pose trajectories. Did you use their Pose Flow system? If so, I would mention this explicitly in the paper.\n\nPage 5. You say that the fingerspelling segments are annotated. How is this done? Are segments delineated in time? Is the entire word annotated? Do you keep the intended spelling or the spelling as it is (including possible errors?)\n\nPage 5. Can you explain what \"lexical forms of fingerspellings\" are?\n\nPage 6. How does hearing status relate to signing level?\n\nPlease perform a check for grammar and spelling. For example, on page 3, you write \"recognition recognition\" and in the appendix you misspelled \"Isolated\".\n\nDid you in any way use the depth information from the lab setup? Would this have been useful? How do you combine information from multiple cameras?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper is clearly written and easy to follow.\n\nThe authors evaluate multiple models from the literature on their datasets, leading to a rigorous evaluation of not only the dataset, but also of these models. This is a valuable contribution to the field.\n\nThe authors also list several fingerspelling related tasks and evaluate models for each of them, which is also valuable. In fact, this makes this paper a good introduction to the field of fingerspelling recognition.", "weaknesses": "I would have liked to read more information on the datasets. More information is given in the \"Questions\" section below. But, for example, \"volunteers\" were selected for fingerspelling data collection. Were these volunteers in any way checked for signing proficiency? What if they submitted particularly bad data? Fingerspelling is not trivial at all.\n\nSimilarly, the lab collection of data could have had more information. For example, several RGB-D cameras are used. Are all data used? What about depth? This section could be expanded.\n\nIt is also not clear if spelling errors were kept in annotations or not. This seems like an important aspect to mention.\n\nThere are several spelling mistakes throughout the paper, but these could easily be fixed if accepted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3P9HvnKcKH", "reviewer": "Reviewer_55F9", "summary": "Here, the authors present a new dataset and benchmarks for fingerspelling detection and recognition, focused upon the two-handed manual alphabet common to the British, Australian and New Zealand Sign Languages, arguing that this covers a gap in fingerspelling datasets that largely focus upon one-handed manual alphabets (e.g. American Sign Languages). The authors curate and annotate fingerspelling segments from three sources (Auslan interpretations from a news broadcast, YouTube videos, and a lab-collected dataset generated by the authors), and assess both in-domain and out-of-domain performance across models.", "questions": "(See weaknesses)", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The authors have curated a comprehensive dataset that addresses a critical gap in fingerspelling detection and recognition research: their curation of not just a large quantity of videos, but through different sources and annotated with various metadata is valuable. The authors correctly note that two-handed manual alphabets pose additional complications not seen in one-handed manual alphabets (e.g. occlusion), and so it is essential that they're releasing this resource. The benchmarks are comprehensive, spanning numerous state-of-the-art models, but also critically assess performance both within and outside of domains. This is overall strong data collection and benchmarking work.", "weaknesses": "1. Although the authors have gone to lengths to collect metadata about segments/videos, I wish there was a more thorough analysis of this. In Tables 3/4, the authors report overall performance, but it would have been really interesting to conduct stratified analyses of what factors impact fingerspelling recognition and detection performance beyond domain. For example, is detection/detection+recognition more challenging with NZSL and BSL, given that the contexts in which fingerspelling occurs is different than the majority of their training data across domains? Are tasks more difficult on Deaf signers, who might fingerspell faster/more fluidly than language learners? Are infrequent letters worse in their performance?\n\n1a. The authors claim that their dataset spans three sign languages - but at least one dataset are exclusively Auslan. It's moreover unclear how much of the dataset is BSL/NZSL - the authors should report this, especially because it is central to their claim that their dataset spans three languages. Although the authors do acknowledge their dataset is imbalanced - to what degree? This should be reported.\n\n2. More details about the stratification of datasets would be critical and are currently missing from the paper. Are the splits a purely random split, or is there some attempt to stratify within each data source such that scenarios/signers are unseen? If the splits are purely random, this might impact interpretation: e.g. if the same signer is in both the test and training dataset, and fingerspelling the same word, in-domain performance may be overinflated by interpretation. \n\n3. The authors provide some examples of fingerspelling phenomena in Figure 2 (abbreviations, acronyms, spelling errors, and self-correction), but it's unclear to me how they actually deal with them in the dataset - I can see arguments both ways. On one hand, it might be important for fingerspelling recognition systems to report the fingerspelling as-is. On the other hand, people perceiving signing will naturally correct and contextualize these issues in language recognition - and critically, providing written language transcriptions of fingerspelling can strip the necessary context to correct (for example, in self-correction, signers will often shake their heads, make facial expressions, or pause to indicate a correction, and these gestures won't make their way into the written language transcription), so it may be important to provide a corrected form. In other words - \"recognition\" and \"translation\" are sometimes at odds. Can the authors better document their design decisions and rationale here on what we consider as ground truth? \n\n4. There are places where the authors could be more culturally sensitive. In line 228, the authors refer to the community as \"deaf and hearing-impaired\" viewers, whereas they use \"Deaf and hard-of-hearing\" in the introduction. The latter is generally considered more appropriate by Deaf communities. In Figure 3, the authors distinguish between \"Deaf individual\" and \"Auslan expert\", but these are not mutually exclusive categories.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)