Uni-Sign: Toward Unified Sign Language Understanding at Scale
Authors: Zecheng Li
Wengang Zhou
Weichao Zhao
Kepeng Wu
Hezhen Hu
Houqiang Li
Venue: ICLR 2025 Poster (2025)
Keywords: Sign language understanding
Pre-training
Large-scale sign language dataset
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Sign language pre-training has gained increasing attention for its ability to enhance performance across various sign language understanding (SLU) tasks. However, existing methods often suffer from a gap between pre-training and fine-tuning, leading to suboptimal results. To address this, we propose Uni-Sign, a unified pre-training framework that eliminates the gap between pre-training and downstream SLU tasks through a large-scale generative pre-training strategy and a novel fine-tuning paradigm. First, we introduce CSL-News, a large-scale Chinese Sign Language (CSL) dataset containing 1,985 hours of video paired with textual annotations, which enables effective large-scale pre-training. Second, Uni-Sign unifies SLU tasks by treating downstream tasks as a single sign language translation (SLT) task during fine-tuning, ensuring seamless knowledge transfer between pre-training and fine-tuning. Furthermore, we incorporate a prior-guided fusion (PGF) module and a score-aware sampling strategy to efficiently fuse pose and RGB information, addressing keypoint inaccuracies and improving computational efficiency.  Extensive experiments across multiple SLU benchmarks demonstrate that Uni-Sign achieves state-of-the-art performance across multiple downstream SLU tasks. Dataset and code are available at github.com/ZechengLi19/Uni-Sign.

Published as a conference paper at ICLR 2025
UNI-SIGN: TOWARD UNIFIED SIGN LANGUAGE UN- DERSTANDING AT SCALE
Zecheng Li', Wengang Zhou 1.2 Weichao Zhao's Kepeng Wu', Hezhen Hu Houqiang l
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China
2 Institute of Artificial Intelligence, Hefei Comprchensive National Science Center
3 University of Texas at Austin
(1 [lizecheng23, saruka, wukp)email ustc. eeu.. cn 2hwg,, @ustte edu. en, alexhu@utexas. edu
ABSTRACT
Sign language pre-training has gained increasing attention for its ability to en- hance performance across various sign language understanding (SLU) tasks. However, existing methods often suffer from a gap between pre-training and fine- mning, leading to suboptimal results. To address this, we propose Uni-Sign, unified pre-training framework that eliminates the gap between pre-training and downstream SLU tasks through large-scale generative pre-training strategy and a novel fine-tuning paradigm. First, we introduce CSL-News, a large-scale Chinese Sign Language (CSL) dataset containing 1,985 hours of video paired with textual annotations, which enables effective large-scale pre-training. Second, Uni-Sign unifies SLU tasks by treating downstream tasks single sign language transla- tion (SLT) task during fine-tuning, ensuring seamless knowledge transfer between pre-training and fine-tuning. Furthermore, we incorporate a prior-guided fusion (PGF) module and score-aware sampling strategy to efficiently fuse pose and RGB information, addressing keypoint inaccuracies and improving computational efficiency. Extensive experiments across multiple SLU benchmarks demonstrate that Uni-Sign achieves state-of-tt ffte--of-hhe-art performance across multiple downstream SLU tasks. Dataset and code are available at ggihub.com//echengg11///0/
INTRODUCTION
Sign languages are the primary means of communication for the Deaf/Hard of Hearing individ- uals, conveyed via hand gestures, facial expressions, and movements (Braem & Sutton-Spence 2001]. Considering the critical benefits for barrier-free communication between Deaf/Hard of Hear- ing and hearing communities, sign language understanding (SLU) has been extensively studied for decades (Camgoz al. 2018 Yin et al., (02T). SLU presents unique challenges that necessitate a comprehensive understanding of the visual cues embedded in individual sign signals, well as the distinctive linguistic rules sign language. SLU can be subdivided into several sub-tasks, including isolated sign language recognition (ISLR), continuous sign language recognition (CSLR), and sign language translation (SLT). ISLR concentrates on classifying individual sign language movements, while CSLR aims learn the alignment of sequences between sign language and their correspond- ing glosses. In contrast, SLT requires the model to generate textual descriptions corresponding to sign language sequences. All these tasks impose indispensable demands on the model's fine-grained comprehension and context awareness capabilities.
Recently, more and more studies have shifted their attention towards the exploration of pre-training techniques for SLU, which benefit from large-scale data to learn discriminative representations. One main thread attempts utilize large-scale self-supervised learning to unleash the statistics in unlabeled data (Hu et al. [2211a] 2023a Zhao et al., (2024)). SignBERT+ designs self-supervised learning strategies
manner. Although these methods demonstrate
Corresponding author.

---

Published as a conference paper at ICLR 2025
(4) Previous SOTA large-senle pre-training methods
(c) Comparative evaluation
MSASLIOP CSLB F-D
Sign Language Backshone
Pretext
pe
60.55
Modhle
Largesceat decoso
Pararetes interiance
Langunge Brcebure
Lrs
Module
Downstrene
WLASLIOB
lioestign (SL3)
(ISLI
(b) Ours
Pre-tranig
Sign Langunge
Pretest Task Moduie
Racéhhre
Lorgeosaie
Pacanetes interisnce
Paranaters ateritnce
WLASSL2000
(ISLI
26/8
Langngge
CSL-Daly
Bockhone
Module
Oars
MSLU
CRRL
SiguBERT~
Figure Comparison of paradigm and performance between previous SOTA pre-training meth- ods and ours. Lpt- Lis, and Lim represent the pretext-task loss, task-specific loss, and language modeling loss, respectively. Our method could mainly adopt the pre-training parameters and uni- fied fine-tuning paradigm, which narrow the gap between pre-training and fine-tuning and therefore embeds versatility capability on multiple benchmarks across different downstream tasks, including ISLR, CSLR, and SLT.
promising improvements in SLU tasks, they primarily focus on capturing visual cues from massive pre-training sign language data and lack joint modeling the textual information, causing the gap with downstream task like SLT. To tackle this issue, some methods try to directly leverage video- gloss/video-text pairs to conduct pre-training, such sign-to-gloss recognition (Chen et al. (2022b) video-text contrastive learning (Zhou et al.) 2023), and pseudo sooudo-loss prediction (Wong et al. 2024
Despite the incorporation of gloss/text data has been proven effective, they are generally limited by the scale of the video-glosse sadeo-gloss/vidco-teer paired data or the transferring capability of downstream tasks.
To address these challenges, we introduce unified pre-training framework that eliminates the gap between pre-training and downstream tasks, while operating at scale. As shown in Figure (a,b), unlike previous pre-training methods, Uni-Sign utilizes generative pre-training on large-scale datasets, enforcing the model to capture the semantics embedded in sign language. Our approach consists of two key innovations. First, we introduce CSL-News, a large-scale Chinese Sign Lan- guage (CSL) dataset that spans 1.985 hours of videos with corresponding textual annotations, sig- nificantly surpassing existing CSL datasets in size and diversity. This dataset provides the foundation for large-scale pre-training. Second, we propose Uni-Sign, pre-training model that unifies SLU tasks by treating downstream tasks as a single SLT task, ensuring seamless knowledge transfer be- tween pre-training and fine-tuning. To further enhance performance, integrate prior-guided fusion (PGF) module and a score-aware sampling strategy, addressing keypoint inaccuracies and improving computational efficiency. In summary, our contributions are follows
We propose unified pre-training framework, Uni-Sign, that achieves state-of-the- performance across SLU tasks by eliminating the gap between pre-training and downstream tasks.
We introduce CSL-News, large-scale dataset with 1,985 hours of Chinese Sign Language videos and text annotations, enabling effective large-scale pre-training for SLU.
We unify the pre-training and fine-tuning paradigm with shared objectives and incorporate a prior- guided fusion (PGF) module and score-aware sampling strategy, which further improve perfor- mance by addressing keypoint inaccuracies and balancing speed with accuracy.
RELATED WORKS
2.1
SIGN LANGUAGE UNDERSTANDING
SLU encompasses various research fields, including ISLR, CSLR, and SLT.
2

---

Published as a conference paper at ICLR 2025
ISLR and CSLR. ISLR focuses on classifying sign language movements. Previous works (Hu et al. b Li et al. 2020b; Zuo et al. 2023) have achieved superior performance by utilizing tai- lored models. CSLR aims to learn the sequence alignment between sign language and sign glosses, where each gloss is a manual transcription for a sign. Thanks to the capability of Connectionist Temporal Classification (CTC) loss (Graves al. 2006) to effectively handle the alignment of two unsegmented sequences without precise alignment, it has become a mainstream approach in recent
years (Pu al. 2020 Min al. 2021 Hu al. 2023b 2024 Jiao et al. 2023).
SLT. SLT requires the model to generate the corresponding text sequence by fully understanding sign language. It can be divided into two paradigms, gloss-based and gloss-free. By employing the gloss-based paradigm, the model acquires intermediate representation of glosses, leading to improved text generation capabilities. SLRT (Camgoz et al., 2020) pioneers the application of a transformer encoder-decoder framework in SLT and incorporates gloss-level supervision into the transformer encoder through CTC loss. STMC-T (Zhou et al.) 2022) tackles the SLT through multi- cue modeling, SLTUNET Zhang et al. 2023a) and MMTLB (Chen et al. (022b) attempt to transfer knowledge from large-scale external text corpus and pre-trained language models into SLT to im- prove performance. However, the costly gloss labeling limits dataset and model scalability, prompt- ing researchers to shift their attention toward the gloss-fre free paradigm. GFSLT- VLP (Zhou et al.] 2023) novelly proposed text-video contrastive loss pre-train translation models, which signifi- cantly boosted the performance of gloss-free methods. Sign2GPT (Wong et al., (2024) and Sign- LLM (Gong et al.] 2024) aimed to take advantage of the linguistic knowledge inherent large language models (LLMs) to enhance gloss-frree SLT. In this paper, we also focus on gloss-free SLT, which is more challenging and easier scale up in terms of both the model and the dataset.
Unlike the task-specific methods, we introduce a unified framework to handle these SLU tasks. Meanwhile, by eschewing any task-specific designs during the fine-tuning phase, our method main- tains simplicity while consistently achieving remarkable performance across various SLU tasks.
2.2 SIGN LANGUAGE PRE-TRAINING
Sign language pre-training approaches leverage pretext tasks to capture semantic representations during the pre-training phase, resulting in notable performance improvements on diverse down- stream tasks. Some researchers (Hu et al., 2021a 2023a: Zhao et al., 2024b) attempt to leverage self-supervised learning to enhance representation capabilities from massive unlabeled data. No- tably, the series of SignBERT (Hu et al. 2021a 2023a) employs masking-and -ndreeeoststtruutiinn strategy to mine contextual information provements in SLU. However, these self-superviser sign language pre-training approaches primarily focus on learning low-level visual semantics while neglecting the acquisition of textual knowledge, resulting in gap with downstream tasks such SLT. Some studies have insightfully identified this issue and attempt to leverage cileo--glos/viii video-text pair data to inject linguistic knowledge into the pre-trained model. MMTLB (Chen al.) 2022b) achieves precise alignment between sign language and text by employing three sub-tasks (sign-to-gloss, gloss-to-text, and sign-to-text), thereby un- locking the potential of pre-trained language models. GFSLT-VLP (Zhou et al. 2023) proposes contrastive learning pretext task that effectively aligns sign language and text in a joint space, signif- icantly advancing the development of gloss-free SLT. Inspired by GFSLT-VLE MSLU (Zhou al.,
sign language, achieving promising performance im-
2024) and C²RL (Chen et al. 2024a) further introduce the pretext tasks of keypoint reconstruction and language modeling, respectively. Despite the effectiveness of incorporating gloss/text data, they are generally limited by the scale of the video-gloss/ dileo-gloos/vvieo-teex paired data or the transferring capa- bility of downstream tasks. YouTube-ASL. (Uthus al.) 2024) directly employs language modeling task for large-scale pre-traine teeirriinin, demonstrating the potential of generative pre-training and empha- sizing the importance of scaling datasets. In contrast prior pre-training methods (Hu et al.) 2021a] 2023a Zhao et al. 2023 Zhou et af.] 2023), we propose a framework that benefits from large-scale pre-training and a unified pre-training and fine-tuning paradigm, thereby fully unlocking the SLU potential of the pre-trained model and transferring it to downstream tasks.
2.3 UNIFYING VIA LANGUAGE MODELING
Inspired by the success of sequence- toaecee-t--equnne (seq2seq) modeling in natural language processing, previous studies (Chen et al. 2022a Wang al. 2022 employ seq2seq approaches unify vari-
3

---

Published as conference paper at ICLR 2025
Name
Language
Vocab.
Hours
Source
KETI (Ko al. 2019
KVK
419
28
Lab
SWISSTXI ( Camgoz et
DSGS
TV
VRI-RAW Cangoz
VGT
TV
PHOENIX-201 20141 Camgoz et
2018
DGS
3K
11
TV
DGS Corpus (Hanke ci ara 4020
DGS
50
Lab
BOBSL (Albante et 2021
BSL
77K
1.447
How2Sign
ASL
1GK
Lab
Video ReeeBumenneeee
OpenASI. er M02
ASL
33K
288
Web
206
YouTube-ASL (Uthus ct 2024
ASL
60K
984
Web
SP-10 (Yin al., 2022
various
Web
AfriSign (Guetwoou
2023b
various
20K
152
Web
CSL-Daily (Zhou 2021
CSL
2K
23
Lab
CSL-News (Ours)
CSL
5K
1,985
TV
Table Summary statistics for different SLT datasets.
Figure Distribution of video
durations and text lengths.
MALEE AN A.R.J.
TV program
38
(This equivalent to providing deaf individuals with personal sign language interpreter.)
TV program
5F #1 M
JR 1T
(New year, new hopos. With diligent offort a bright future awaits.)
Samples of videos and text annotations in the CSL-news dataset. The signer's face is masked in here to protect their privacy.
Figure 3:
ety of tasks. Built these advancements, vision LLMs (Li et al. 2023 Zhu et al. 2024 extend the
capabilities of LLMs to vision-language understanding by leveraging language modeling objective.
LLaVA (Liu et al. 2023) demonstrates remarkable multimodal Enstruction-folloiinn capabilities by utilizing the extensive world knowledge embedded in LLMs. VisionLLM-v2 (Wu et al., (2024) utilizes language modeling tackle hundreds of vision-language tasks, further highlighting the ef- fectiveness of the unified paradigm. Motivated by these advancements, we propose Uni-Sign, which aims address various SLU tasks via language modeling, while also achieving both simplicity and scalability.
2.4 SIGN LANGUAGE UNDERSTANDING DATASETS
Collecting large-scale and high-quality datasets is crucial for improving neural network perfor- mance and has been widely explored. For ISLR, various benchmarks (Joze & Koller 2019 Li
et al. 2020a Hu et al., 202 have been proposed to comprehensively evaluate model performance.
Phoenix2014 T-1 (Camgoz et 2018) CSL-Daily (Zhou al. 2021) are introduced to tackle CSLR
and SLT tasks. Although numerous efforts (Shi et al.) 2022; Duarte et al. [021 Yin et al.) 2022] Hanke et af.) 2020) have been made to develop high-quality datasets to boost SLU, the field still constrained by the size of the available datasets. BoBSL (Albanie et al. (2221) and YouTube-
ASL (Uthus et al., 2024) insightfully identified this issue and proposed a 1,447 hours British Sign
Language (BSL) dataset and 984 hours American Sign Language (ASL) dataset, respectively. Additionally, YouTube-SK SL-22 (Tanzer & Zhang, 2024) and JWSign (Gueuwou et al. 2023a) have also collected large-scale multilingual sign language datasets, which crucial for training unified multilingual sign language models. Previous works primarily focused on collecting BSL (Albanic
al. 2021) and ASI. (Shi al. 2022
Uthus et al. 2024 Tanzer & Zhang 2024) datasets, leaving
4

---

Published as a conference paper at ICLR 2025
Largescclle data curation
Progroms list:
Dataset characteristics:
(2) Primetime-Neus
(1)
FUNASR
Timestomps
Segment Crop video
video-ten pairs
resourcs: 751,320
Text amnatations
(3)
Large vecabulory:
Speech neegnilionn
Ouigat nealt
Estract sign language vodoo clip
Lage-Lagguge--i Nloes
Long duration:
Mowhh TV
(b) Unified pre-training and fine-turing
Pre-training phose:
PSLT: "(trondaaion)
Modality:
Multi-modal Fusion
Pre- traited Large
Fine-tuning phose:
(1) Pose: Sequence
Language Model
Lpp
ISL: "(ection description]
(2) RSB Video
CSLR: "(glesst) (gloss2) (glosshi"
Nadel taput
Sign language sadesizading model
Objective
S.T: "(translation"
Supertaed target
Figure 4: The overview our two key innovations: (a) Pipeline for large-scale data curation. (b) Unified pre pre-training and fine-tuning, utilizing pre-training parameters and single language model- ing loss to address diverse SLU tasks.
CSL datasets relatively underexplored. To fill this gap, we propose CSL-News, a 1,985 hours CSL translation dataset.
METHOD
3.1 LARGE-SCALE DATA CURATION: CSL-NEWS
Currently, the larger publicly available SLT datasets are mainly sourced from ASL (Shi et al.
2022
(02T). there still exists an urgent Uthus et al. 2024 Tanzer & Zhang 2024) and BSL. (Albanie need to collect a large-scale CSL dataset. As illustrated in Table CSL-Daily (Zhou et al. 2021) currently the largest existing CSL dataset which only contains a total duration 23 hours and insufficient to train robust CSL model. We therefore gather the CSL-News dataset, a large-scale SLT dataset with 1,985 hours of videos, approximately times larger than the CSL-Daily dataset.
To construct this dataset, we primarily collect four TV programs from three different TV station to construct our dataset. The duration statisties for each TV station are follows: CCTV-13, 1,342 hours; Dragon TV, 623 hours; and Hebei Radio and TV Station, 20 hours. After downloading the massive TV programs and considering the strong temporal alignment between sign language and news broadcasts, we employ the FunASR (Gao et al. 2023) toolkit to extract textual annotations from the audio. Subsequently, the news videos are segmented based on the timestamps of punctua- tion marks ( ? to generate video-text pairs. Finally, we crop the sign language videos using predefined relative coordinates to climinate background interference. Through these processes, we curate a large-scale CSL translation dataset that plays crucial role in the pre-training of large- scale CSL model. As shown in Figure 2, the dataset comprises video clips with an average duration of 9.5 seconds and an average text length of 40 words (Chinese characters) in total of 751,320 video clips. However, in this paper, we utilize only 722,711 video clips shorter than 512 frames for training. More discussion could be found in Appendix A.5 Figure illustrates the videos and text annotations within the CSL-News dataset, while Figure (a) presents the complete pipeline for large-scale data curation.
3.2 UNIFIED PRE-TRAINING AND FINE-TUNING
For training efficiency, the process is divided into three stages, Stage 1: Stage 2: RGB-pose interaction continue pre-training, and Stage downstream task fine-tuning. The framework of Uni-Sign is illustrated in Figure (a).
pose-only pre-training,
Preliminaries. Given 133 keypoints, we selectively utilize keypoints, categorizing them into three sub-pose groups: 21 for each hand, 9 for the body, and 18 for the face. The keypoints sequence
Commonn-Cnnerms
Primetime-News
News-30' Sign-Language-NNws
5

---

Published as a conference paper at ICLR 2025
of group is then processed by its corresponding pose encoder, producing gesture features F. E TR. TxNNNXCC Here, T represents the length of the pose sequence, N, denotes the number of keypoints in group i, C' is the dimension of the features, and i E {th, rh, b, f). In this paper, pose encoder of group is composed of a three-layer spatial GCN.
Following the idea of decoupling visual cues, the vision encoder focuses learning representa- tions from both hands rather than the entire image. To achieve this, videos cropped using keypoint coordinates and resized to 112 x 112 pixels processed by the vision encoder, producing vision | xx'xxxxxx' Notably, in Stage 2, we fuse J¹ and
features denoted as Fih E I xXxxxxxCC and Fr E IR" F. to compensate for the visual information lost due to inaccurate keypoints, obtaining the fused features F.. Specific details of the fusion module will be provided in Section 3.3 After those processes, we employ a three-layer ST-GCN (Ya et al. 2018) to construct the short-term temporal encoder. The features from each group (Fi or F.') are then fed into the temporal encoders, aggre- gated
via a mean pooling layer, and concatenated across all groups to produce the final feature Fsign E RR" Tx4C which is subsequently input the language model.
Pre-training Uni-Sign. Previous works (Chen et al. 2022b Zhao et al., 2024a Wong et al. 2024
Chen et al. 2024b Zhou et al. 2023) have designed indirect pretext tasks (e.g.. gloss-to-text transla- tion, pseudo-gloss prediction) to unlock the pre-trained language model's potential. Different from them, directly employ the generative pre-training paradigm to utilize the knowledge embedded within the pre-trained large language model. Specifically, we project the feature Feign to match the dimension of the language model and then feed it into the language model. The loss function as follows:
U
Lim =- Σ
log p( sulssus Fsign)
(1)
where su represents the u-th token, and Sea. denotes all preceding tokens in the sentence s. During the pre-training phase, we leverage Lim as the objective function, denoted as LPSLT. as depicted in Figure (a).
Fine-tuning Uni-Sign. Although there are specific fine-tuning methods tailored to each task (e.g.. ISLR employs MLP head for classification, while CSLR commonly utilizes CTC loss enforce temporal constraints), we innovatively treat ISLR, CSLR, and SLT as single SLT task, allowing us employ a unified fine-tuning paradigm to fine-tune all SLU tasks without the aels-and-whistlee as shown in Figure (b). To construct supervision targets, ISLR uses action description, CSLR employs sequences glosses separated by spaces, and SLT utilizes the translation text, denoted as Wword Vglosss Vxentence. respectively. Through this setting, the robust SLU capabilities integrated into the model during the large-scale pre-training phase will seamlessly transferred to downstream tasks, thereby unlocking the full potential of the pre-trained model.
In summary, the objective function of each phase are as follows:
Pre-training
{ [CPSLT = Lym( (Fsign: Vsentence)
CISLR Lim (Fsigns Blwrd)
(2)
Fine-tuming
LESLR = Lim (Frign Nglass)
LSL Lim (Frigns Vsentence
3.3 MULTI-MODAL FUSION
Prior-guided fusion. Multimodal networks (Jiang et al.) 2021 Chen al. 2022c; Zuo et al. 2023 Jiang et al. 2024) have been widely explored in SLU. However, most existing methods sim- ply perform apatial-temporty fusion (c.g., concatenation, cross-attention. without considering the fine-grained spatial relationships, which are crucial for narrowing the representational gap between modalities. Hence, propose a prior-guided fusion (PGF) module that leverages keypoint coordi- nates as priors to model fine-grained spatial consistency between modalities, as illustrated in Figure 5 (b). Given F and F,,,, where (lh, rh). we first employ multi-head attention module to incorporate the global RGB information into Fü, Then, by utilizing the keypoint coordinates J as priors to initialize the reference points in deformable attention (Xia et al. 2022), fine-grained spatial

---

Published as a conference paper at ICLR 2025
Overview framework
Prior-guided fusion module
find your do first, read
you enjoy
Pase-oely
--- RGk pato
Defouth
Pre-tralted I. arge L. Language M odel
Festure Concalmuaion
Lee
Vision Euonder
Podfico trit
Defommable Ataention
Resiation Clp
Tempocal Enooder
Tempori
Temponi Eacoder
Scare Sompling
Mull-bead ttnnton
PCF Module
Pose
Royposts Sepunce
Sub- para
Figure 5: (a) The framework of Uni-Sign. In the pose-only setting, the keypoints divided into sub-pose (hands, face, and body) and fed into pose encoders and temporal encoders to capture the fine-grained visual cue. Subsequently, features from each part at the same time step are concatenated along the feature dimension and processed by a pre-trained large language model to generate text. In the RGB-pose setting, score-aware sampling strategy is introduced to sample low-confidence frames and crop the corresponding hand regions. The hands are further encoded by vision encoder, interacting with hand pose features through PGF module to mitigate the impact of inaccurate keypoints. (b) The overview of PGF module, which fuses RGB and pose features frame by frame.
modeling across modalities is achieved. The fused features denoted as F Finally, Fi, and Pi, are fed into a gater to expedite convergence during Stage 2 training. The implementation details follows:
Gate k.t. Fl,.), F = (1 g) F., g
(3)
where Gate is gate module initialized to zero, aimed to preserve the knowledge learned in Stage the beginning of Stage 2. The notation [+, indicates the concatenation operation. Although {F F are not fused with RGB, we also convert their notation
{F F.]} for ease of expression.
Score-aware sampling strategy. Although RGB-pose fusion compensates for the visual cues lost due to inaccurate keypoints, the inclusion of RGB poses a significant challenge to computational resources. Considering the information redundancy between RGB and high-confidence keypoints, we propose a score-aware sampling strategy that selectively chooses RGB frames corresponding to low-confidence keypoints, thus balancing performance and speed. To this end, confidence hand keypoints as the reliability score rs, subsequently calculating the sampling score 1 rs. Next, we randomly sample P Peamp% % of RGB frames based these sampling scores. Finally, by employing indexing, the sampled RGB frames are efficiently interacted with their corre- sponding pose features. The relevant pseudocode is presented in Appendix A.3
the average
4
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Config
Stage
Stage
Stage
For Stage 1 and Stage 2, we utilize CSL-News and YoLL (Uthus et al. 2024) as pre- training datasets for CSL and ASL, respectively. In Stage 3, fine-tuning conducted separately for each downstream dataset. We implement Uni-Sign ing Py Torch (Paszke et al. 2019). employing mT5- Base [ue al. 2021) pre-trained language
optimizer buse learning rate weight decay optimizer momentum learning rate schedule training epochs butch
AdamW 3c-4
le.4
AL. 9,009, 0.999
cosine decay
4
gradient accumulation
8
Table Training recipe each stage.

---

Published as a conference paper at ICLR 2025
Methal
Modallis
CSI -tally
Poo RGB
P-1
P-C
P-1
P-C
P-1
P.C
P-1
P:C
Pos: RGB
Dey
Texi
ST.ONN Narstal
50.78
51.62
34.40
32:53
50.73
51.62
34.40
32.53
MSLU AI S
22.9
77.68
28.3
$0.96
58.42
54.87
43.50
SignöT
Sig
$4.06
85.23
02.42
60.15
79.34
50.72
46.35
46.57
31.3
50.7
MSUL
91.54
53.28
71.33
$9.26
19.25
15.29
50.20
SEN Ri 1i RUAS
31.1
30.7
HMA Poreett b04dy
73:45
11.55
25.10
66.27
37.91
33.00
Commer BAA surds
306
30.1
NLA-SR (30
91.04
72.56
60:35
91.47
92.17
63.85
T5. SLR Clen al
25.4
25.3
Sign
(Dars)
12.78
Uni-Sipn
UniiSig (Dam))
93.79
78.16
76.99
02.67
63.92
61.32
Uti-Sign (Ous))
26,7
26,0
Table 3: ISLR results on various benchmarks. & denotes meth- ods reproduced by (Hu al. 2021a). Blue and Green denote the best results of previous methods and ours, respectively.
Table 4: CSLR results on CSL-
Daily dataset with WER scores.
Mlchad
Row 90
Noeelity
10
Bey
RJ REPII
Teit
RONGE
Methed
Pour RCB
BLEUE
INTUGE
SLEURT
$1.45
1.:4
13-65
37.55
56.26
A01
11.0
12.4
72.
46.6
51.46
00
81.81
$1.82
71.33
13:11
34.0
9.0
30.01
02
35.31
23/01
35.25
23.1
9.6
27.0
75 111
55:31
35.75
15.13
17.90
35.50
25.15
SigsMalio
501
41.4
16.3
35V7-512
1144
41.3
155
35.4
31:00
33.51
1997
93.3
16.9
36.8
GAMP
1.00
30.25
16-10
11:00
35.53
34.33 37
11:00
14-4
ChrT--W Saraal Joad
21,56
7.05
21.25
35.35
Sux20F
37.33
13.41
3122
Cor, CRI Sepee Be hilk
16:11
4.58
16.00
35.55
42.45
1333
33.38
33.35
15755
3601
Opeak
5,99
21/2
31.09
Clk
14.33
31.11
28.11
ClL Laad
31,66
11.3
$1.36
Ue Signkkn)
$3.24
55.33
51334
51.65
35.11
51.00
00.00
26-38
04.00
54.84
16.96
16.1
Uni-Sige
8035
23.34
40.22
68.40
Table SLT results on CSL-Daily dataset. f and $ denote methods reproduced by (Zhou et al. 2021) and (Zhou et al. 2023). respectively. Underline indicates the best gloss- bassd SLT result.
Table 6: fress-free SLT results on How2Sign and OpenASI. indicates methods repro- duced by (Shi et al. 2022).
model. The mT--Base model benefits from pre-training on the mC4 (Xue et al. 2021 corpus, which enhances its multilingual understanding capabilities. Additionally, the vision encoder is an EfficientNet-Bl (Tan & Le, 2019) pre-trained on ImageNet (Deng et al. 2009). We did not use any data augmentation during training. The detailed training recipe is presented in Table
4.2 DATASETS AND EVALUATION METRICS
Datasets. We evaluate our model on various benchmarks to demonstrate its effectiveness. For ISLR, we adopt WLASL (Li et al. 2020a) and MSASL (Joze & Koller, 2019) datasets for evaluation. For CSLR, we utilize CSL Daily (Zhou et al. 2021). SLT task conducted on the CSL-Daily, How2Sign (Duarte et al. 2021). and OpenASL (Shi et al. 2022) datasets.
Evaluation metrics. Following previous works (Hu al., 2021a Zhou et al. 2024). we report per-instance (P-I) and per-class (P-C) Top-1 accuracy, as well as word error rate ( (WER), to eval- uate ISLR and CSLR, respectively. For SLT, we adopt BLEU (Papineni et al., 2002) from the SacreBLEU (Post) (2188) library and ROUGE-L (Lin) 2004) as evaluation metrics. For English SLT
datasets, we also report BLEURT (Sellam et al., 2020) scores using the BLEURT-20 checkpoint, as it has been shown to correlate strongly with human judgments.
4.3 COMPARISON WITH STATE-OF-THE-AN METHODS
To validate the effectiveness of our framework, we conduct series of experiments across diverse range of SLU tasks. To provide additional references for future research, we present both the perfor- mance of the RGB-pose setting and the pose-only setting, where the posc-only experiments bypass the training in Stage 2 entirely. Due to page limitations, the qualitative visualization is presented in Appendix A.4
Results on ISLR and CSLR. We compare the results of Uni-Sign with previous studies on ISLR benchmarks in Table 3 Our model surpasses previous SOTA these benchmarks without any task-specific designs. Compared to the previous ofate-of-the-an methods (MSLU and NLA-SLR), approach achieves improvements of 4.09% and 2.47% in per-instance Top-1 accuracy the
8

---

Published as a conference paper at ICLR 2025
MSASL.100 (ISLR)
CSt-Dally (CSLR)
CSL-Daily (CSLR)
CSL-Dally (SLT)
Setting
P-1
P.C
Dey WERI
Test WER1
Setting
Text WERL
Dev ROUGEE
Test
Top-11
Top-Ir
WFRJ
BLFUUIT
BI.EU41
ROUGE1
Task-specific fine waning paradigm
0%
74.7
73.6
3.75
20.46
3.51
20.56
$6.92
37.4
36.4
25%
31.5
31.0
20.68
49.68
21.13
49.9
Preuee
70.97
39.2
38.3
50%
31.5
30.1
21.85
50.98
22.58
51.62
Unitted line toring parddigm
75%
28.8
28.5
24.74
54.28
24.95
54.87
Ours
77.88
76.55
28.2
27.4
100%
28.2
27.4
25.27
54.34
25.61
54.92
Table 7: paradigm in pose-only setting.
Impact of fine-tuning
Table 8: Impact of pre-training data scale in pose-only set-
ting.
CSL-Daily (CSRR)
CSL CSI-baily (SLT)
Prans
Time
Dev WURI
Test WCRL
BLCU4T
ROUGET
DLDU4T
Test
ROUGET
Setting
CSI.Dbaily (CSLR)
Der WER1
Tal
CSL-Duil
0%
27.4
25.27
54.32
25.61
54.92
WERI
BLEU4
ROUGE1
BLEU4
ROUGE1
36.0
26.25
26.36
56.51
27.3
55.63
26.11
56.22
26.2
26.37
26.7
56.03
26.36
56.51
26.4
26.30
57.43
Table 10: Impact of fusion module in RGB- pose setting.
Table 9: Impact of score-aware sampling strategy in RGB-pose setting.
challenging MSASL1000 and WLASL2000 datasets, respectively. Moreover, we evaluate the per- formance on CSLR, as shown in Table 4 Despite not employing CTC loss to impose temporal constraints sign language, our model still shows competitive performance, with only 1.3% and 0.7% performance drop compared to TS-SLR. We argue that the performance gap between Uni-Sign and TS-SLR may be attributed the more complex model architecture and the dense intermediate state constraints incorporated in TS-SLR. The experiments demonstrate that our approach learns ro- bust SLU capabilities via pre-training and successfully transfers the knowledge through generative fine-tuning to downstream tasks.
Results on SLT. Table 5 and present comparisons of SLT performance between our method and prior approaches on the CSL-Daily, How2Sign, and OpenASL datasets. We observe that Uni-Sign beats previous gloss-free SOTA on CSL-Daily and OpenASL, with a substantial performance in- crease in BLEU4. On the CSL-Daily dataset, under the same gloss-free paradigm, Uni-Sign sur- passes previous SOTA, achieving improvements of 14.02 and 4.75 in the BLEU4 scores on the dev and test sets, respectively. Moreover, surprisingly found that Uni-Sign also outperforms certain gloss-based SLT models, such as SLTUNET and TS-SLT, which integrated gloss information into their frameworks through CSLR training. The series of results above emphasizes the importance of large-scale generative pre-training, which endows the model with robust SLU capabilities. On the OpenASL dataset, our method outperforms RL by 9.93 in BLEU4 and achieves a remark- able BLEURT score (60.40 vs. 36.35). Meanwhile, our performance on How2Sign is also notable, achieving comparable results with RGB-based models (SignMusketteers, SSVP-SLT) under the same pre-training dataset. Although SSVP-SLT employs a larger-scale vision encoder, a more complex pre-training strategy, and a longer pre-training duration, Uni-Sign demonstrates only a slight perfor- mance difference in terms of BLEU4 (14.9 vs. 15.5), highlighting its competitive potential.
4.4 ABLATION STUDY
We conduct various ablation studies to investigate the contribution of each key component in Uni-Sign. Specifically, the MSASLI000 dataset is selected for ISLR, while the CSL-Daily dataset is used for the other tasks.
Impact of fine-tuning paradigms. We separately utilize features from the temporal encoders and the language model encoder as the representations of sign language, denoted as Fsign and Fine respectively. These features are then performed aask-specific fine-tuning settings investigate the impact of different fine-tuning paradigms. For ISLR, the selected features undergo mean pool- ing followed by a classification head, supervised by cross-entropy loss. For CSLR, the features are passed through LSTM layer and optimized by CTC loss. As depicted in Table 7 our pro- posed fine-tuning paradigm achieves the best performance, demonstrating a notable margin over spsc-ipe fine-tuning paradigms. We also observe that the features Fim enc yield better results in ISLR than Fsign, while performing worse in CSLR. This suggests that while Fim captures high-level semantics of sign language, it compromises short-term temporal understanding. Despite

---

Published as a conference paper at ICLR 2025
the rich semantic information encoded in Fim encs improper fine-tuning method resulted in performance drop of 6.91% for P-I and 8.01% % for P-C in ISLR. Furthermore, the unified fine-tuning paradigm leverages its robust SLU understanding and linguistic restructuring capabilities, signifi- cantly outperforming the mainstream CSLR fine-tuning paradigm that uses CTC loss for temporal constraints (28.2 37.4 and 27.4 vs. 36.4). The above results highlight that our proposed unified fine-tuning paradigm can effectively transfer the SLU capabilities within the pre-trained model.
Impact of pre-training data scale. We randomly sample portion of data from the CSL-News dataset for pre-training to explore the impact of pre-training data scale on model performance. In Table we observe that as the quantity of pre-training data increases, the performance of various tasks progressively improves, indicating that our model can benefit from larger datasets and high- lighting the critical role of large-scale pre-training.
Impact of score-aware sampling strategy. To evaluate the effectiveness of the score-aware sam- pling strategy. we perform hyperparameter selection on the sampling probability Psamp- As illus- trated in Table 9 increasing Psamp results in gradual improvement in SLT, achieving maximum gain of 1.25 B BLEU4 on the CSL-Daily test set when Psamp reaches 50%. However, time consump- tion also increases significantly. To balance performance and time consumption, we select 10% as the default value, which still yields promising results.
Impact of fusion module. To evaluate the impact of the fusion module, replace the deformable attention (DA) in the PGF module with tross-attention (CA). The results presented in Table I0 demonstrate that DA outperforms CA in the CSLR, and SLT tasks, highlighting the effectiveness of using keypoint coordinates as priors.
CONCLUSION AND FUTURE WORK
In this paper, we introduce Uni-Sign, a unified pre-training framework that leverages a large-scale generative pre-training strategy and novel fine-tuning paradigm, narrowing the gap between pre- training and downstream SLU tasks. Specifically, we first propose CSL-News, a large-scale CSL translation dataset containing 1,985 hours of video-text pairs, which enables effective large-scale pre-training. Next, we unify the tiniitunng paradigm by treating downstream SLU tasks as a single SLT task, which significantly narrows the gap between pre- pre-training and fine-tuning while facilitating the transfer of SLU capabilities to these tasks. Moreover, we introduce the PGF module and a score-aware sampling strategy efficiently capture visual cues from both RGB and pose modalities while achieving a trade-off between performance and speed. Despite the simplicity of Uni-Sign, we achieve remarkable results across multiple SLU tasks, demonstrating notable improvement over previous state-of-the-ar methods.
In the future, we are interested in exploring large-scale pre-trained multilingual SLU models and SLU tasks in complex scenarios (such as complex backgrounds, multi-signer situations, long- duration sign language understanding). We are also keen to investigate sign language production, research field as important as SI.U, to ensure that the Deaf/Hard of Hearing communities can equally benefit from technological advancements.
REPRODUCIBILITY STATEMENT
To facilitate reproducibility, we have provided details of the training settings in Section 4.1 with further details of our framework to be presented in Appendix[A.. The CLS-News dataset and the code of Uni-Sign have been open-sourced and made available, aiming to promote further research on SLU.
ACKNOWLEDGMENTS
This work was supported by National Natural Science Foundation of China under Contract U20A20183 & 62021001, and the Youth Innovation Promotion Association CAS. It was also sup- ported by the GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC, and the Supercomputing Center of the USTC.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Samuel Albanie, Gül Varol, Liliane Momeni, Hannah Bull, Triantafyllos Afouras, Himel Chowd- hury, Neil Fox, Bencie Woll, Rob Cooper, Andrew McParland, et al. Bbc-oxford british sign language dataset. arxiv, 2021.
Boyes Braem and RL Sutton-Spence. The Hands Are The Head of The Mouth. The Mouth as Arficulator in Sign Languages. Hamburg: Signum Press, 2001.
Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. Neural sign language translation. In CVPR, pp. 7784-7793, 2018.
Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. Sign language trans- formers: Joint end-to-end sign language recognition and translation. In CVPR, pp. 10023-10033, 2020.
Necati Cihan Camgöz, Ben Saunders, Guillaume Rochette, Marco Giovanelli, Giacomo Inches, Robin Nachtrab-Ribbacd and Richard Bowden. Content4all open research sign language trans- lation datasets. In FG, pp. 1-5, 2021.
Ting Chen, Saurabh Saxena, Lala Li, Tsung- Yi Lin, David Fleet, and Geoffrey E Hinton. unified sequence interface for vision tasks. In NeurIPS, 2022a.
Yutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, and Stephen Lin. A simple multi-modalih transfer learning baseline for sign language translation. In CVPR, pp. 5120--5-51 5120-5130, 2022b.
Yutong Chen, Ronglai Zuo, Fangyun Wei, Yu Wu, Shujic Liu, and Brian Mak. Two-stream network for sign language recognition and translation. In NeurIPS, 2022c.
Zhigang Chen, Benjia Zhou, Yiqing Huang, Jun Wan, Yibo Hu, Hailin Shi, Yanyan Liang, Zhen Lei, and Du Zhang. rl Content and context representation learning for gloss-free sign language translation and retrieval. arxiv, 2024a.
Zhigang Chen, Benjia Zhou, Jun Li, Jun Wan, Zhen Lei, Ning Jiang, Quan Lu, and Guoqing Zhao. Factorized learning assisted with large language model for gloss-free sign language translation. In I.RECCOLING, pp. 7071-7081, 2024b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-254- 2009.
Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giro-i Nieto. How2sign: a large-scale multimodal dataset for continuous american sign language. In CVPR, pp. 2735-2744, 2021.
Biao Fu, Peigen Ye, Liang Zhang, Pei Yu, Cong Hu, Xiaodong Shi, and Yidong Chen. A token-level contrastive framework for sign language translation. In ICASSP, pp. 1-5, 2023.
Zhifu Gao, Zerui Li, Jiaming Wang, Haoneng Luo, Xian Shi, Mengzhe Chen, Yabin Li, Lingyun Zuo, Zhihao Du, Zhangyu Xiao, and Shiliang Zhang. Funasr: A fundamental end-to-end speech recognition toolkit. In INTERSPEECH, 2023.
Jia Gong, Lin Geng Foo, Yixuan He, Hossein Rahmani, and Jun Liu. Lims are good sign language translators. In CVPR, pp. 18362-18372, 2024.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tempo- ral classification: labelling unsegmented sequence data with recurrent neural networks. In JCML, pp. 369-3766 2006.
Shester Gueuwou, Sophie Siake, Colin Leong, and Mathias Müller. JWSign: A highly multilingual corpus of Bible translations for more diversity in sign language processing. EMNLP, pp. 9907- 9927, 2023a.
Shester Gueuwou, Kate Takyi, Mathias Müller, Marco Stanley Nyarko, Richard Adade, and Rose- Mary Owusuaa Mensah Gyening. Afrisign: Machine translation for african sign languages. In AfricaNI.PW, 2023b.
11

---

Published as a conference paper at ICLR 2025
Shester Gueuwou, Xiaodan Du, Greg Shakhnarovich, and Karen Livescu. Signmusketeers: An efficient multi-stream approach for sign language translation at scale. arxiv, 2024.
Thomas Hanke, Mare Schulder, Reiner Konrad, and Elena Jahn. Extending the Public DGS Corpus in size and depth. In LRECW, pp. 75 82, 2020.
Hezhen Hu, Weichao Zhao, Wengang Zhou, Yuechen Wang, and Houqiang Li. SignBERT: pre- training of hand-model-awwand representation for sign language recognition. In ICCV, pp. 11087- 11096, 2021a.
Hezhen Hu, Wengang Zhou, and Houqiang Li. Hand-model-aware sign language recognition. In AAAI, pp. 1558-1566, 2021b.
Hezhen Hu, Wengang Zhou, Junfu Pu, and Houqiang Li. Global-local enhancement network for nmf-aware sign language recognition. ACM TOMM, 17(3):1-19, 2021c
Hezhen Hu, Weichao Zhao, Wengang Zhou, and Houqiang Li. SignBERT+: Hand-model-aware self-supervised pre-training for sign language understanding. IEEE TPAMI, 45(0):11221-11239, 2023a.
Hezhen Hu, Junfu Pu, Wengang Zhou, Hang Fang, and Houqiang Li. Prior-aware cross modality augmentation learning for continuous sign language recognition. IEEE TMM, pp. 593-606, 2024.
Lianyu Hu, Liqing Gao, Zekang Liu, and Wei Feng. Continuous sign language recognition with correlation network. In CVPR, pp. 2529-2539 2023b.
Lianyu Hu, Liqing Gao, Zekang Liu, and Wei Feng. Self-emphasizing network for continuous sign language recognition. In AAAI, pp. 854-862, 2023c.
Lianyu Hu, Liqing Gao, Zekang Liu, Chi-Man Pun, and Wei Feng. Adabrowse: Adaptive video browser for efficient continuous sign language recognition. In ACM MM, pp. 709-718, 2023d.
Longtao Jiang, Min Wang, Zecheng Li, Yao Fang, Wengang Zhou, and Houqiang Li. SEDS: Se- mantically enhanced dual-stream encoder for sign language retrieval. In ACM MM, 2024.
Songyao Jiang, Bin Sun, Lichen Wang, Yue Bai, Kunpeng Li, and Yun Fu. Skeleton aware multi- modal sign language recognition. In CVPRW, pp. 5408-3418, 2021.
Tao Jiang, Peng Lu, Li Zhang, Ning Ma, Rui Han, Chengqi Lyu, Yining Li, and Kai Chen. RTM- Pose: Real-time multi- multi-person pose estimation based on mmpose. arxiv, 2023.
Peiqi Jiao, Yuecong Min, Yanan Li, Xiaotao Wang. Lei Lei, and Xilin Chen. CoSign: Exploring co- occurrence signals in skeleton-based continuous sign language recognition. In ICCV, pp. 20676- 20686, 2023.
Sheng Jin, Lumin Xu, Jin Xu, Can Wang, Wentao Liu, Chen Qian, Wanli Ouyang, and Ping Luo. Whole-body human pose estimation the wild. In ECCV, 2020.
Hamid Reza Vaezi Joze and Oscar Koller. MS-ASL: A large-scale data set and benchmark for understanding american sign language. BMVC, pp. 1 1-16, 2019.
Sang-Ki Ki Ko, Chang Jo Kim, Hyedong Jung, and Choongsang Cho. Neural sign language translation based on human keypoint estimation. Applied Sciences, 9(33), 2019.
Dongxu Li, Cristian Rodriguez, Xin Yu, and Hongdong Li. Word-level deep sign language recogni- tion from video: A new large-scale dataset and methods comparison. In WACV, pp. 1459-1469, 2020a.
Dongxu Li, Xin Yu, Chenchen Xu, Lars Petersson, and Hongdong Li. Transferring cross-domain knowledge for video sign language recognition. In CVPR, pp. 6205-62144 2020b.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping Language-imag pre-training with frozen image encoders and large language models. ICML, pp. 19742, 2023.
12

---

Published as a conference paper at ICLR 2025
Chin-Yes Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, 2004.
Kezhou Lin, Xiaohan Wang, Linchao Zhu, Ke Sun, Bang Zhang, and Yi Yang. GLoFE: gloss-free end-to-end sign language translation. In ACL, pp. 12904-12916, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023.
Yuecong Min, Aiming Hao, Xiujuan Chai, and Xilin Chen. Visual alignment constraint for contin- uous sign language recognition. In ICCV, pp. -1551, 2021.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu, Bleu: a method for automatic evaluation of machine translation. In ACL, pp. 311-318, 2002.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In NeurIPS, 2019.
Matt Post. A call for clarity in reporting BLEU scores. In WMT,, pp. 186 191, 2018.
Junfu Pu, Wengang Zhou, Hezhen Hu, and Houqiang Li. Boosting continuous sign language recog- nition via cross modality augmentation. In ACM MM, pp. 1497-1505, 2020.
Phillip Rust, Bowen Shi, Skyler Wang, Necati Cihan Camgoz, and Jean Maillard. Towards privacy- aware sign language translation at scale. In ACL. pp. 8624-8641. 2024.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. In ACL, pp. 7881-7892, 2020.
Bowen Shi, Diane Brentari, Gregory Shakhnarovich, and Karen Livescu. Open-domain sign lan- guage translation learned from online video. In EMNLP, pp. 6365-6379, 2022.
Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural net- works. In ICMI., pp. 6105-6114, 2019.
Garrett Tanzer and Biao Zhang. Youtube-sl-25: A large-scale, open-domain multilingual sign lan- guage parallel corpus. arxiv, 2024.
Laia Tarrés, Gerard I. Gállego, Amanda Duarte, Jordi Torres, and Xavier Giró Nieto. Sign language translation from instructional videos. In CVPRW, pp. 5625-5635, 2023.
Dave Uthus. Garrett Tanzer, and Manfred Georg. Youtube-asl: A large-scale, open-domain american sign language-english parallel corpus. In NeurlPS, 2024.
Peng Wang. An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through simple sequence-to- serenceet--sequence learning framework. In ICML, 2022.
Ryan Wong, Necati Cihan Camgoz, and Richard Bowden. Sign2GPT: Leveraging large language models for gloss-free free sign language translation. In ICLR, 2024.
Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Zhe Chen, Wenhai Wang. Xizhou Zhu, Lewei Lu, Tong Lu, Ping Luo, Yu Qiao, and Jifeng Dai. VisionLLM v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks. In NeurlPS, 2024.
Zhuofan Xia, Xuran Pan, Shiji Song. Li Erran Li, and Gao Huang. Vision transformer with de- formable attention. In CVPR, pp. 4793-4803, 2022.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In NAACI, pp. 483-498. 2021.
13

---

Published as a conference paper at ICLR 2025
Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-bai based action recognition. In AAAI, pp. 1-10, 2018.
Aoxiong Yin, Zhou Zhao, Weike Jin, Meng Zhang, Xingshan Zeng, and Xiaofei He. MLSLT:
Towards multilingual sign language translation. In CVPR, pp. 5099-5109, 2022.
Aoxiong Yin, Tianyun Zhong, Li Tang, Weike Jin, Tao Jin, and Zhou Zhao. Gloss attention for gloss-free sign language translation. In ICCV, pp. 2551-2562, 2023.
Kayo Yin, Amit Moryossef, Julie Hochgesang, Yoav Goldberg, and Malihe Alikhani. Including signed languages in natural language processing. In IJCNPP, pp. 7347-7360, 2021.
Biao Zhang, Mathias Müller, and Rico Sennrich. SLTUNET: A simple unified model for sign language translation. In ICLR, 2023a.
Huaiwen Zhang, Zihang Guo, Yang Yang, Xin Liu, and De Hu. C2st: Cross-modal contextualized sequence transduction for continuous sign language recognition. In ICCV, pp. 21053-21066 2023b.
Rui Zhao, Liang Zhang, Biao Fu, Cong Hu, Jinsong Su, and Yidong Chen. Conditional variational autoencoder for sign language translation with cross-modal alignment. In AAAI, pp. 19643- 19651, 2024a.
Weichao Zhao, Hezhen Hu, Wengang Zhou, Jiaxin Shi, and Houqiang Li. BEST: Bert pre-training for sign language recognition with coupling tokenization. In AAAI, pp. 3597-3605, 2023.
Weichao Zhao, Hezhen Hu, Wengang Zhou, Yunyao Mao, Min Wang, and Houqiang Li. Masa: Motion-aware masked autoencoder with semantic alignment for sign language recognition. IEEE TCSVT, 2024b.
Benjia Zhou, Zhigang Chen, Albert Clapés, Jun Wan, Yanyan Liang, Sergio Escalera, Zhen Lei, and Du Zhang. Gloss-free sign language translation: Improving from visual-language pretraining. In ICCV, pp. 20871-20888 2023.
Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li. Improving sign language translation with monolingual data by sign back-translation. In CVPR, pp. 1316-1325, 2021.
Hao Zhou, Wengang Zhou, Yun Zhou, and Houqiang Spatial-temporal multi-cue network for sign language recognition and translation. IEEE 7MM, 24:768-779, 2022.
Wengang Zhou, Weichao Zhao, Hezhen Hu, Zecheng Li, and Houqiang Li. Scaling up multimodal pre-training for sign language understanding. arxiv, 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In ICLR, 2024.
Ronglai Zuo, Fangyun Wei, and Brian Mak. Natural language-assisted sign language recognition. In CVPR, pp. 14890-14999 14900, 2023.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 FRAMEWORK IMPLEMENTATION
Keypoints extraction. We employ the RTMPose-x (Jiang et al. 2023) from MMPose extract whole-body keypoints. The visualization of whole-body keypoints are shown in Figure 6 As men- tioned in Section 3.2 we divide the pose into sub-pose (left hand, right hand, face, and body). We select the indices for the left hand (192-1 (22)1 right hand [113-133); body ((1, 4-11)), and face ((24, 26, 28, 30, 32, 34, 36, 38, 40, 54, 84-91)) }) to represent each group. Additionally, we select 92, 113, and 54 as the root indices for the hands and face to normalize the keypoints, while the body is
not normalized using root coordinates.
Figure 6: The visualization of the whole-body 133 keypoints, derived from (Jin et al. 2020
Feature extraction. We detail the output dimensions of feature extraction in Table |1 It is im- portant to note that the weights are not shared among each group. Hence, we create three separate linear layers, pose encoders, and temporal encoders to capture the representation of each group individually.
Layer
Dimensions
Temporal kernel
Linear
64
None
Pose encoder
[64. 128, 256]
None
Temporal encoder
[256, 256, 256]
Table 11:
Output dimension of each layer in feature extraction.
Pre-trained large language model. We leverage the HuggingFace library for the pre-trained large language model from httpp:://hugiinggtttt/e
co/ http:/ogggl/mbaaaaaaeeeeeeee
Parameters Uni-Sign. The parameters of Uni -ni-Siin shown in Table 12 while the parameters for the compared methods are estimated based on their original papers. Compared to previous methods, Uni-Sign demonstrates significant advantages in both parameter efficiency and performance.
Method
Visual encoder
Language model
Auxiliary text encoder
FLa-LLLM
ResNet18 (11 (177M)
MBart-large-cc25 (610.8M)
Sign2GPT
DinoV2 (VIT-S/14) (22.0M)
XGLM-1.7B (1732.9M)
SigoLLM
ResNet18 (11.7M)
LLaMA-7B 7B (6738.4M)
C²RL
ResNet18 (11.7M)
MBart-large-ec23 (610.8M)
MBart Encoder (408.2M)
Uni-Sign
EfficientNet-BO GCN (5.2M 4.5M)
miS--Base (582.4M)
Table 12: Comparison of parameters across different methods.
15

---

Published as a conference paper at ICLR 2025
A.2 ADDITIONAL ABLATION STUDIES
Impact of different sub-pose. To conduct experiments to explore the impact of different sub-pose, directly the training from scratch pose-only settings reduce the time consumption. As presented in Table T3] each sub-pose indispensable, prompting us to incorporate all sub-poses
into our model.
MSASL1000 (ISLR)
CSL-Daily (CSLR)
hands
body
face
P-I
P-C
Dev WERI
Test
Top-1t
Top-11
WERL
36.07
34.20
54.3
53.9
41.56
38.28
53.0
52.9
43.45
41.10
51.2
50.7
Table 13: Impact of sub-pose in pose-only setting.
A.3 PSEUDOCODE OF SCORE-AWARE SAMPLING STRATEGY
In order to provide a more detailed explanation of this strategy, we present the pseudocode score- aware sampling strategy here.
Algorithm Pseudocode of the score-aware sampling strategy in PyTorch like style.
feat_h: pose features of hand, shape [T, 21, C]. 2 score_h: keypoints confidence of hand, shape [T, 21]. coor_h: coordinates hand, shape LT, 21, 2].
1
4
P_samp: sampling probability.
5
Step 1: Pre-define total duration of the sign language sequence 7 feat_h. feat_h.shape[8)
8
Step Calculate reliability scores (rs)
keypoint confidence
10
rs = Iconfidence.meatt for confidence in score_h]
Step 3: Calculate sampling scores as - sanpling_seores [1 score for store in rs]
13
Step 4: Perform randon sampling sampled_indies indices randon.choices( (range()),, weightsssamllngg.sores keint(T P_samp))
16
Step 5: Extract RGB franes, pose features and coordinates RCB_franes [read_hand_ hand_inage(( for i sampled implled_idiics]]
19
pose_features [feat_h[i]
sampled indices]]
21
pose_coordinatee [coor_h[i] for in sampled_indicees]
22
the pose modality RGB_ features vision_encader (RddddrRRR__Fameess
Interact the modality
RGB
cross_modins featurr PGF (RGB_features, features, pose__oordinates)
27 Step 7: Fuse cross modality features to pose features Reat_h[aapled_indiee indicess]__i cross_nodalittyfesttrss
A.4 QUALITATIVE EXAMPLES
Visualization ISLR and CSLR. Figure presents representative examples from the ISLR task, showcasing the capability of Uni-Sign to effectively address ISLR challenges. Table 14 presents the CSLR results the CSL-Daily dataset. Uni-Sign demonstrates powerful SLU capabilities by achieving notable performance on the CSLR task, emphasizing large-scale generative pretraining as a promising direction for scaling up CSLR models. However, failure cases reveal challenges in distinguishing semantically similar words (e.g., 4 f7" (you) - 4 f'1" (you). Il* (receive) - d2 x (receive), % t (excited) & * (happy)), underscoring the importance of fine-grained control over output targets, which could further enhance model performance and reliability.
16

---

Published as a conference paper at ICLR 2025
Reference: nice
Uni-Sign: nice
Reference: enjoy
Uni-Sign: enjoy
Reference: big
Uni-Sign: big
Reference: hope
Uni-Sign: hope
Figure 7: Visualization examples derived from the WLASL and MSASL datasets.
Reference:
2
AG AU
Uni-Sign:
A2
Reference:
#6
#
i2
4 #
3 t tM
Uni-Sign:
A11
#0
#
Bl
Reference:
x
4
11
41
Uni-Sign:
-
Reference:
Uni-Sign:
12
Table 14: Visualization examples derived from the CSL-Daily dataset.
17

---

Published as a conference paper at ICLR 2025
SLT examples. In Table 15
16 and 17 present several SLT results across different datasets.
We found that our model successfully captures semantic information in sign language, generating sentences that are close in meaning to the references, despite occasional differences in sentence structure. However, we also observed that the model sometimes fails to translate complex sentence structures, as demonstrated in the last example of Table T6]and 17
Reference:
FBT $
(It's snowing and it's really cold today.) F 18
Uni-Sign:
(It's snowing and it's very cold today.)
Reference:
$ il $ f
$ BAggAAK A &
(My grandfather knew sign language
had many deaf friends.)
Uni-Sign:
& $ $ t f ie & $ A
(My grandfather can sign language and he has many deaf friends.)
Reference:
#1 11 it A k
(I forgol to bring my mobile phone when I went out loday, which is really unlucky.)
Uni-Sign:
$ & di [] & - T.
#
(When I went out today, forgot my cell phone. What bad luck.)
Reference:
$ #1 T i f A 66 z zu to $ *
(My brother received the admission notice from Tsinghua University and was
very happy.)
Uni-Sign:
- 1 te 31 $6 * a 34 to Dt, io & X R/ (It not difficult to imagine how excited my brother was when he received the admission notice from Tsinghua Universily.)
a tt 18
Table 15: Translation examples on the CSL-Daily dataset.
Relerence:
Alright.
Uni-Sign:
Okay.
Reference:
A little speed.
Uni-Sign:
Just little bit faster.
Reference:
A really basic
ropes that are kind medium length.
Uni-Sign:
The basic coil we're going to do for ropes is some kind of medium length.
Reference:
After you dealt the three cards, you would look down your cards and you would decide if you wanted continue play.
Uni-Sign:
I'm going cut three of these out and I'm going decide if I want con- tinue.
Table 16: Translation examples on the How2Sign dataset.
Reference:
An official letter, right. So it's letter.
Uni-Sign:
Reference:
Before the meeting, you should share with your captioner what topics will be discussed, so the captioner will be better prepared for your meeting. Before the meeting, you should discuss with your captioner what topics should be discussed, so that the meeting is smooth.
Uni-Sign:
Reference:
After graduated, went to Gallaudet University and double majored in Biol- ogy and Chemistry.
Uni-Sign:
Eventually, graduated and went to Gallaudet University for my Bachelor Biology.
Reference:
Besides dog. Lieutenant Dan, there is mini horse, pig. Ilama, hamster. duck and two cats.
Uni-Sign:
addition to the dogs, the dogs and other volunteers include miniature horses, elephants, llamas and hamstrings.
Table 17: Translation examples on the OpenASL dataset.
18

---

Published as a conference paper at ICLR 2025
A.5 MORE DISCUSSION ABOUT CSL-NEWS DATASET
To facilitate more detailed comparison between CSL-News and existing datasets, we provide fur- ther analysis of the CSL-News dataset. The vocabulary distribution of CSL-News is presented in Figure 8 In addition to the advantage of a longer duration, we further emphasize several other advantages of CSL-News over existing datasets, outlined below:
High diversity of content. The CSL-News dataset is sourced from news content, encompassing diverse topies such as culture, economy, sports, science and daily life. Compared small scale datasets (Camgoz et al.] 20188) [Zhou et al. 2021). it exhibits more diverse data distribution that is restricted specific domains.
High quality and standardization. Unlike datasets scraped from YouTube (Li et al. 2020a Shi al. 2022 Uthus et al., 2024), CSL-News dataset is derived from news segments featuring sign language experts, ensuring more standardized signing and thereby enhancing the reliability and overall quality of the CSL-News dataset.
Benefiting from the comprehensive CSL knowledge in the CSL-News dataset, models pre-trained on can acquire robust sign language understanding and generalization capabilities.
2500
Number Words
1500
500
1000 1250
1500
1750 >2000
Word Frequency
Figure Vocabulary distribution of CSI--News dataset.
B ETHICS STATEMENT
In this paper, Uni-Sign uses keypoints and cropped hand video clips input. This ensures that our method not only achieves impressive performance, also protects the privacy the Deaf/Hard of Hearing communities.
C LIMITATIONS
Although Uni-Sign has achieved impressive performance across multiple benchmarks, we still lack finely annotated, open-domain, large-scale benchmark to further investigate its capabilities and lim- itations. Furthermore, fine-tune all parameters of Uni-Sign in all training stages, which presents urgent challenges to computational resources.
D COMPLETE RESULTS OF EXPERIMENTS
Due to page limitations, some experimental results have been omitted from the main paper. The complete experimental results are provided here facilitate future research in referencing the rele- vant results.
19

---

Published as a conference paper at ICLR 2025
Method
Modality
MISASLI00
MISASL200
MISASL1000
WLASLIBU
WLASL301
WLASS2200
Fose RGB
F-1
P-C
P-1
P-C
P-1
P-C
P-
P-C
P-1
P-C
P-]
P.C
ST-GCN
2015
45.29
90.78
51.62
34.40
32.53
SignBERT
76.29
70.92
49.54
76.56
77.68
39.40
36.74
BEST
76.50
76.75
77.83
46.25
Signll
$4.94
78.51
79.25
79.84
50.72
48.85
46.37
MSLU
Z2ou aa
91.54
91.78
$7.99
$8.58
74.07
71.81
88,
89,
82,04
82.71
56.29
51, 29
HMA
73:45
74.59
06.30
67.47
49. 19.16
46.27
37.91
83.04
50.31
$1:14
77.53
77.
NLA-SER Zanrel 2023
90.49
91.04
59.74
89.23
72.56
60.86
91.47
92.17
86.23
86.07
61.05
50.15
93.26
93.16
90.95
91.38
92.24
92.75
63.13
Uni Sign (Durs)
93.79
94.02
91.82
91.56
78.16
76.97
92.25
92.67
88.47
88.92
63.53
61.32
Table 18: ISLR results on various benchmarks. f denotes methods reproduced by (Hu et al. Blue and Green denote the best results of previous methods and ours, respectively.
2021a
Method
Modality
Dev
Test
Pose RGB
BLBUT
BLEU2
BLEUS
BLBL4
ROUGE
BLBUT
BLEU2
BLEUS
BLBU1
ROUGE
Gloss-used
SLKT"
Campor et
2020
37.47
24.67
16.36
11.38
37.96
37.38
24.36
16.55
11.79
36.74
ConSLr
Tore 2023
14.50
41.46
14.53
40.98
Sign8T
2hnu 272
51.46
37.23
27.51
20.80
49.49
51.42
37.26
27.76
21.34
49.31
SLILNET
20250
53.58
-4.998
41.44
25.01
MMTIR
23.02
CV-SLT
56.36
45.15
28.04
$7.06
TS-SLT
55.21
12.31
32.71
25.76
55.10
42.59
32.87
25.79
55.72
Gloss- free
MSLU
Zayi al. 2024
33.28
21.31
10:27
33.13
33.97
22.20
11.42
33.80
smppy Mi00
21:03
9.97
30.51
4.93
3.03
19.67
GASI.
Tm Diza
19.90
9:94
5.98
4.07
20.35
34:23
7.56
25.02
36.70
FLa
Chen dt di
37.13
25.12
18.38
14.20
37.25
Sign2GP1
20124
41.75
28.75
20.50
13.40
42.36
SignLLM
Gong et
2024
20.07
15.75
C'KL
Chen et 20124a
49.33
36.28
27.54
21.61
48.21
Uni-Sign (Durs)
53.24
40.54
31,05
54.34
40.96
32,022
25.61
Uni (Ours)
55.30
42.21
32.94
26.25
56.03
55.08
42.14
32.98
26.36
56.51
Table 19; SLT results on CSL-Daily dataset. f and denote methods reproduced by (Zhou et al.
202T and (Zhou et al. 2023). respectively. Underline indicates the best gloss-based SLT result.
Method
Medalitt
Dey
Tist
Pise RCB
BLEUT
BLEUU
BLEUS
BLEU4
ROUGE
BLEURT
BLEUI
BLBU2
BLEUS
BLEU
ROUGE
BLEURT
Closs free
GiFFE VN Lin al 2025
21.06
12.35
8.68
6.68
21.37
36,75
21.55
12.74
9.15
7.56
21.75
36.35
Cow-GRR:
16.25
15.36
6.15
25.65
13D innn
10.25
7,17
18.85
29.17
7.19
18.54
29.92
OpenASI
Nh da 20224
20.10
11.11
3.49
5.57
70
31.23
30.92
17.06
8.90
6.32
21.00
31,09
RL. Chia al M034
16.58
31.36
(Cus))
25.12
42.77
Uni-Sign (Ous)
50.84
37.82
29.53
24.16
44.58
61.28
49.35
36.32
28.55
23.14
43.22
60.40
Table 20: SLT results OpenASL dataset. denotes methods reproduced by (Shi et al. 2022
Modality
Test
RGB
BLEU2
BLEU!
BLEL4
ROUGE
GRFEEVV
Lin st
YouTube-AA Crine
A074
37.8
24.1
16.9
12.4
46.6
17.2
34.0
12.2
8.0
C²RL
Chea al (4L
29.1
18.6
12.9
9.4
27.0
FLa-LLST [LAe
A3425
29.8
19.0
13.3
9.7
27.8
SignMucieegir Greetoo
2024
41.5
27.2
19.3
14.3
(024
Uni Sign (Ouns)
40.4
26.8
19.3
14.5
34.3
45.6
(Ours)
Table 21: SLT results on How2Sign dataset.
20


Reviews:
[{"review_id": "NcV7NLGo2j", "reviewer": "Reviewer_w7tC", "summary": "This paper presents a new pre-training framework that bridges the gap between pre-training and downstream sign language understanding tasks through a large-scale generative pre-training strategy and a novel fine-tuning paradigm that achieves impressive performance in multiple benchmark tests.", "questions": "1. Although there is a difference for traditional sign language recognition methods employing such means as MLP and CTC loss, the authors propose for different tasks still use different supervision, for example, words, glosses, and sentences, and why it is still referred to as a unified paradigm.\n\n2. In Fig. 5, why the feature information of the face has to be forwarded to the left Pose Encoder after it has been encoded by the Pose Encoder is not mentioned in the paper..\n\n3. In line 479 of the paper, the authors show a boost of 1.36 on BLEU-4, but the corresponding value is not found in Table 9.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The Uni-Sign framework proposed by the authors utilizes a large-scale generative pre-training strategy and a novel fine-tuning paradigm to bridge the gap between pre-training and downstream sign language understanding tasks in traditional approaches.\n\n2. The Uni-Sign framework achieves significant performance gains on both sign language recognition and translation tasks, and experiments are conducted on multiple datasets.\n\n3. The related work of paper is adequate, investigating research on sign language tasks including pre-training strategies, dataset development, and so on, from a variety of perspectives.", "weaknesses": "1. The paper is not clear and detailed enough to explain the score-aware sampling strategy, and does not give a detailed analysis of the process or a corresponding explanation in Figure 5, which could lead to potential misunderstandings or errors.\n\n2. The author omitted experimental results on several widely used datasets, such as Phoenix14, Phoenix14T, USTC-SLR 500, USTC-CSL100, etc.\n\n3. As shown in Tables 4 and 6, the proposed Uni-Sign method does not achieve the best performance on multiple datasets of continuous sign language recognition and sign language translation. It even performs worse when more modalities are introduced, which makes me worried about the performance of this work.\n\n4. The number of parameters of the model is not mentioned in the paper. This feedback highlights the importance of including these key performance metrics, as they are critical for evaluating the practicality of the model.\n\n5. It is recommended that the authors make font color changes for the tables throughout the article, due to the large amount of experimental data, while bolding may mislead the reader, especially for Tables 3 through 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TBfyUIVOCG", "reviewer": "Reviewer_y6Ev", "summary": "This paper proposes Uni-Sign, a unified pre-training framework for sign language understanding (SLU) tasks, addressing the challenges in existing methods that struggle with transferring knowledge across different tasks. The framework uses a new large-scale dataset, CSL-News, which contains 1,985 hours of Chinese Sign Language (CSL) videos paired with textual annotations. Extensive experiments demonstrate that Uni-Sign achieves state-of-the-art performance across multiple SLU benchmarks.", "questions": "Please refer to the Weakness section above. If the authors can address these concerns, I would consider raising the rating.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Uni-Sign effectively unifies multiple SLU tasks, such as isolated sign language recognition (ISLR), continuous sign language recognition (CSLR), and sign language translation (SLT), under a single framework. \n2. The introduction of CSL-News, a substantial CSL dataset, provides a significant resource for the SLU field and addresses the limitations of prior smaller datasets.", "weaknesses": "1. Compared to other datasets, what unique advantages or characteristics does the proposed CSL-News dataset offer besides its longer duration? Additionally, why is the vocabulary size relatively limited, and could the restricted language variety impact pre-training effectiveness?\n2. In the comparisons of downstream tasks in Section 4.3, did other methods also use the CSL-News dataset for pre-training? If not, does this raise any concerns about fairness in the comparisons?\n3. In the comparative experiments, while high-performing results are analyzed, the reasons behind lower performance should also be provided, such as in Tables 4 and 6.\n4. In Tables 3 to 6, what would the results of Uni-Sign be if it used only RGB video?\n5. How do the computational costs, inference time, and memory usage of the proposed model compare to other methods? Does Uni-Sign maintain a competitive advantage in these aspects?\n6. The manuscript includes numerous comparative results, but it lacks visualizations to intuitively demonstrate the model’s effectiveness. More visual presentations for each downstream task are recommended.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sz5lY2TjzG", "reviewer": "Reviewer_CGke", "summary": "This paper presents Uni-Sign, a novel framework for Sign Language Understanding (SLU) that leverages large-scale generative pre-training and a unified fine-tuning paradigm. The paper presents a well-motivated and well-executed approach to SLU. The introduction of the CSL-News dataset and the innovative Uni-Sign framework are significant contributions to the field, demonstrating state-of-the-art performance across various SLU tasks. The paper is well-written and clearly explains the proposed methodology and experimental results.  The authors make several notable contributions:\n•Introduction of CSL-News: The authors introduce CSL-News, a large-scale Chinese Sign Language (CSL) dataset comprising 1,985 hours of video-text pairs. This dataset significantly surpasses existing CSL datasets in size and diversity\n•Unified Pre-Training and Fine-Tuning:  During fine-tuning, it treats downstream SLU tasks, such as isolated sign language recognition (ISLR), continuous sign language recognition (CSLR), and sign language translation (SLT), as a single SLT task. This unified approach facilitates seamless knowledge transfer and eliminates the need for task-specific fine-tuning methods.\n•Prior-Guided Fusion (PGF) Module: To address the limitations of inaccurate keypoints, the authors propose a PGF module that fuses pose and RGB information using keypoint coordinates as priors. \n•Score-Aware Sampling Strategy: The authors introduce a score-aware sampling strategy to improve computational efficiency. \n•Comprehensive Evaluation: The paper includes a comprehensive evaluation of Uni-Sign across various SLU benchmarks, demonstrating its superior performance in ISLR, CSLR, and SLT tasks.", "questions": "The paper in general addressed the ideas and motivations it introduces. The following question will help add more comprehensive understanding. \nGeneralization and Applicability\n1. Multilingual Evaluation: The sources primarily focus on CSL and ASL. Could the authors comment on the applicability of Uni-Sign to other sign languages? How might the model's architecture and pre-training strategies need to be adapted for multilingual SLU? This is important to assess the generalizability of Uni-Sign and its potential impact on a broader range of sign language communities\n2. Multi-signer Scenarios: How well does Uni-Sign perform in situations involving multiple signers? What challenges might arise in such scenarios, and how could the model be modified to handle them effectively? Addressing this question would provide a more realistic assessment of Uni-Sign's capabilities in real-world applications where multiple signers may be present\n\nComparison and Analysis\n1. Comparison with LLM-based SLT Methods: Recent studies like Sign2GPT and Sign-LLM have explored the use of LLMs for gloss-free SLT. Could the authors provide a comparative analysis of Uni-Sign against these LLM-based approaches? This would help clarify Uni-Sign's contributions and position it within the broader landscape of SLT research\n2. In-depth Analysis of the Unified Fine-tuning Paradigm: How does the shared objective function influence the performance of individual tasks like ISLR and CSLR? Are there any potential task-specific adaptations that could be incorporated within the unified framework to further optimize performance? This analysis would provide a more nuanced understanding of the paradigm's strengths and weaknesses", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: \n1. The paper presents Uni-Sign, a novel unified pre-training framework for Sign Language Understanding (SLU) that bridges the gap between pre-training and downstream tasks by treating them as a single Sign Language Translation (SLT) task during fine-tuning. This approach deviates from previous methods that relied on indirect pretext tasks or were limited by data scale and transfer capability\n2. The authors introduce CSL-News, a large-scale Chinese Sign Language (CSL) dataset containing 1,985 hours of video with text annotations, considerably larger than existing CSL datasets. his dataset enables effective large-scale pre-training, addressing a gap in CSL resources compared to American Sign Language (ASL) and British Sign Language (BSL)\n3. The paper proposes a Prior-Guided Fusion (PGF) module that utilizes keypoint coordinates as priors to model fine-grained spatial consistency between pose and RGB modalities, going beyond simple spatial-temporal fusion techniques. This approach addresses the representational gap between modalities and leverages keypoints to enhance accuracy. \n4. A score-aware sampling strategy is introduced to address the computational challenges of RGB-pose fusion by selectively choosing RGB frames corresponding to low-confidence keypoints, balancing performance with speed\n\nQuality:\n1. The paper is well-written and presents a clear and comprehensive methodology. The authors provide detailed descriptions of their approach, including data curation, pre-training and fine-tuning strategies, and multi-modal fusion techniques\n2. The ablation studies thoroughly investigate the contribution of each key component, offering insights into the model's performance and the impact of design choices\n3. Quantitative results show that Uni-Sign surpasses previous state-of-the-art methods on multiple benchmarks, including significant improvements in BLEU4 scores for SLT tasks\n\nClarity:\n1. The paper is well-organized and easy to follow.\n2. Figures and tables effectively illustrate the framework, data distribution, and experimental results\n3. Mathematical notations and equations are clearly defined and explained\n4. Qualitative translation examples provide further insights into the model's capabilities\n\nSignificance:\n1. The introduction of the CSL-News dataset addresses a significant need for large-scale CSL resources, potentially fostering advancements in CSL research\n2. The unified pre-training and fine-tuning framework with a generative approach demonstrates a promising direction for improving SLU performance, particularly for SLT tasks\n3. The proposed PGF module and score-aware sampling strategy offer effective solutions for multi-modal fusion and computational efficiency, potentially benefiting future SLU research\n4. The paper's findings have implications for advancing sign language technologies, promoting accessibility and communication for the Deaf/Hard of Hearing community\n5. The authors' commitment to open-sourcing the code and dataset further contributes to the significance of the work, facilitating reproducibility and future research in SLU", "weaknesses": "1. Discussion on Computational Complexity: While the authors introduce a score-aware sampling strategy to improve efficiency, a more in-depth discussion on the computational complexity of Uni-Sign would be beneficial. This could include analyzing the trade-offs between accuracy and computational cost for different sampling probabilities and exploring potential optimizations.\n2. Further Analysis of CSL-News: While the paper describes the creation of CSL-News, further analysis of the dataset's characteristics, such as vocabulary distribution and linguistic complexity, would be valuable. This would provide a more comprehensive understanding of the dataset's potential and limitations.\n3. Cross-Dataset Generalization: Evaluating Uni-Sign's performance on unseen sign language datasets would demonstrate its generalization capabilities. This could involve fine-tuning the pre-trained model on a different CSL dataset or even a dataset from another sign language, like American Sign Language (ASL). Successful cross-dataset generalization would highlight the robustness of the learned representations and the effectiveness of the unified approach.\n4. Analysis of Error Patterns: A qualitative analysis of the translation errors made by Uni-Sign would provide valuable insights into its limitations and potential areas for improvement. This could involve categorizing errors based on linguistic features, such as sentence complexity, sign ambiguity, or finger-spelling. Identifying common error patterns could guide future research directions.\n5. Exploration of Multi-Signer Scenarios: The authors mention their interest in exploring SLU tasks in complex scenarios, such as multi-signer situations. Including preliminary experiments or discussions on adapting Uni-Sign to handle such scenarios would further enhance the paper's impact and contribution to the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D7Wcy9lMqF", "reviewer": "Reviewer_3Yy5", "summary": "This paper propose Uni-Sign, a unified pre-training framework that eliminates the gap between pre-training and downstream SLU tasks through a large-scale generative pre-training strategy and a novel fine-tuning paradigm. It also introduce CSL-News, a large-scale Chinese Sign Language (CSL) dataset containing1,985 hours of videos paired with textual annotations.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 4, "strengths": "Pros:\n1. This work proposes a unified framework to conduct pretraining and finetuning, which demostrates novelty.\n2. This work shows promising performance across a wide range of benchmarks.\n3. The paper is easy to understand.", "weaknesses": "Questions and cons:\n1. During the data curation process, the authors use a ASR toolkit (FunASR) to convert the speech into texts as labels. There exist some problems. First, as the speech signal has time delay with the sign language expressed by the signer, how to assure that the temporally cropped clips are exactly aligned with the transcribed texts? Second, the authors have stated the averaged length of words are 40 and the averaged length of clips are 9.5s. It's very hard to express 40 words within 9.5s for a signer. Thus, it's most probably that the signer has neglected some meanings in the sentence, and only expressed a part of mearnings in the signs. In this condition, the signs are probably not aligned with the transcribed texts. Third, i observed that in the paper, the authors don't organize a double-check process for the cropped videos from the TV shows to check the alignment between texts and clips, the correctness of transcribed texts,  the correctness of transcribed signs and other things. Thus, how to assure the compleness and correctness of the curated datasets?\n2. During the experiments for CSLR, PHOEXNI14 and PHOENIX14-T are also broadly used datasets. Why not report the results on these datasets? It's due to the language gap between pretraining data and downstream data? How about the performance on these two datasets?\n3. In table 3 and table 5, some other numbers are bolded except the results reported by the proposed method. The authors may clarify on this or use another way to emphsize the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qvbLHY9J6y", "reviewer": "Reviewer_pQAF", "summary": "This paper has two main contributions. 1) A Uni-Sign method for tackling the three sign language understanding tasks in a unified manner. The model first pre-trains on a large sign language dataset via language modeling, then is fine-tuned on each of the individual tasks separately. 2) A CSL-News dataset, which is a large-scale Chinese Sign Language dataset. Some other minor architectural designs are also proposed. Overall, the proposed method performs quite well across the three sign language understanding tasks, and particularly performs well in Sign Language Translation.", "questions": "In Table 6, the performance of the proposed method is somewhat lower than the existing baseline SSVP-SLT. Although it is not a very big issue to me, but I would like to know more about it. Why is this the only (rather large) SLT dataset where the proposed method achieves sub-optimal results?\n\nThe ablation results shown in Table 7 are rather strange as compared to Tables 8-10, because the settings are different. Table 7 runs experiments on ISLR and CSLR, Tables 8-10 run experiments on CSL-Daily for CSLR and SLT. Why are these different? Moreover, Table 7 and 8 are run in the pose-only setting while Tables 9 and 10 are in the RGB-Pose setting, why should this be the case?\n Furthermore, some of the more important experiments (Table 7 and 8 in my opinion) should be evaluated on all three different sign language understanding tasks.\n\n\nWhat is the impact of the pre-training? This crucial aspect as not been evaluated properly. For instance, what if the model is trained only using the fine-tuning stage (Stage 3), but for a longer time (i.e., matching the overall training time of the pre-train then fine-tune approach)? How does this affect the performance? This is important as it shows us the benefits of pre-training. Although some results have been provided in table 7, the results and implications are not clear to me. Furthermore, the task-specific training settings and details have not been mentioned.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Developing a unified approach to handle the various sign language understanding tasks is meaningful. In some sense, the work extends some recent LLM-based sign language understanding works by including the aspect of unifying across the sign language understanding tasks. \n\nThe authors introduce a new large-scale sign language dataset for Chinese Sign Language. This dataset could be quite useful for further progress in the field.\n\n\nThe experiment results are quite impressive, especially on the gloss-free SLT task. In my opinion, gloss-free SLT is the setting that is the closest to real applications, so this is quite good.", "weaknesses": "The proposed method is not very novel. The proposed pre-training approach is to train the model in a language modelling manner, while also using visual features from the sign videos. Then, for the fine-tuning, the language modelling loss is again used for the various tasks. There are some minor contributions, such as a prior-guided fusion module and a score-aware sampling strategy, but these do not seem quite so substantial.\n\nI think that in the related works discussion, there should be a part discussing some other works in other fields employing language modelling (or sequence modeling) for tackling various tasks in a unified manner. For instance, this has been done for image-based tasks, and may have also been done for pose-based tasks. This will give the reader a better understanding of the developments of the “unifying via language modeling” paradigm.\n\n\nMore specific concerns and questions are in the “Questions” section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)