MixSignGraph: A Sign Sequence is Worth Mixed Graphs of Nodes
Authors: Shiwei Gan
Yafeng Yin
Zhiwei Jiang
Lei Xie
Sanglu Lu
Hongkai Wen
Venue: NeurIPS 2025 poster ()
Keywords: Sign Language Recognition and Translation
Graph convolutional network
Primary area: applications

Abstract: Recent advances in sign language research have benefited from CNN-based backbones, which are primarily transferred from traditional computer vision tasks (\eg object detection, image recognition). However, these CNN-based backbones usually excel at extracting features like contours and texture, but may struggle with capturing sign-related features. 
To capture such sign-related features, SignGraph model extracts the cross-region sign features by building the Local Sign Graph (LSG) module and the Temporal Sign Graph (TSG) module. However, we emphasize that although capturing cross-region dependencies can improve sign language performance, it may degrade the representation quality of local regions.
To mitigate this, we introduce MixSignGraph, which represents sign sequences as a group of mixed graphs for feature extraction. 
Specifically, besides the LSG module and TSG module that model the intra-frame and inter-frame cross-regions features, we design a simple yet effective Hierarchical Sign Graph (HSG) module, which enhances local region representations following the extraction of cross-region features, by aggregating the same-region features from different-granularity feature maps of a frame, \ie to boost discriminative local features.
In addition, to further improve the performance of gloss-free sign language task, we propose a simple yet counter-intuitive Text-based CTC Pre-training (TCTC) method, which generates pseudo gloss labels from text sequences for model pre-training. Extensive experiments conducted on the current five sign language datasets demonstrate that MixSignGraph surpasses the most current models on multiple sign language tasks across several datasets, without relying on any additional cues. Code and models are available at: \href{https://github.com/gswycf/SignLanguage}{\textcolor{blue}{https://github.com/gswycf/SignLanguage}}.

MixSignGraph: A Sign Sequence is Worth Mixed Graphs of Nodes
Shiwei Gan
Yafeng Yin Zhiwei Jiang' Lei Xiet Sanglu Lu Hongkai Wen
State Key Laboratory for Novel Software Technology. Nanjing University, China Department of Computer Science, The University of Warwick, UK {aw, yafeng jzw, lxie, wwwxxxiniiiieeeeeee sangluldnnjuuuu..uu edu. hongkai wwnwwawwick.aa
Abstract
Recent advances in sign language research have benefited from CNN-based back- bones, which are primarily transferred from traditional computer vision tasks (e.g., object detection, image recognition). However, these CNN-based backbones usu- ally excel at extracting features like contours and texture, but may struggle with capturing sign-related features. To capture such sign-related features, SignGraph model extracts the cross-region sign features by building the Local Sign Graph (LSG) module and the Temporal Sign Graph (TSG) module. However, we empha- size that although capturing cross-region dependencies can improve sign language performance, it may degrade the representation quality of local regions. To mitigate this, we introduce MixSignGraph, which represents sign sequences as a group mixed graphs for feature extraction. Specifically, besides the LSG module and TSG module that model the intra-frame and inter-frame cross-regions features, we design simple yet effective Hierarchical Sign Graph (HSG) module, which enhances local region representations following the extraction of ressoon fea- tures, by aggregating the same-region features from different-grase -raaulaattt feature maps of a frame, i.e., to boost discriminative local features. addition, to further improve the performance of gloss-free sign language task, propose simple yet counter-intuitio Text-hased CTC Pre-training (TCTC) method, which gener- ates pseudo gloss labels from text sequences for model pre-training. Extensive experiments conducted on the current five sign language datasets demonstrate that MixSignGraph surpasses the most current models on multiple sign language tasks across several datasets, without relying on any additional cues. Code and models are available http::/ggihbb.comm/sw//////in
Introduction
Advancements in computer vision (CV) and natural language processing (NLP) technologies have significantly facilitated the development of sign language (SL) research, including Sign Language Recognition (SLR) and Sign Language Translation (SLT). The SLR task encompasses Isolated SLR (ISLR) ① and Continuous SLR (CSLR) 3 4), aiming to recognize isolated or continuous signs corresponding glosses gloss sequences. While the SLT task focuses translating continuous signs into spoken language (5). the current SL framework, the standard approach first uses 2D/3D CNN-based backbones to extract visual features (6) 7 (8. then employs temporal modules (5 9 10 to capture dynamic changes in sign frames, and finally adopts a Connectionist Temporal Classification (CTC) decoder to obtain the gloss sequence
translation model to generate the spoken sentence.
Unlike other CV tasks where contour and texture representations are crucial, SL tasks need to focus on both manual and non-manual features, particularly the collaboration of these cues in different
Yafeng Yin the corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

while the traditional CNNs may fail to capture the collaboration of signs across different regions, thereby limiting their ability to extract effective sign features. To enhance the CNN-based backbone's is ability to extract sign-related features, existing work has focused on designing various CNN-based backbones by incorporating domain knowledge, such as skeleton information ()) depth images (I3), or local areas (10). Additionally, some studies have adopted different training strategies e.g., back translation (14), knowledge distillation (15) and contrastive learning (6). Besides, in SLT tasks, many current SOTA models also rely gloss annotations to pre-train their backbones with CTC loss, aiming to improve the SLT performance. When gloss annotations are missing, i.e., in Gloss-Free SLT (GFSLT) tasks, how to improve the SLT performance is still challenging task.
regions (TI [12,
To address the limitation of CNN- backbones, SignGraph (4) transforms SL frames into graph nodes and proposes the Local Sign Graph (LSG) and Temporal Sign Graph (TSG) modules to model spatial and temporal cross-region dependencies. However, such strategies may inadvertently degrade fine-grained local features due to the emphasis on cross-region aggregation. Regarding improving the gloss-free SLT performance, the existing models have attempted introduce attention mechanisms (16), contrastive language-image pre-training (I7) or vector quantization (18).
Building upon prior work, we propose simple yet effective sign graph neural net- work, MixSignGraph, for SL tasks (CSLR and SLT). In addition to extracting cross- region features, MixSignGraph introduces Hierarchical Sign Graph (HSG) module to enhance local representations from fea- ture maps of different granularities, mitigat- ing the degradation of local features that may occur during cross-region modeling in SignGraph. The main intuitions are: Cross- region feature extraction may lead to the local representation of the current node be- ing diluted by features from other regions, resulting in the degradation of local informa- tion. Moreover, the content within single region can be represented at different levels of granularity through hierarchical feature maps generated by down-sampling, There- fore, we propose Hierarchical Sign Graph (HSG) module, which connects correspond- ing regions across multi granuulrrtty feature maps. This hierarchical design allows the model reinforce local representations by aggregating complementary information from different granularities, thereby mitigating the degra- dation caused by cross-region feature fusion. Besides, further improve the performance of sign language tasks without gloss annotations, we propose simple yet counter-intuitive Text- based CTC Pre-training (TCTC) method, which generates pseudo gloss labels from text labels for model pre-training. We make the following contributions:
Traditional CNN: in spatial, temporal dimension
Traditional downsampling: from high to low resolution
LGG/TSG across regions in one adjacent frames
Our HSG: the same regions across feature maps
Figure I: (Left) Comparison between CNNs and LSG/TSG in SignGraph. (Right) Message passing of the same regions among different-resslltion feature maps between downsampling in CNNs and our HSG.
To address the limitations of SignGraph (4,, we propose a simple yet effective sign graph neural network, called MixSignGraph. In addition to extracting cross-region features, MixSignGraph introduces a Hierarchical Sign Graph (HSG) module, which enables bidirec- tional exchanges of feature maps with different granularities, to enhance local representations from hierarchical feature maps. The proposed HSG can mitigate the degradation of local features that may occur during cross-region modeling.
A simple, effective, yet counter-intuitive training method named Text-based CTC Pre- training (TCTC) is proposed for the GFSLT task, which generates pseudo gloss annotations based on text labels. The proposed TCTC effectively improves the performance of GFSLT and narrows the gap with gloss- based SLT models.
The extensive experiments on common SI. tasks (including CSLR and SLT) over five public datasets demonstrate the superiority of the proposed MixSignGraph, which shows promising results and does not any extra cues.

---

Nx
LSG
Donen
Projectoo Layer
Proietiion Laysz
Kerelingg
Global Fenure Medale
Gloons (CSLR)
Coop
Plas
Local Graph
Building
Teaporal Griph Bellding
Hicrarchical Graph Huildiing
Treedation Model
Poojatton Layer
Projectonn
Precetioz Layer
Mie1
Wies plaz) Tayu (ostpu) high sag
operation
Figure 2: The proposed MixSignGraph architecture.
2
Preliminaries
For SL sequence f=[fi)= with 9 frames, the target of CSLR task to get a recognized gloss sequence g {9. {91) with & glosses, while the target of SLT task is to generate spoken language sentence f = {+ with ş words based on input f. SignGraph (4) introduces the following modules model the SL sequence
Patchify Stem. To apply graph convolutional networks (GCNs) directly to SL frames instead of skeleton data, patchify stem is employed to divide each RGB frame with height H, width W into N = HW/P* P2 patches (or nodes) U; = {ei 1/- with the corresponding feature embedding Mig E DR Here, fij represents the j-th patch in the i-th frame ft, D denotes the feature dimension, and P the patch size.
Local Sign Graph Learning, To capture correlations between different spatial regions, such as the face and hands, SignGraph 4 introduces Local Sign Graph (LSG) module to extract intra-frame cross-region features. Specifically, it computes the feature distances between patches within each frame using distance function DIS (9). Based on these distances, K-nearest neighbor (KNN)- based graph construction strategy is used to obtain local sign graph, and graph convolution is applied to integrate features across nodes, enabling effective modeling of intra-frame spatial dependencies.
Temporal Sign Graph Learning. To capture the movements of the body, hands, and face, Sign- Graph (4) introduces Temporal Sign Graph (TSG) module that dynamically establishes connections between regions across consecutive frames, enabling the learning of inter-frame cross-region features. Similarly, calculates feature distances between patches in adjacent frames and adopts KNN-based graph construction strategy to effectively model cross-region features between frames.
Patch Merging. Considering that patches with fixed window may effectively capture sign features (e.g., hand regions may be split by different patches), SignGraph (4) adopts the patch merging module to downsample feature maps by factor of 2, to generate larger-size patches.
3 Method
Overall Framework. As shown in Figure when given the video frames, patchify stem is adopted convert each frame into set of patches (i.e., nodes) v; {wij } Then, the three key modules are utilized to construct graphs and capture sign-related features as follows. First, by adopting the LSG and TSG modules, MixSignGraph can learn both the correlation of cross-region features within one frame and the interaction of cross-region features among adjacent frames. Second, by aggregating the same-region features from different feature maps of frame, build a hierarchical sign graph G to bidirectionally exchange features and mine different-grann granularity one-region features. After that, we adopt a global feature module (2) ( to learn global changes of whole frames. To tackle CSLR tasks, classifier and a widely-used CTC loss (20) are adopted to predict the probability p(gl.) of the target gloss sequence. SLT tasks, we adopt a translation model to convert feature sequences or gloss sequences into text sequences. To further improve GFSLT performances, we propose the Text-based CTC Pre-training (TCTC) mechanism, which generates pseudo gloss labels from text sequences for model pre-training.
3

---

Hierarchical Sign Graph Learning. Despite the benefits of capturing cross-region features for im- proving SI. performance, it may degrade the repre- sentation quality of local regions. Besides, traditional CNNs can only downsample one region from coarse to fine representations, lacking the ability to facili- tate interaction between different-grane -rgarulatga feature maps. To address this issue, we introduce the HSG module, which enhances local information from fea- ture maps of different granularities after cross-region feature extraction. In this way, we can combine hi- erarchical features of the same region and make the model focus both global shapes and detailed local features of region the time. As shown Figure given two feature maps fu R xxc""" in high resolution and μ qi xw xc"] in low resolution for frame ft, we have N Hh nodes D for μ and N H'W' nodes of for u4. Here, 11"h Wh, Cth H1,, Wi and C¹ are the height, width and the number of channels of y and n² respectively, and 1/"h H = W = > 0, where 8 is the downsampling factor. As shown in Figure and to construct the hierarchical sign graph, we first use a projection layer with weights O to map u pld" xW"x""| to n م 0f, ensuring the same dimensionality as u4, where " E x xC'), Then, we add an undirected edge e(ufy, " if the node oi in p and the node u/x μ correspond to the one region.
One frame
Hicrarchical graph of the same regions across feature mups
Figure Graph construction in hierarchical sign graph module.
W
W + j% 1)
e (e(r)) vire 1 [0, )), k
1
(I)
In this way, we can get the edge set et between μ and uu for the frame fi, and obtain the hierarchical sign graph G = {(u w e. After that, we apply graph convolutional layer GCN' и to enable bidirectional exchanges of the same regions across different-granullrity feature maps perform feature fusion.
/ 1 = GCN (1 ef). mf = + e
Unlike LSG and TSG, which dynamically build edges based on the KNN algorithm to capture cross-region features, the HSG focuses on fusing features belonging to the same region at different granularities, aiming to enhance local features of one region.
Mixed Sign Graph Learning. Based on the previous work SignGraph, insert the proposed HSG module after the LSG and TSG modules to enhance local information from feature maps of different granularities. For convenience, we represent the ith LSG, TSG and HSG module LSG,, TSG; and H SG, respectively.
3.1 Text-based CTC Pre-training
Gloss-based Tasks. For CSLR tasks, CTC loss with gloss annotations has become the de-facto loss function in current CSLR models. While in gloss-based SLT tasks, the current SOTA SLT models (22) 7 rely pre-training, in which CTC loss with gloss sequences used optimize the backbone to extract effective video-level features, as described in Equation 3 After the pre-training, these SLT models further introduce pre-trained translation model (e.g., mBART (23)) and fine-tund the translation model for SLT, as defined in Equation 4 Here, LOTC denotes CTC loss function, LcE denotes cross-entropy loss function, I and g respectively denote recognized gloss sequence and labeled gloss sequence, f and t respectively denote translated sentence and labeled sentence, Og and O denote the parameters of recognition and translation models.
min Lcrc((..))
(3)
8x
min (Corc(g.g) + (ce(f,t)) 8x.87
(4)
Gloss-free Tasks. The above pre-training process enables the backbone to learn segmentation and semantic information from gloss annotations, which are crucial for improving the SLT performance.

---

Laput
Imput Fecnure
140
Z8
Gloss Embnodkingg
Text Embodding
13D Featare
Text Emberiing
(a) Visualization of feature distribution of one test sample in PHOENIXI4T dataset.
(b) Visualization of feature distribution of one test sample in the How2Sign datasct.
Figure 4: Visualization of feature distribution, which is displayed via t-SNE (21). Blue, green, and pink points represent the input features of the translation model by NOT using TCTC, using TCTC, using CTC pre-training with gloss annotations.
However, in the other SLT tasks, i.e., gloss-free SLT tasks, there gloss annotations, since annotating glosses is a labor-intensive task.
Text-based CTC Pre-training (TCTC):
To improve the performance of these gloss-free SLT tasks, we propose a simple, effective, yet counter-intuitive training method called Text-based CTC Pre- training (TCTC), which generates pseudo gloss sequences from labeled sentences for pre-training. Specifically, in TCTC, we obtain the pseudo gloss sequence t' by simply pre-processing labeled sentence f, including removing punctuations, lemmatization and word-level tokenization. Then, directly use the CTC loss with pseudo gloss sequence to optimize the backbone during initial pre-training, and then fine-tune the entire model for translation, as depicted in Equation 5
min Lere(f, OR
(5)
min (Lorelf, tt) Lce(f,t) 2g.er
(6)
The principle of TCTC: Typically, CTC loss requires that the source sequence and the target sequence aligned in the same order. Thus, CTC loss is often used in the CSLR task, where the sign video and gloss sequence have the same order. However, SLT tasks, the alignment between the SL video and the text sequence is usually non-monn non-monotonic, due to the difference in grammatical rules between SL and spoken language. To tackle the challenge of GFSLT task, TCTC generates pseudo gloss sequences from labeled sentences and then adopts CTC loss for model pre-training. Although the generated pscudo gloss sequences from text/sentences may be different from the ground-tron gloss sequences, our TCTC is still crucial for improving SLT performance. The two possible reasons can be concluded as follows. First, given the discrete nature of languages, the inputs and outputs of translation models (e.g., mBART used in SLT) are in a categorical, discrete- discrete-valued space (i.e., typically as token embeddings (24)). Second, without CTC pre-training, the features of adjacent frames/clips (extracted by the backbone) tend to remain pre-training with gloss sequences guides the model to segment video features into discrete units. Therefore, by using CTC with pseudo gloss sequence to pre-train train the backbone, TCTC provides a weak tokenization function, even if it may result in inaccurate segmentation.
continuous-valus space, while CTC
To verify the effectiveness of TCTC, model, i.e., the features output from the CSLR model, while using or NOT using TCTC. As shown in the bottom left part of Figure 4 the embedding distributions of gloss sequence (i.e., red points) and words in spoken sentence (i.e. black points) are normally in a discrete-valluee space, as described in first reason the previous paragraph. When moving to the top left part of Figure 4 the features extracted by NOT using TCTC are located at close positions (e.g., blue points on continuous curve), while the features extracted by using TCTC scatter at different positions (i.e., green points discrete- vacuue space), as described in the second reason of the previous paragraph. It demonstrates
visualize the feature distributions input the translation
5

---

that introducing CTC loss with pseudo gloss annotations can guide the model to extract discrete and tokenized video features. From the perspective of feature distribution space, the pattern of feature distribution by using TCTC is similar to that of word embedding in spoken sentences, í.e., in discrete-valuee space. It is worth mentioning that the TCTC mechanism not only makes sense for our proposed MixSignGraph, but also works for other backhones. To verify this phenomenon, we also visualize the feature distribution of the other pre-trained backbone I3D in right part of Figure which adopted for How2Sign and OpenASL datasets (see Section I for details). The above phenomenon proves that TCTC can provide a weak tokenization function, even TCTC may bring inaccurate segmentation.
Experiments
Datasets. Our experiment evaluation is conducted on five publicly available SL datasets, including PHOENIX14 (25), PHOENIX14T CSL-Daiy (14), How2sign (26) and OpenASL (27). For
more detailed descriptions of these datasets, please refer to Appendix A.2
Architecture Setting. Our architecture is im- plemented using PyTorch 1.11. The setup in- cludes the following components: (1) Patchify stem and Patch merging: We use patchify stem and patch merging module in SignGraph to ob- tain patch (node) embeddings. (2) Distance func- tion: the baseline setting, we measure the dis- tance between two nodes using the Euclidean distance. (3) Global feature module: This mod- ule comprises two 1D convolution blocks, 2- layer BiLSTM with a hidden size of 1024 for global feature modeling, and fully connected layer for the final prediction. (4) Translation network: For fair comparison, following the current SOTA SLT model (7), we adopt the pre-
Table I: Effects of sign graph modules.
ESCG TSG, HSG, 189ss TSG, HSG.
Dev
Test
DelAns WER Dellis
X
22.3 8 22.2 8.1/2.7
19.2 5.6/2.2
21.0 4.8/2.3
19.6 5.6/2.1 21.5 5.1/2.5
19.9 6.7/2.1 21.6 6.2/3.1
19.3 5.3/2.1 20.8 5.8/2.0
19.5 6.6/1.7 21.2 5.4/2.4
19.4 6.2/2.9
21,5 5.9/3.6
18.7 5.1/2.3
20.6 5,2/1.7
18.6 4.3/18
20.2 5.5/1.7
17.8 6.1/233
19.6 5,3/29
13.1 5.3/2.0
19.7 5.03.0
17.4 5,7/211
19.6 5,1/2.9
16.7 4,9/2.1
19.0 4,9/2.9
trained mBART model provided by Hugging- face as our translation network.
Training Setting. For fair comparisons, adopt the same data preprocessing and training settings as in previous work (2) 44. In regard to How2Sign and OpenASI. datasets, we adopt the visual features generated by a pre-trained I3D model (8) In addition, to ensure the model can be trained end-to-end on three 3090 GPUs with 24GB memory, the entire model is trained in half-precision.
Evaluation. We evaluate the performances of MixSignGraph on the following tasks: CSLR and SLT (including Sign2Gloss2Text, Sign2Text, Gloss-free Sign2Text). Please refer to Appendix A.. for detailed descriptions of these tasks. To evaluate model. adopt the Word Error Rate (WER) metric for the CSLR task, and ROUGE-L F1 Score (28) and BLEU-1-4 (29) for SLT tasks.
4.1 Ablation Study
Following the previous work (5). we perform ablation studies on PHOENIX14T dataset to verify the effectiveness of the proposed MixSignGraph model.
Effects of Sign Graph Modules. As shown in Table I we conduct an ablation study to evaluate the individual and combined contributions of the LSG, TSG, and HSG modules. The baseline model without any graph module yields a WER of 22.3% 22.2% on the Dev/Test set. Furthermore, when HSG is combined with LSG and TSG, observe consistent performance gains, which demonstrates that the proposed HSG module effectively mitigates the degradation of local features caused by cross-region feature extraction, and helps enhance the model's overall representation capability and CSLR performance.
Most importantly, the best performance is achieved when the HSG modules is inserted after LSG and TSG modules, resulting the lowest WER of 16.7% (Dev) and 19.0% (Test), which demonstrates the importance of hierarchical local feature enhancement via HSG in complementing cross-region

---

Table 2: Effect of proposed TCTC for SLT. Besides, we also show the CSLR performance based on the pseudo gloss sequence obtained by text labels in the right part.
Test
Dey Dellins
Test DeVillns
Doinet
Model
ROUGE
BLEUT
BLEU2
BLEU3
BLED4
ROUGE
BLEUI
BLBU2
BLEUS
BLEU4
WER
WER
PHOENIXICT
w/a TCTC
32.63
33.57
18.71
13:47
9.20
34,56
35.31
21.05
16:15
9.40
-1-
---
TCTC
91.71
51.07
37.97
20.98
24.87
51.14
50.01
38.04
29.95
24.02
59.97
368/2.5
59.55
35.9/2.7
w/
55.77
55.01
42,64
34.94
29.00
53.84
54.90
42.53
34.50
28,97
16.72
4.9/2.1
19.01
4.9/2.9
CSI-Dally
w/o TCTC
34.11
33.78
20.13
12.61
8.01
32.21
32.86
14.34
10.67
6.78
w/ WTTCCC
49.16
49.98
36,42
26.89
20.45
49.93
50.24
36.91
27.54
21:01
66.77
39.
11/4.9
666.05
18.48449
w. gloss
54.54
55.87
42.45
32.75
25.77
54.67
55,41
42.43
32.84
25.87
25.13
6.42.11
25.01
7.0/1.6
How2Sign
TCTC
18.35
23,88
12,92
7.89
5.05
18,06
23.41
12.55
7.61
w/ TCTC
29.24
34.82
22.47
15.61
11:28
28.01
32.74
20.83
14.41
10.41
72.38
46.35/2.69
74.88
50.27/2.07
OpenASI.
w/o TCTC
12.86
11.95
4.89
2.78
1.88
12.56
11.29
4.64
2.77
1.95
--
w/CCCC
25.41
16.70
11.48
8.36
26.65
81.53
59.88/1.08
61.33
60.39/1.29
Table 3: CSLR qualitative results on PHOENIX14, PHOENIX14T and CSI.-Daily. We use different colors represent substitutions, deletions, and insertions, respectively.
example(a)
PHOENIXI4 14 dataset
Gavendrruth
MORGEN DASSELAR BE SCHAUER REGION SONST VIEI SONNE REGION THE WHHEN STARK
MultiSiguCraph
MORGEN DASSELBE SCHAUER REGION SONST VIEL SONNE REGION TEILWEISE WEHEN SCHWACH
MisSignGraph
MORGEN DASSELBE SCHAUER REGION SONST VIEL SONNE REGION TEILWEISE WEHEN STARK
example(h)
PHOENIXI4T dutaset
Groundruth
DARUNTER NEBEL LANG KO-KOMEED DANEBEN SONNE BERU OBEN DANN DURCHGEREND SONNE
MultiSinnCruph
ON DARUNTER NEBEL LANG IN KOMMEND DANEDEN SONNE BERG OBEN DANK DURCHGEHEND SONNE
MisSignGraph
DARUNTER
IN KOMMEND DANEBEN SONNE BERG OBEN DURCHGEHEND SONNE
csampleb)
CSL-Daaly dataset
Groundrruth
MultiSiguKrrphh
t 2
MixSigoGrraph
t 01
Table 4: SLT qualitative results on PHOENIX147 14T, CSL-Daily, How2Sign and OpenASL.
example(a)
PHOENIXI4T dataset
Groundtruth
und zum wochenende wird dann sogar wieder ein bisschen kälter
w/o TCTC
und zum wochenende teil recht freundliche hochdruckwetter beglcitet uns und zum wochenende wird es dann sogar ein bisschen kälter und zum wochenende wird dann sogur wieder bisschen käller
w/ TCTC
Gloss-bused
example(h)
CSI daily dataset
Groundtruth
fF
44
37
30
+
4
Nf
SE
HI
w/o TCTC
$
H
fl
2
37
2
Fu 4
w/ TCTC
13
4 X
OF KF
ML
Gloss-bused
ff
5
X0
f
ft
4
example(c)
How2sign dataset
Groundruth
You can take this forward and back, you can take it a circle, you can take in a lot different directions.
TCTC
Take a step forward, back; you can do circular mothion that can go from on direction to the other.
w/ TCTC
You can take this forward and back. you take it circles, you can take it in lot of different ways.
example(d)
OpenASL dataset
Groundtruth
There are results pending for 20 other tests
TCTC
There are now fires reports other people injured
w/ TCTC
There are wuiting for the results of the 20 other tests
spatial and temporal modeling. It confirms that HSG plays a critical role in reinforcing local features and integrating multi gulliiirranllritty graph-based representations.
Effect of TCTC for Gloss-Free SLT. To demonstrate the effectiveness of our proposed TCTC mechanism, we compare a SLT model trained end-to-end (i.e., NOT using TCTC) and the same model pre-trained with TCTC, in terms of SLT performance. As shown in Table 2 our model pre-trained with TCTC significantly outperforms the one trained end-to-end, e.g., our model improves gloss-free SLT performance by 23.14 ROUGE score and 19.8 BLEU4 score on PHOENIX 14T dev set. In addition, PHOENIXI4T and CSL-Daily, the performance of our model pre-trained with TCTC is close to that of the gloss-based SLT model (i.e., pre-trained with gloss annotations), e.g., 55.77 ROUGE and 54.54 ROUGE score. It indicates that the proposed TCTC mechanism substantially bridge the performance gap between GFSLT and gloss- based SLT.
4.2 Qualitative Results
CSLR Qualitative Results. As shown in Table 3 we conduct qualitative analysis for MixSignGraph in the CSLR task, and show one sample from the test set of PHOENIX14, PHOENIX14T and
7

---

Table 5: Comparison of CSLR performance PHOENIX 14 and PHOENIX144 14T datasets. (F: face,
M: mouth, H: hands, St skeleton, P: pre-training backbone with ImageNet, v": pre-training on other datasets. Same applies the tables below.)
Extra cues
PHOENIX14
PHOENIX14T
Model
Backbone
TEST
DEV
TEST
FM
P
WER
del/ins
WER
del/ins
WER
WER
STMC (1)
VGGIT ResNet18
21.1
7.7/3.4
20.7
7.4/66
19.6
21.0
C2SLR 50
20.5
20.4
20.2
20.4
TwoStream
S3D
18.8
17.7
19.3
CrossL--Two
S3D
15.7
16.7
16.9
18.5
CrossL-Single 3
S3D
20.6
21.3
RTG-Net
RepVGG
20.0
8.4/1.5
20.1
8.6/1.7
19.63
Joini-SLRI
GooleNet
24.6
24.5
TwoStream
S3D
22.4
-/-
23.3
21.1
22.4
VAC (
ResNet18
21.2
7.9/2.5
22.3
8.4/2.6
SMKI)
ResNet18
6./22.5
21.0
6.3/2.3
20.8
ConNet (32
ResNet18
18.8
5.6/2.8
19.4
5.7/2.3
18.9
20.5
FCN (33
Customed
23.7
-/-
23.9
Contrastive
ResNet18
5.1/2.7
19.8
5.8/300
20.0
HST-GNN
Customized(CCN)
19.8
19.5
19.8
CoSign 35
ST-GCN(GCN)
19.7
-/-
20.1
19.5
20.1
SignCiraph
Customized(GCN)
18.2
4.9/2.0
19.1
5.3/19
17.8
19.1
MixSignGiraph
Customized(GCN)
4.9/2.0
17.3
4.9/2.2
16.7
19.0
CSL-Daily, respectively. It can be found that MixSignGraph yields more accurate gloss predictions than MultiSignGraph, demonstrating that the newly- proposed MixSignGraph is more effective.
SLT Qualitative Results. As shown in Table we present qualitative analysis of MixSignGraph in the SLT task, and show one sample from the test set of PHOENIXI41, CSI.-Daily, How2Sign and OpenASL, respectively. It shows that the gloss-based Sign2Texx model achieves the best performance while MixSignGraph pre-trained with TCTC also achieves satisfactory translation results.
4.3 Comparisons on CSLR Tasks
Evaluation on PHOENIX14T.
As shown in Table 5 we com- pare our model with existing models on CSLR performance, and we provide both the per- formances on the validation set and test set. Most of the cur-
Table 6: Comparison of CSLR performance CSL-daily.
CSLR
Backhone
cues
DEV
TEST dellins
S
P
WFR
dellins
WER
Joint-SLRT (3
GoogleNet
33.1
10.3/44
32.0
9.6/4.1
TwoStream
S3D
25.4
25.3
-
TwoStream
7
S3D
28.9
-/-
28.5
-
GeogLeNet
13.9/3.4
13.5/3.0
rent models adopt existing CNN- based backbones, and achieve good performance by injecting extra cues [1 77 30) adding
CorNNer 99
ResNet18 8
30.6
-/-
30.1
-/-
Contrastive
ResNet18
11.5/3.0
11.2/3.5
CoSign (35
ST-GCN
28.1
-/-
27.2
-/-
SiguCimph 4
GCN
27.3
7.9/2.3
26.4
7.8/2.1
SinSighrraph
GCN
25.1
6.4/2.1
25.0
7.0/1.6
tra constraints (6 (5) or intro-
ducing attention mechanism (32). As for the GCN-based models, CoSign (35) mainly relies
pre-processed fine-grained skeleton data, and achieves 19.5%, 20.1% WER on dev, test set. While HST-GCN (34) adopts both CNN-based backbone and GCN-based backbone to extract RGB features and skeleton features respectively, and achieves 19.5%, 19.8% WER on dev, test set. is worth men- tioning that the SOTA model CrossL-Two (3) utilizes both RGB features and fine-gre me--griieed skeleton features (i.e., keypoints hands, body and face), and pre-trains its backbone on both PHOENIXI4 and CSL-daily datasets, achieving 16.9%, 18.5% WER dev, test respectively. When moving to our MixSignGraph, it does not use any extra cues or pre-train on other SL datasets, but still achieves comparable performance with CrossL-Two on test set. While on dev set, our model even outperforms CrossL-Twe model by 0.2% WER.
Evaluation on PHOENIX14. We conduct comparative analysis of our model and current CSLR models using the PHOENIX14 dataset. As illustrated Table 5 our straightforward yet robust MixSignGraph model achieves the superior performance (16.5% WER) on the dev set and surpasses the majority of existing models, thus demonstrating the effectiveness of our MixSignGraph.
Evaluation on CSL-daily. We also show CSLR performance of our model on CSL-daily dataset. Table shows that our model achieves 25. 1,, 25.0% WER on the dev, test set, respectively. indicates that model surpass the SOTA model by only using RGB modality.

---

Table 7: Comparison of SLT performance PHOENIX14T dataset.
Extra
PHOPNIXI4T
Sign2Glos2Text
ROUGE
BLEUZ
BLEUN
44.14
42.88
M0.40
21.02
18.40
41.30
45.25
10.39
22.83
18:15
Join-SLRR
47.73
34.82
27.11
22.11
45.47
35.35
27.57
22.45
SignBT 14]
49.53
49.33
36.43
28. 66
25.51
49.35
48.55
36.13
28.47
23.51
SIMC-Traut 36
48.27
35.20
27.47
22.47
46.77
36.55
29.03
24,00
MMTI 22
50.36
24.63
37.28
29.67
25.
TwxSineam-S1
52.01
52.35
39.76
31.85
26. 26-47
$1.59
39.81
52.00
26.71
MisSignGrrpp
52.59
$2.40
39.54
31.95
25.56
51.46
52.35
39.23
32.26
26.04
Sign2
BLEUS
BLEUZ
BLBU3
BLEU4
ROUGE
BLEU1
BLEU2
BLEBUS
Join-SL..
HST-GINN
34,20
STMC-T. (To
47.60
36.09
28.70
SignD' [10
50.29
51.11
37.90
29.80
24.45
49.54
50.90
37.25
29.72
24.32
CoSLRT 65
52.47
$2.29
39.60
31.34
27.83
52.24
52.48
41.17
32.30
27.88
MMIL8 [2)
53.30
53.95
41.12
33.14
27.61
52.65
53.97
41.75
33.84
28.39
Twestreamn-S1 SI
$4.08
54.3
41.99
34.15
28,
$3.48
54.90
42.44
34.46
28.95
MisSignGraph
55,77
55.01
34,98
29.00
53.84
58,90
12.53
34,50
28.97
Gloss-free SLT
FM
H S
ROOGI
RI FU71
BL,F122
RUB
FF14
ROUGE
B1,F111
BUFFU2
BI.F113
RI F114
St.-Loox
31,80
31.87
19.11
13.16
9.94
31.80
32.24
19.03
12.83
9.58
1SPNet 3d
34.96
36.10
23.12
16.88
13.41
GASLT 16
39.86
39.07
26.74
21.86
15:74
CSGCK
18.65
GFSLTT
GFSLT
Sign2GPP (1)
22.52
MisSiguGimpe
51.71
$1.07
37.97
29.98
24:87
51.14
30.01
38.04
29.95
24,02
Table 8: Comparison of SLT performance on CSL-Daily dataset.
cues
Daily
SignXGlnseTeext
ROUGE
BLBUZ
ROUGE
SL-Loong
40L1S
-1.45
25.71
16.37
11.00
20115
+1.35
25.73
10.35
11.03
Inins-SLRR
X1
44.18
45.82
32:22
22.40
1..04
44:81
47.09
32.49
22.61
16:24
SignET [3]
48.38
50.97
35.16
25.26
19.53
65.21
50.68
36.00
26.20
19.67
MMTLD 22
51.35
50.59
37.56
28.53
21.55
51.43
50.33
37.44
28.08
21.46
TweSream SLT
53.91
53.55
40.49
31.67
23.71
54.92
54.08
41.02
31.18
24.13
MisSiesCraph
53.48
53.73
41.53
30.30
23.36
53.65
$4.07
41.04
31.32
24.44
SignZTest
RM
S
ROUGE
BLLBU
BLBU2
BL.BU
BL.B14
ROUGG
BLEBU
BLEBL2
BLEEU
BLEB4
SLRT
37.
SignET
Conmrstive
50.73
27.89
MMTIAA
TooSSneannSii
MisSipnOrph
54.54
$5.87
42.45
32.75
25.77
$4.67
55.41
12:43
32.84
25.87
ROTIOR
RI.FE
SL-Laoong X
34.28
34.22
19.72
12.34
7.96
34.54
34.16
19.57
11.84
7.56
GASLT (16
20.35
19.90
9.94
5.98
4.07
GESLT [
36.44
39.20
16.35
36.70
39.31
24.93
11.00
SignXCPP
GFSLI- Sightl
49.76
Signl.1M
Misph
Table 9: Comparison of SLT performance on How2Sign dataset.
Hogn
Gloss-free SLT
DEV
TEST
10M
8
ROUGE
BEPOT
BLB02
BLEDS
BLE14
ROUGE
BL.611
BL.UU
BL.E1
BLE11
SLT-IV 1S
35.20
20.62
13.25
8,89
19.30
12.18
8.03
You.hhhe-SITT
T 435
14.96
5.11
2.26
1.22
SSVP-SLT 53
25.70
30.30
16.70
10.50
7,00
GloFE-VW VN
12.98
15.21
7.38
4,07
2,37
12.11
14.94
7.27
3.93
2.24
3488
11.28
10.41
4.4 Comparisons on SLT tasks
Evaluation on PHOENIXI4T Dataset. As shown in Table 7 we compare our model with existing models using PHOENIXI4T dataset on SLT tasks, which include Sign2Gloss2Text, Sign2Text, and Gloss-free Text2Text (also known as Gloss-free SLT). Our MixSignGraph achieves comparable performance to the SOTA model (TwoStream) on both Sign2Gloss2Text and Sign2Text tasks, When moving to the gloss-free SLT task, our MixSignGraph pre-trained with TCTC mechanism significantly surpasses the current SOTA model, e.g., 8.65 ROUGE score higher than GFSLT-VIL (16) on test set. surpasses Besides, MixSignGraph also narrows the performance gap between the gloss-free SLT model and the gloss-based SLT model, i.e., 51.14 ROUGE score vs. 53.84 ROUGE score.
Evaluation on CSL-daily. As shown in Table we also show the SLTT performance of our MixSign- Graph on CSL-daily dataset. It can be found that MixSignGraph outperforms the most of current

---

Table 10: Comparison of SLT performance OpenASL dataset.
OpenASI.
Glose-free SLT
TEST
ROUGE
BLEUT
BL.F0
BLEF1
ROUGE
BLE01
BLEU2
BL.EOS
BL.
(2)
16.72
16.10
16.11
BD-rranfommmm
18.26
10:26
7.17
18.31
10.15
7.19
Open
25.31
24.35
14.94
10.72
8.39
24.83
23.87
14,08
9.50
7.54
2p
v
20.43
20.10
11.81
8.43
6.57
21.02
20.92
12.08
$.59
6.72
GloFE-VN
21.37
21.06
12.34
3.08
6.68
21.75
21.56
12.74
9.05
7.06
Ouns(15D)
25.41
26.82
16.70
11:48
8.36
25.71
26.65
16.55
11.68
8.69
models on all the Sign2Gloss2Text, Sign2Text and gloss-free SLT tasks, especially the gloss-free SLT task (i.e,, 2.7 ROUGE score higher than the SOTA model).
Evaluation on How2Sign. How2Sign is large American Sign Language dataset. Considering that the How2Sign dataset only provides sentence labels (i.e., NO gloss labels), we compare our proposed model with existing models in terms of gloss-free SLT performance. Note that we adopted the I3D features provided by previous work to accelerate training, which may compromise the SLT performance. As shown in Table our MixSignGraph pre-trained with TCTC achieves BLEU4 score of 1 11.28/10.4. 10.4 on the dev/test set, outperforming the baseline model (which also uses I3D features) and thus establishing new baseline for future work.
Evaluation on OpenASI. Similar to the How2Sign dataset, OpenASI. is also large dataset that contains only sentence labels and has a very rich vocabulary. We compare our proposed model with these baseline models, which adopt I3D as backbone for fair comparisons in terms of gloss-free SLT performance. As shown in Table 10 our model outperforms the baseline models and provides new baseline for future work.
5 Conclusion
We propose MixSignGraph, simple yet effective architecture that models sign sequences as graphs for feature extraction. To address local feature degradation caused by cross-region modeling in SignGraph, we design a Hicrarchical Sign Graph (HSG) module that connects corresponding regions across multi-grame uulti-graauulrity features by bidirectional information exchange. We further introduce Text-based CTC Pre-training (TCTC) strategy that generates pseudo-glos gouus- sequences via simple NLP processing. Experiments on five benchmarks across CSLR and SLT tasks show that our model achieves strong performance without relying any additional cues.
Limitations and Broader Impacts
Here, we list some potential ideas that can be further explored to improve the performance of SL tasks. Our TCTC mechanism for the gloss-free SLT task works well based on processed text, but it also encounters problems such as difficulty in CTC loss convergence, especially in cases with large vocabularies. Therefore, more effective methods to obtain pseudo labels for CTC pre-training are expected. In addition, we also highlight the potential negative social impacts. First, our method may experience unpredictable failures, so it should not be used in scenarios where such failures could have serious consequences. Second, our method a data-driven approach and its performance may be influenced by biases in data, thus, caution is advised in the data collection process.
7 Acknowledgments
This work supported in part by National Natural Science Foundation of China under Grant Nos. 62172208, 92467202, 62272216; Natural Science Foundation of Jiangsu Province (Key Program) under Grant No. BK20243040. This work partially supported by Collaborative Innovation Center of Novel Software Technology and Industrialization. Training and testing were partially supported by Sulis Tier HPC platform hosted by the Scientific Computing Research Technology Platform the University of Warwick (EPSRC Grant 1P//022108// and the HPC Midlands+ consortium).
10

---

References
[1] H. Hu, W. Zhou, and H. Li, "Hand-modellawarc sign language recognition," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, 2021, pp. 1558-1566.
[2] Y. Min, A. Hao, X. Chai, and X. Chen, "Visual alignment constraint for continuous sign language recognition," " in ICCV, 2021, pp. 542-11
[3] F. Wei and Y. Chen, "Improving continuous sign language recognition with cross-lingual signs." in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 23 23 621.
[4] S. Gan, Y. Yin. Z. Jiang, H. Wen, L. Xie, and S. Lu. "Signgraph: A sign sequence is worth graphs of nodes," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 13 474--11347.
[5] N. C. Camgoz, S. Hadfield, O. Koller, H. Ney, and R. Bowden, "Neural sign language translation," in CVPR, 2018, pp. 7784-7793.
[6] S. Gan, Y. Yin, Z. Jiang. K. Xia, L. Xie, and S. Lu, "Contrastive learning for sign language recognition and translation." in Proceedings of the Secony International Joint Conference Artificial Intelligence, IJCAI-23, 2023, pp. 763 772.
[7] Y. Chen, R. Zuo, F. Wei, Y. Wu, S. Liu, and B. Mak, "Two-stream network for sign language recognition and translation," " Advances Neural Information Processing Systems, vol. 35, pp. 17 043-17 056, 2022.
[8] L. Tarrés, G. I. Gállego, A. Duarte, J. Torres, and X. Giró-i Nicto, "Sign language translation from instructional videos," Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023, pp. 6624-5634.
[9] S. Gan, Y. Yin. Z. Jiang, L. Xie, and S. Lu, "Skeleton-aware neural sign language translation," in MM, 2021. pp. 4353 43611.
[10] H. Zhou, W. Zhou, Y. Zhou, and H. Li, "Spatial-temporal multi-cue network for sign language recognition and translation," " TMM, 2021.
[11]
"Spatial-temporal multi-cue network for continuous sign language recognition." in AAAI, 2020.
[12] S. Gan, Y. Yin, Z. Jiang, L. Xie, and S. Lu, "Towards real-time sign language recognition and translation on edge devices," in Proceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 4502 4512.
[13] S. Tang, D. Guo, R. Hong, and M. Wang, "Graph-based multimodal sequential embodding for sign language translation," TMM, 2021.
[14] H. Zhou, W. Zhou, W. Qi, J. Pu, and H. Ii, "Improving sign language translation with monolingual data by sign nack-translation? in CVPR, 2021, pp. 1316-1325.
[15] A. Hao, Y. Min, and X. Chen, "Self-mutual distillation learning for continuous sign language recognition." " in ICCV, 2021, pp. 11 303 312...
[16] A. Yin, T. Zhong, L. Tang, W. Jin, T. Jin, and Z. Zhao, "Gloss attention for gloss-free sign language translation," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2551-2566
[17] B. Zhou, 7. Chen, A. Clapés, J. Wan, Y. Liang, S. Escalera, 7. Lei,, and D. Zhang, "Gloss-free sign language translation: Improving from visual-language pretraining," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 20 20871-28881.
[18] J. Gong, L. G. Foo, Y. He. H. Rahmani, and J. Liu, "Lims are good sign language translators." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 18 362- 18 372..
[19] K. Han, Y. Wang. J. Guo, Y. Tang, and E. Wu, "Vision gnn: An image is worth graph of nodes." Advances in Neural Information Processing Systems, vol. 35, pp. 8291-8303, 2022.
[20] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber. "Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks." in ICML, 2006. pp. 369-376.
[21] L. Van der Maaten and G. Hinton, "Visualizing data using t-sne." Journal of machine learning research, vol. 9, no. 11, 2008.
11

---

[22] Y. Chen, F. Wei, X. Sun, Z. Wu, and S. Lin. "A simple multi-modalityy transfer learning baseline for sign language translation." in CVPR, 2022. pp. 5120-5130.
[23] Y. Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad. M. Lewis, and L. Zettlemoyer, "Multilingual denoising pre-training for neural machine translation." Transactions of the Association for Computational Linguistics, vol. 8. pp. 726-742, 2020.
[24] T. Kudo and J. Richardson, "SentencePiece:: A simple and language independent subword tokenizer and detokenizer for neural text processing." " Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, E. Blanco and W. Lu, Eds. Brussels, Belgium: Association for Computational Linguistics, Nov. 2018, pp. 66-71. [Online]. Available:
https///cclmmbology.orD/DD 2012
[25] O. Koller, J. Forster, and H. Ney. "Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signers," CVIU, vol. 141, pp. 108-125, Dec. 2015.
[26] A. Duarte, S. Palaskar, I.. Ventura, D. Ghadiyaram, K. DeHaan, F. Metze, J. Torres, and X. Giro-i Nieto, "How2sign: large-scale multimodal dataset for continuous american sign language." in Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, 2021, pp. 2735-2744.
[27] B. Shi, D. Brentari, G. Shakhnarovich, and K. Livescu, "Open-domain sign language translation learned from online video." in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022, pp. 6365-6377
[28] C.-Y. Lin, "Rouge: A package for automatic evaluation of summaries," in Text summarization branches out, 2004, pp. 74-81.
[29] K. Papineni, S. Roukos. T. Ward, and W.-J. Zhu, "Bleu: method for automatic evaluation of machine translation," " in ACL. 2002, pp. 311-318.
[30] R. Zuo and B. Mak, "C2str: Consistency ennhenced continuous sign language recognition," in CVPR, 2022, pp. 3131-5140.
[31] N. C. Camgoz, O. Koller, S. Hadfield, and R. Bowden, "Sign language transformers: Joint end-to-end sign language recognition and translation," " in CVPR, 2020, pp. 20023-10033
[32] L. Hu, L. Gao, Z. Liu, and W. Feng, "Continuous sign language recognition with correlation network," in CVPR, 2023, pp. 2529-2539.
[33] K. L. Cheng, Z. Yang. Q Chen, and Y.W. Tai, "Fully convolutional networks for continuous sign language recognition." " in ECCV. Springer, 2020.
[34] J. Kan, K. Hu, M. Hagenbuchner, A. Tsoi, M. Bennamoun, and Z. Wang, "Sign language translation with hicrarchical spatio-temporal graph neural network," in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp. 3367-3376.
[35] P. Jiao, Y. Min, Y. Li, X. Wang, I.. Lei,, and X. Chen, "Cosign: Exploring co-occurrence signals in skeleton-hard based continuous sign language recognition," in ICCV, 2023, pp. 20 6766200066 686.
[36] K. Yin and Read, "Better sign language translation with STMC-transformeer" in COLING, Barcelona, Spain (Online). Dec. 2020, pp. 5975-5989, [Online]. Available: http://www.ddwee..ggzzlnllg/y// coling-main.522
[37] D. Li. C. Xu, X. Yu, K. Zhang, B. Swift, H. Suominen, and H. Li, "Tspnet: Hierarchical feature learning via temporal semantic pyramid for sign language translation," in NIPS, vol. 33, 2020.
[38] Zhao, W. Qi, W. Zhou, Duan, M. Zhou, and H. Li, "Conditional sentence generation and cross-mode reranking for sign language translation." IEEE Transactions on Multimedia, vol. 24, pp. 2662-283 2272,6672222 2021.
[39] R. C. Wong. N. C. Camgöz, and R. Bowden, "Sign2gpt: leveraging large language models for gloss-free sign language translation," in ICLR 2024: The Twelfth International Conference on Learning Representa- tions, 2024.
[40] J. Ye, X. Wang. W. Jiao. J. Liang, and H. Xiong, "Improving gloss-free sign language translation by reducing representation density." in NewrIPS, 2024.
[41] D. Uthus, G. Tanzer, and M. Georg, "Youtube-ad" asl: A large-scale, open- doma-domin american sign language- english parallel corpus," Advances in Neural Information Processing Systems, vol. 36, pp. 29 029 29 047, 2023.
12

---

[42] P. Rust, B. Shi, S. Wang, N. C. Camgoz, and J. Maillard, "Towards privacy-aware sign language translation at scale," in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers). Bangkok, Thailand: Association for Computational Linguistics, 2024, pp. 8624 8641. [Online]. Available: https://ccamttooooy..2222224.44.
[43] K. Lin, X. Wang. L. Zhu, Sun. Zhang, and Y. Yang. "Gloss-free end-to-end sign language translation." arXiv preprint arXiv:2305. 12876, 2023.
[44] R. Geirhos, Rubisch, Michaelis, Bethge, F. A. Wichmann, and W. Brendel, "Imagenet-trained cans are biased towards texture; increasing shape bias improves accuracy and robustness," in International Conference on Learning Representations, 2019.
[45] N. C. Camgoz, O. Koller, S. Hadfield, and R. Bowden, "Multi-channel transformers for multi-articuulaoryy sign language translation," in ECCV. Springer, 2020, pp. 301-319.
[46] J. Pu, W. Zhou, H. Hu, and H. Li, "Boosting continuous sign language recognition via cross modality augmentation," in MM, 2020, pp. 1497-1505.
[47] O. Koller, C. Camgoz, H. Ney, and R. Bowden, "Weakly supervised learning with multi-stream enn-lstm- hmms to discover sequential parallelism in sign language videos," TPAMI, 2019.
[48] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, "Language models are unsupervised multitask learners," 2019.
[49] T. N. Kipf and M. Welling, "Semi-supervised classification with graph convolutional networks," arXiv preprint arXiv: 0690009099900.... 2016.
[50] M. Parelli, K. Papadimitriou. G. Potamianos, G. Pavlakos, and P. Maragos, "Spatio-temporal graph convolutional networks for continuous sign language recognition," in ICASSP. IEEE, 2022.
[51] A. Dosovitskiy, I.. Beyer, A. Kolesnikov, D. Weissenbomn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly et al., "An image is worth 16x16 words: Transformers for image recognition at scale," in International Conference
Learning Reprexentations, 2020.
[52] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang. S. Lin, and B. Guo, "Swin transformer: Hierarchical vision transformer using shifted windows," in Proceedings of the IEEE/CVF international conference on computer vision, 2021. pp. 10012-10022.
[53] A. Vaswani, N. Shazeer, N. Parmar. J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. "Attention is all you need," in NIPS, 2017, pp. 5998-6008.
[54] S. Brody, U. Alon, and E. Yahav, "How attentive graph attention networks?" arXiv preprint arXiv:2105. 14491, 2021.
[55] W. Hamilton, Z. Ying, and J. Leskovec, "Inductive representation learning large graphs." " Advances in neural information processing systems, vol. 30, 2017.
[56] Y. Wang. Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon, "Dynamic graph can for learning on point clouds," " ACM Transactions on Graphics (log), vol. 38, no. 5. pp. 1-12, 2019.
[57] Y. Rong, W. Huang. T. Xu, and J. Huang, "Dropedge: Towards deep graph convolutional networks on node classification," " in International Conference on Learning Representations, 2020. [Online]. Available: https://opentevie..eettooo
"id=HHxllkKKPP
[58] T. Xiao, M. Singh, E. Mintun, T. Darrell, P. Dollár, and R. Girshick, "Early convolutions help transformers see better," Advances in neural information processing systems, vol. 34, pp. 30 3900-300000 400, 2021.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly summarize the main contributions the paper, including the proposal of HSG and the TCTC training strategy. (see Sections 1 3)
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We include a dedicated "Limitations and Borader Impacts" section in the main paper, where we discuss practical challenges, and the potential negative social impacts including high resource demands, data bias, and model reliability in real-world applications. (see Sections 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification: Our work focuses on empirical model design and evaluation for sign language translation. does not include theoretical results that require formal assumptions or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide comprehensive details of the model architecture, training settings, and evaluation protocol (Section 4). These include datasets, hyperparameters, and imple- mentation details necessary to reproduce our results. We also plan to release the code and model checkpoints upon publication.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
15

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We plan to release the full codebase, model checkpoints, and detailed instruc- tions for reproducing all experimental results, including data preprocessing. training, and evaluation scripts.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
public//gulas//Cdeeebbl.sss wow.eosbbssoioooocoooooooooo.oie for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips https///cccpllicgggeeeeeeeeeeeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The paper provides comprehensive training and testing details, including data splits, model architecture, hyperparameters, learning rate, optimizer, and training epochs.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
16

---

Justification: While we report the main experimental results, we do not include error bars or statistical significance tests due to limited computational resources. In future work, we plan to provide more extensive statistical analysis over multiple random seeds or cross-validation splits.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper specifies the type of computing resources used, including GPU model (e.g., 3090), memory (e.g., 24GB) (See Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conforms in every respect with the NeurlPS Code Ethics. No part of our work involves deception, human subjects, sensitive data, potentially harmful outcomes. We have ensured transparency, reproducibility, and fairness our methodology and reporting.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics. If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
17

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
Our work discusses both potential positive and negative societal impacts.
Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The model weights used in this work are trained solely on publicly available datasets, and we do not release any new datasets or pretrained models that pose high risk for misuse. Therefore, no additional safeguards are necessary.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
18

---

Answer: [Yes]
Justification: We used publicly available datasets and pretrained model weights, all which are properly cited in the paper. The licenses and terms of use for these assets carefully reviewed and respected. We have also specified the sources, versions, and relevant URLs where appropriate.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, ppppesswitheeee throde com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release any new datasets, models, code assets. All experiments are based on publicly available datasets and pretrained models, which are properly cited.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve any crowdsourcing or research with human subjects. All the datasets used in this paper are publicly available.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
19

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
This paper does not involve crowdsoureing or research with human sub- jects. All datasets used are publicly available and do not contain personally identifiable information.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [No]
Justification: Our core methodology does not involves Large Language Models (LLMs) as central component.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https://neurip: auttep///earrrssccc///foeeeerrre for what should or should not be described.
20

---

A Appendix
A.1 Related Work
Sign Language Tasks. The unique grammatical rules of sign language lead to some differences between the frameworks of CSLR and SLT tasks. Nevertheless, both frameworks contain a visual backbone to capture visual features from vidcos. However, it is a challenging task for the widely adopted CNN-based backbones to focus sign-related features, since CNN backbones are initially designed to capture the texture and contours of objects (44). To guide CNN-based backbones to extract sign-related features, some current SL models introduce prior knowledge (e.g., local areas (45), skeletons (g 30 7 and depth images (13) to manually inject expert knowledge into the backbone, making it focus on SL-related areas. Some other SL models try add additional constraints (e.g., knowledge distillation (15) 2 and contrastive learning (6) to optimize their backbones. Besides, there are also some SL models that turn their attention to obtaining more sign language samples by back translation (14), cross modality augmentation (46); pre-training with other sign language related datasets (22 3) and adopting pre-trained visual models as their backbones (39). To capture effective
SL-related features, SignGraph (4) represents frames as nodes and builds LSG and TSG modules to capture cross-region features both in one frame (i.e., the correlation of hands and face regions) and among adjacent frames (e.g., hand motions, facial expression changes).
In CSLR tasks, the CTC module (47) is adopted align gloss sequences with SL videos. While SLT (5) models often adopt a pre-trained language model (e.g., mBART (23), GPT2 (48) to get the translated sentence. In these SLT models, the traditional training paradigm usually involves two stages: first, pre-training their backbone using gloss sequences (equivalent to the CSLR task), then connecting a pre-trained translation model and fine-tuning the entire model. In the first stage, labeled gloss sequence is critical for improving SLT performances, as pre-training with gloss sequences can assist the model in learning segmentation and semantic information of SL. However, annotating gloss sequences for SL videos is labor-intensive and cumbersome. Therefore, gloss-free SLT tasks have gained attention (66 17 39 40). The existing gloss-free free SLT models usually adopt attention mechanisms (16) or contrastive moguage-image pre-training (17) to improve SLT performances, and have achieved promising results.
Graph Convolutional Network. Due to the graph structures, graph convolution layers (49) are more flexible for message passing and aggregation. Naturally, GCN has been effectively applied to prcleton-based data, where the connections between joints render the skeleton data inherently suitable for GCN. With the skeleton data, GCN has been applied to SL tasks, in which some models utilized GCNs to guide CNN-based backbones to extract skeleton-related features ( 50, 34 B5). However, these approaches often rely heavily on structured inputs (e.g., skeleton). To reduce the dependence structured data, the recent model VIT (51). which processes an image patches with a transformer model, has replaced CNNs as a de facto architecture in many fields. After that, the VIG model (19) directly represents an image as a graph structure to capture irregular and complex objects, and achieves superior performance on image recognition tasks.
A.2 Datasets
Our experiment evaluation is conducted on five publicly available SL datasets, including three widely used datasets (i.e., PHOENIX, PHOENIXI4T and CSL-Daily) and two new and large datasets (i.e.. How2Sign and OpenASL),, as described below.
PHOENIX14 (25): A widely used German SL dataset with 1295 glosses from 9 signers for CSLR. It includes 5672, 540 and 629 weather forecast samples for training, validation, and testing, respectively.
PHOENIX14T (5): German SL dataset with both gloss annotations and translation annotations. It contains 7096, 519 and 642 samples from 9 signers for training, validation and testing, respectively. In regard to the two-stage annotations, the sign gloss annotations have a vocabulary of 1066 different signs for CSLR, while the German translation annotations have vocabulary of 2877 different words for SLT.
CSL-Daily (1): A Chinese SL dataset with 18401, 1 1176, 1077 labeled videos from 10 signers for training, testing and validation. It includes 2000 gloss annotations for CSLR and 2343 Chinese words for SLT.
21

---

Dataset
Train
Test
Samples
Validation
Train
Gloss Vacabularr
Test
Validation
Train
Token Vocabulary
Test
Validation
Train
Vocabllry
Test
Validation
PHORNIXI4
25
5672
629
540
1103
497
462
PHOENIXI4T
1085
CSL-Daily 14
18401
1176
1077
2000
1345
1358
2342
1358
1344
2332
1351
How
26
30904
2328
1713
8816
3312
3046
9098
2573
2302
OpenASL
3336
Table 11: Details of datasets used our paper.
Figure 5: Visualization of graph construction: LSG and TSG modules (LSG, and TSG: in the first row, LSG and 2S2 in the second row) are in the left part, HSG module is in the right part. The graph in LSG, TSG and HSG module is shown in yellow, blue and green, respectively. We also show some 'unimportant" edges between nodes of the background in red color.
A multimodal and multiview continuous American Sign Language (ASL) dataset. consists of a parallel corpus, which has more than 80 hours of sign language videos and set of corresponding modalities (i.e., speech, English transcripts, and depth). In this dataset, there are 31 128, 1741 and 2322 samples for training, validation and testing, respectively. In regard the annotations, there are no gloss-level annotations, thus the dataset is used for SLT tasks. It is worth mentioning that our experiment only uses the RGB modality (i.e., NOT all modalities) with frontal view (i.e., NOT all views) train and test our model.
How2Sign (6):
OpenASL (27): A large ASL dataset with translation annotations for SLT. It includes 280 hours of ASL videos from more than 200 signers, and contains 96476, 997 and 999 samples for training, validation and testing, respectively.
We provide detailed information the five datasets in Table I1 In addition, we also provide the token vocabulary obtained by the mBART tokenizer and the vocabulary of pseudo gloss labels in TCTC for reference. It is worth noting that the datasets we downloaded (i.e., shown in Table [T) may slightly differ from the official versions described in their own papers.
A.3 Evaluation Tasks.
We evaluate the performance of our model on the following tasks: CSLR and SLT (including Sign2Gloss2Text, Sign2Text, Gloss-free Sign2Text. (1) CSLR: Recognizing a sign sequence as corresponding gloss sequence, also known as Sign2Gloss. (2) SLT: Translating sign sequence into spoken sentence, and it can be classified into the following three categories: (2.1) Sign2Gloss2Text:: First, the recognized gloss sequence is obtained based on the CSLR model. Then, the predicted glosses are translated to spoken sentence by translation model. (2.2) Sign2Text: Directly translating SL video into a spoken sentence. Gloss annotations are required for pre-training a CSLR model, and visual features output by the CSLR model are sent to translation model for generating the spoken sentence. (2.3) Gloss-free Sign2Text: Directly translating a SL video into spoken sentence, while NOT using gloss annotations to pretrain CSLR model. This task is also known as gloss-frre free SLT.
22

---

Table 12: Model details.
Modd
Visual backbone Translation model FLOPs (G) MM
Pre-trinned Dataset
CroaL-TWoo
CSLR
Dual S3D
CSL CSL-Daily Duull-PPoomis 144Phoomix141
Sign2GPT
Gloss free SLT
Dino-V2 VIT
GPT2
1771,65
LND- 122eeeeTeet
TwoSircum
CSLR+52T-52G22
Daal
323.41
38
Kimktia-1004CCC
MulhiSinngraph S2tesSST+222GTggaaaaa free SLT
SignGraph SignGraph
mBART mBART
256.13
395.80
ImageNet-11+CCOO
MisSigngragh CSLR+S2T+S202T= 2a0-240-222------eeoe free SLT
320.88
402.20
ImageNet-1l+CC25
A.4 Visualization of MixSignGraph
To verify whether model can effectively capture sign-related features, select a sign video from the PHOENIX14T test set and visualize the constructed graph structure in both stages. In Figure 5 show the graphs in the LSG, TSG and HSG modules. For clarity, only a subset of the edges is displayed. In the LSG module, our model links nodes with similar contents or related semantic representation (e.g., hand regions and face regions), extract better intra-frame cross-region features. In the 7SG module, our model builds edges among adjacent frames to track dynamic changes in gestures and facial expressions, to capture inter-frame cross-region features. In the HS module, our model connects the same regions with different granularities (i.e., the corresponding regions in feature maps). enhance one-region features. In addition, we also highlight some 'unimportant" edges between nodes in the background in red. As seen in Figure the background nodes in the LSG module are naturally connected to their neighboring nodes, and there are still few background nodes connected in the TSG module. Fortunately, background nodes do not 'disturb' nodes in sign-related regions, demonstrating the effectiveness of our model.
A.5 Model Comparison and Analysis
To provide a more comprehensive analysis, we compare our model with the existing models in the following aspects: tasks, architecture information, flops, number of parameters, and pre-trained datasets. As shown in Table [2] among the existing models, the TwoStream model adopts a dual S3D backbone and pre-train their backbone with Kinetics-400, which is an action recognition dataset. Based on the backbone in TwoStream, the CrossL-TWe model further pre-trains their backbone with existing SL datasets. Sign2GPT adopts Dino-Vz V2 ViT as the visual backbone, which is pre-trained on LVD-142M (a very large vision dataset), and adopts GPT2 as the translation model, which is also pre-trained on a very large text dataset WebText. regard to MixSignGraph, it contains fewer parameters and FLOPs, and it is only pre- pre-trained on small datasets (i.e., ImageNet- IK and CC25 datasets), while achieving the promising performance on multiple SL. tasks.
A.6 Other Ablation Study
Effect of Hyperparameter K. In the MixSignGraph model, four hyperpa- rameters: K? and Kł in the LSG mod- ules, K; and K? in the TSG mod- ules need be set to proper val- ues. To determine the best values for these hyperparameters, uate the CSLR performance by chang- ing K₁ (i.e., both K} and K3), Ke (i.e., both KJ and K2) values from 2 to 9,7 to 91. respectively. However, due to the large number of possible combina- tions of K/. K?, K! and K?, it is im- practical to make an exhaustive search to find the globally optimal values of these hyperparameters. Therefore, we fix three of the parameters at a time and adjust the fourth to find suitable values. Finally, based on the WER performance on the DEV set, we set K,, K2 K2 and K? to 3, 4, 49 and 49, respectively, as shown in Figure 6
29.0
28.0
13.0
13.0
19.5
Dev
WER
18.5
Dev
17.0
17.0
K}
eval-
23.
19.3
33.0
WER
Test
17.
KF
K?
Figure 6: Effects of K₁ and K..
Effect of Graph Type. In MixSignGraph, we build dynamic dense graphs (i.e., connecting the top neighbors for each node) in the LSG module, dynamic sparse graphs (i.e., connecting only the top K
23

---

Table 13: Effect of porposed graph modules
(a) Effect of graph types.
(b) Effect of order of proposed sign graph modules.
LSG, 7SG, BSG;
Dev
Test
Model
Dev
Test
WER Del/ins
WER Del/Ins
WER
Del/ins
Dense Spurse Fixed
16.7 4.92.1
19.0 4.9/2.9
HSG + LSG - TSG
16.8 5.1/2.0
19.6 5.0/3.2
Dense Dense Fixed
19.3 4,9/2.9
20.1 5.2/2.6
Sparse Sparse Fixed
19.0 5.3/2.1
20.2
5,4/2.1
IISG -, TSG LSG
16.9 5.1/2.1
19.5 4.9/3.2
Sparse Dense Fixed
20.2 5.3/.6
20.6 5.5/2.9
LSG - HSG - TSG
17.2
6.3/2.0
19.8
4.8/2.7
Sparse Dynamic
18.1 5.1/2.2
5.1/2.1
LSG - TSG - HSG
17.1
5,4/2.0
19.0 5.0/2.7
Dense Dense Dynamic
19.9 5.6/3.5
21.8
5.5/2.6
TSG + HSG + LSG
16.9
5.5/1.8
19.3
5.0/3.2
Sparse Sparse Dynamic
19.1 5.0/19
21.1
5.8/2.1
Sparse Dense Dynamic
20.7 6.0/2.1
21.5 6.0/2.5
TSG LSG HSG
16.7 4.9/2.1
19.0 4.9/2.9
Table 14: Ablation experiments SignGraph on PHOENIXI4T dev set.
(a) Backbone. Comparison of dif- ferent backbones.
(b) Distance function. Compari-
Graph Convolution Effect of different GCN layers.
son distance functions.
BackBone WER Del Ins
Distance
WER
Del
Ins
GraphConv
WER Del Ins
Swin' (52)
45.4 16.3 1.3
Cosine
17.4
5.5
2.0
GATv2Conv (54)
17.0
5.1
2.0
PyVIG (19)
35.4
12.1 1.3
Chebyshev
17.3
5.1
3.2
SAGEConv 55)
17.7
5.9
1.9
SA (33
39.2
15.3
0.9
Euclidean
16.7
4.9
2.1
GCNConv 49)
17.7
5.5
2.0
Ours 16.7
4.9 2.1
EdgeConv (56)
16.7 4.9
2.1
(d) Patch size. Comparison of dif- ferent patch sizes with one LSG, TSG and BSG modules.
(e) Multiscale SignGraph. Effect of the number of stages multi- scale SignGraph.
(1) Drop edge. Adding DropE-
dge (57) does not improve perfor-
mance.
PatchSize
WER Del Ins
Stages
WER
Del
Del
DropRate
WER
Del
Ins
8
17,4
5,4
3.0
16-+32
16.7
4.9
2.1
0
16.7
4.9
2.1
16
17.1
5.3
2.0
8-+16-32
17.1
5.2
2.0
15%
17.7
6.1
1.4
32
17.5
5.7
2.4
4-8--16
>32
17.8
5.4
3.1
30%
17.5
6.0
2.1
node pairs) in the TSG module, and fixed graphs (i.e., connecting the same regions between feature maps) in the HSG module. To verify the effectiveness of the above graph types, we evaluate the CSLR performance by changing the graph types in LSG, TSG and HSG modules. For simplicity, we only use one LSG module, one TSG module and one HSG module in the experiment. As shown in Table T3a when using a spare graph in the LSG module, dense graph in the TSG module, or dynamic dense graph (i.e., building in the similar way with the LSG module) in the HSG module, there noticeable performance drop. That say, it is effective to adopt dense, sparse and fixed graphs in the LSG, TSG and H SG modules, and the combination of these mixed graphs can make our model achieve the best recognition performance.
Effect of Ordering of Sign Graph Modules. We also evaluate the CSLR performance by changing the ordering of three graph modules. As shown Table different orderings of the graph modules lead to slight differences in the CSLR performance, i.e., the ordering has little effeect the overall effectiveness of feature extraction and integration. Nevertheless, the ordering TSG LSG SS can help the model to achieve the best CSLR performance. Thus model MixSignGraph adopts the ordering TSG > LSG HSS by default.
Effects of Different Backbones. To verify the effectiveness of the MixSignGraph model, replace our backbone with other patch-based networks (e.g., CvT (58), Swin Transformer (52). ) or replace our graph modules with Self-Attention (SA) layers (53). As shown Table [4a] simply adopting the patch-based SOTA backbones or substituting our graph modules with SA layers does not lead appealing performances on the CSLR task. In contrast, our proposed MixSignGraph backbone achieves satisfactory performance by effectively capturing both cross-region and one-region features.
Effect of Distance Function. In the MixSignGraph, we use the KNN algorithm find the nearest neighbors for node/patch based on the distance between two nodes. To sclect suitable distance

---

function to measure the distance between nodes, we evaluate the CSLR performance with the following commonly used distance functions, i.e., Cosine distance, Chebyshev distance and Euclidean distance. As shown in Table 14b there are subtle performance differences, when using different distance functions. In MixSignGrraph, we adopt Fuclidean distance for better performance.
Effect of GCN Layer. To verify the effectiveness of the graph convolution layer, we test the CSLR performance by adopting the following representative variants of graph convolution, i.e. GATv2Conv (54), SAGEConv (55), GCNConv (49), and EdgeConv (56). As shown in Table T4c even using different GCN layers, our model still achieves good performance. It indicates that our GraSignrraph model has good flexibility in GCN layer selection. Nevertheless, among the above GCN layers, EdgeConv can help our model to achieve the best CSLR performance, i.e., 16.7% WER. Therefore, we adopt the EdgeConv layer MixSignGraph for better performance.
Effect of Patch Sizes. Patches with different sizes have different receptive fields, thus can capture different-range features and further affect the model performance. To observe the effect of patch sizes, we evaluate the CSLR performance by only adopting one L.SG module, one TSG module and one HSG module, while changing the patch size. As shown in Table [4d] when using patch size that is too small (i.e., 8 8x8) or too large (i.e., 32x32), the model MixSignGraph shows worse performance. We found that a suitable patch size strikes the best balance, leading to good performance.
Effect of Multiscale Sign Graphs. To leverage the vcale-invarian property of images (I9), VIG adopts a pyramid architecture that gradually increases patch size from 4 to 32 by shrinking the spatial size of feature maps to enhance feature extraction capability. Similarly, to extract better sign features and achieve higher CSLR performance, we gradually add LSSG, TSG and HSG modules after the early stage of the patchify stem. According to Table [4c] adding more LSG, TSG and HSG modules does not always bring performance gains. Therefore, MixSignGraph adopts two LSG, TSG and H SG modules, i..., increasing patch size from 16 to 32, aiming to achieve the best performance.
Effect of DropEdge. DropEdge (57) can alleviate the over-smoothing and overfitting problems in dense graphs by randomly removing a certain number edges from the input graph at each training epoch. Considering that we build dense graphs for the LSG and HSG modules, we introduce DropEdge into these two modules and change the drop rates of DropEdge to evaluate the CSLR performance. As shown in Table 14f adding DropEdge does not improve the performance of our MixSignGraph. It indicates that our model can achieve excellent performance without relying on external modules (e.g., DropEdge) and it can handle over-smoothing and overfitting problems well.
25


Reviews:
[{"review_id": "hdBwRa8g3Z", "reviewer": "Reviewer_MqJV", "summary": "This paper proposes MixSignGraph, a novel graph neural network architecture for sign language tasks, including Continuous Sign Language Recognition (CSLR) and Sign Language Translation (SLT). Building upon SignGraph, the authors introduce a Hierarchical Sign Graph (HSG) module to enhance local representations by aggregating same-region features across multi-scale feature maps. This design mitigates the degradation of local information typically caused by cross-region feature fusion. Furthermore, the authors present a simple yet effective Text-based CTC Pre-training (TCTC) strategy for gloss-free SLT, which generates pseudo gloss labels from raw text to guide model pre-training. Extensive experiments on five public datasets demonstrate the model’s superiority over existing state-of-the-art (SOTA) approaches, without relying on any additional modalities or annotations.", "questions": "Plz refer to the weaknesses.", "limitations": "Yes", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. Effective Enhancement of Local Representations via HSG:\nThe HSG module strengthens local feature quality by establishing hierarchical connections across multi-resolution feature maps, complementing spatial and temporal cross-region modeling.\n\n2. Innovative Gloss-Free Pre-training Strategy (TCTC):\nThe TCTC method enables pre-training using only textual labels, significantly improving gloss-free SLT performance and narrowing the gap with gloss-supervised models.\n\n\nWeaknesses\n\n1. Incremental Contribution over SignGraph.\nWhile the proposed method introduces performance gains by adding modules such as the Hierarchical Sign Graph (HSG) on top of SignGraph, the technical novelty appears limited. The framework largely builds upon prior work, and the architectural modifications, though effective, may not be conceptually groundbreaking.\n\n2. Insufficient Comparison with Recent State-of-the-Art.\nThe paper lacks comprehensive comparisons with several recent state-of-the-art models (e.g., [1, 2, 3]). Some of these models, including those that rely solely on pose or skeleton data, report superior or comparable results, yet are not adequately discussed or benchmarked against MixSignGraph.\n\n[1] Li, Zecheng, et al. \"Uni-Sign: Toward Unified Sign Language Understanding at Scale.\" arXiv preprint arXiv:2501.15187 (2025).\n\n[2] Jiao, Peiqi, Yuecong Min, and Xilin Chen. \"Visual Alignment Pre-training for Sign Language Translation.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[3] Chen, Zhigang, et al. \"C 2 RL: Content and Context Representation Learning for Gloss-free Sign Language Translation and Retrieval.\" IEEE Transactions on Circuits and Systems for Video Technology (2025).\n\n3. Semantic Inaccuracy in Pseudo Glosses.\nThe pseudo glosses used in TCTC are generated via basic linguistic preprocessing (e.g., lemmatization and tokenization), which may fail to capture the true semantics and structure of authentic gloss annotations. The paper should provide a more detailed analysis of how such semantic discrepancies affect performance, especially in gloss-free SLT tasks.", "was_revised": true, "final_justification": "While the paper introduces a promising framework, its core contributions remain incremental, and key claims—particularly regarding TCTC's effectiveness in gloss-free SLT—are not supported by sufficiently rigorous or fair comparisons. The evaluation emphasizes less meaningful metrics (e.g., ROUGE over BLEU-4), and recent relevant SOTA works are either overlooked or inadequately addressed. Additionally, if the focus is on improving visual representations, SLT may not be the most appropriate task for validation. For these reasons, I lean toward rejection.", "rebuttal": "Thank you for your insightful comments. Please find our point-by-point responses below.\n\n## Q1: Incremental Contribution\n\nWe would like to clarify that our work introduces two substantial and original contributions that go beyond a simple architectural refinement.\n\n\n(1) **Identifying and addressing a core limitation of SignGraph**: SignGraph builds LSG and TSG modules to model the intra-frame and inter-frame cross-regions features. While cross-region interactions have been shown to improve CSLR performance, we find that cross-region interactions may also weaken the feature representation and location sensitivity of a single/local region.\n\n- (a) The distinctiveness of a local region’s representation is disrupted and weakened by features from other connected regions. \n\n- (b) The spatial structure of regions (i.e., the location of a region, such as the hand area, within a frame) is also weakened by the interactions with other connected regions. \n\nTo alleviate the above issue, we propose the Hierarchical Sign Graph module, which aggregates the same-region features from different-granularity feature maps of a frame. By highlighting the features in a single region, which is located at a specific position within a frame, HSG is possible to enhance the distinctiveness and location sensitivity of a local region itself, even after cross-region feature extraction. This is not merely an incremental change, but a theoretical and technical rethinking of how region interactions should be modeled in sign sequences.\n\n\n\n(2) **Identifying and addressing a core limitation of Gloss-Free SLT (GFSLT)**:  Additionally, our approach extends beyond the task scope of SignGraph. While SignGraph focuses solely on CSLR tasks, our MixSignGraph can address both CSLR and SLT tasks(including Sign2Text, Sign2Gloss2Text, and GFSLT).\n\nMore importantly, in SLT tasks, we find that the current **bottleneck in Gloss-Free SLT** lies in the **continuous feature distribution** of SL videos, which does not align with the **discrete feature distribution** required by translation models, where the inputs are typically discrete tokens (see Figure 4 and Section 3.1). In contrast, gloss-based SLT benefits from gloss annotations with CTC constraints to effectively discretize SL features. Building on this insight, we propose a TCTC pretraining method, which leverages pseudo gloss sequences and CTC to guide the discretization of SL video features.\n\n\n ## Q2: Insufficient Comparison \n\nBefore making a comparison, we sincerely want to state the following important notes.\n\n-  *Contemporaneous comparison* : According to the **NeurIPS 2025 guidelines**[See NeurIPS website CallForPapers], papers published online after March 1st, 2025 are considered \"contemporaneous\"  i.e., NOT required for comparisons. While (1) Uni-Sign (published at **ICLR 2025, April 24, 2025**), \n(2) C2RL (published on **March 19, 2025**, [See IEEE 10933970]) can be considered contemporaneous.  \n\n-  *Possible unfair comparison*:  Uni-Sign utilizes large-scale SL data ( CSL-NEWs and YouTube-ASL) for pretraining, and adopts both video and pose data. In contrast, we do not use any SL-related datasets for pretraining, relying solely on video data. These differences between UniSign and our model may lead to an unfair comparison.\n\n \n\n| DataSet    | ours |ours(Full)  |C2rl| VAP | Uni-Sign*|\n|------------|------|---------|------|----|----------|\n| Phoenix14T |51.14|51.14    |50.96|51.28|-\n| CSL-daily  |49.93| 49.93   |48.21|48.72| 55.30  \n| How2Sign   |28.01| 39.46   |27.02|30.27| 49.35 \n| OpenASL    |25.71| 42.49   |31.36|42.47| -\n\n\nNevertheless, we also provide a detailed comparison and show the ROUGE score on the test set of four datasets, as shown in the table. \n\nIn the main paper, for Phoenix14T and CSL-daily datasets, we fully trained our MixSignGraph and reported the results. While for the large-scale How2Sign and OpenASL datasets, we used a pretrained I3D for feature extraction (see Section 4: Training Settings) to speed up training. **Importantly, I3D was not trained during the process** and this may lead to lower performances on How2Sign and OpenASL. Now, we further provide the full training results (i.e., ours(Full)) on How2Sign and OpenASL, to show the better performances achieved by our MixSignGraph.\n\nOur model MixSignGraph achieves promising performances across four datasets, especially the model with full training. In regard to the higher performance of Uni-sign, it may be caused by the unfair pretraining on other datasets and the usage of extra pose data, as mentioned before.\n\n\nWe will further cite these papers and provide a detailed analysis and discussion in the revised version of the paper. \n\n## Q3: Semantic Inaccuracy  \n\n- The pseudo glosses did introduce some semantic inaccuracies. However, directly generating gloss sequences from text without actual gloss annotations is nearly impossible, and these discrepancies are inevitable. Nevertheless, despite these inaccuracies, the proposed TCTC, with inaccuracies in pseudo glosses, can still effectively discretize video features and significantly improve Gloss-free SLT performance.\n\n- In Table 2 of the main paper (on page 7) (also shown below), we compare SLT performance under three conditions: using ground truth(GT) glosses (w/gloss), using pseudo glosses (TCTC), and not using pseudo glosses (w/o TCTC). The results indicate that, despite these inaccuracies, pretraining with pseudo glosses significantly reduces the performance gap compared to gloss-based SLT, demonstrating the effectiveness of TCTC.\n\n- **We also conducted detailed experiments to analyze how different levels of semantic discrepancies affect SLT performance.** First, we performed an ablation study on the proposed TCTC, including experiments with punctuation removal (TCTC w/o RP) and lemmatization removal (TCTC w/o LE). Second, we simulated different levels of semantic discrepancies by randomly corrupting the GT gloss sequence, i.e., using deletion, addition, replacement, and swapping at varying percentages. As shown in the table below, even under highly semantically inaccurate conditions (i.e., when glosses are highly corrupted up to 80%), the SLT performance still outperforms the baseline (w/o TCTC), demonstrating the effectiveness of TCTC.\n\n|Method|Rouge|BLEU1 |BLEU4|WER|Del/Ins|\n|-|-|-|-|-|-|\n| w/ gloss   | 53.84 |54.90 |28.97| 19.01| 4.9/2.99|\n| w/o TCTC |34.56|35.31 |9.40|-|- |\n| w/ TCTC    |51.14| 50.01| 24.02| 59.55 |35.9/2.7|\n|||||||||\n|TCTC w/o RP|50.01| 49.90| 23.14| 61.55 |35.7/3.1\n|TCTC w/o LE|49.14| 49.62| 22.18| 63.55 |37.1/3.5\n|||||||||\n|10% gloss corrupted| 52.98 |53.87 |28.89| 30.01| 15.9/5.0\n|20% gloss corrupted| 51.79 |52.99 |28.01| 39.91| 16.5/6.1\n|40% gloss corrupted|51.01| 49.92| 25.02| 59.53 |55.9/2.7\n|80% gloss corrupted|40.14| 40.62| 20.18| 87.66 |87.1/3.5\n|100% gloss corrupted|37.63| 39.01| 10.02| 100.00 |99.00/0.3\n\n\nWe appreciate your questions and comments. Please let us know for any further questions."}, {"review_id": "ZGTmsvBdsg", "reviewer": "Reviewer_rodL", "summary": "The paper introduces a new method named MixSignGraph for sign language recognition from videos. The method is an extension to SignGraph [4] that introduces local sign graph and temporal sign graph for sign language recognition. The proposed method adds an additional hierarchical sign graph to capture correlations between visual features. Furthermore, the paper introduces Text-based CTC Pre-training to further improve the performance.", "questions": "Please see weaknesses above regarding questions.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Positives:\n\n-\tThe paper shows that by adding the hierarchical sign graph, the proposed method can improve the sign language recognition performance.\n\n-\tThe paper has conducted extensive experiments on several datasets and alation studies to verify the proposed method.\n\nNegatives:\n\n-\tSince the method is an extension of the SignGraph [4] method, the novelty and technical contribution of the method is limited. The main difference is by introducing the hierarchical sign graph into the method.\n\n-\tAlthough the Text-based CTC Pre-training shows improvements, however, the model may overfit to these datasets by using the texts in the datasets for pre-training. As a result, the generalization ability of the method may be reduced.\n\n-\tIn tables 3, 4, 5, 6, 7, why not comparing to the original SignGraph [4] method?\n\n-\tModels such as the Swin transformer also utilize hierarchical representations. Will a Swin transformer perform as good as the hierarchical sign graph in the proposed method?", "was_revised": true, "final_justification": "The authors have addressed most of my concerns with the rebuttal.", "rebuttal": "We appreciate your thoughtful comments. Below we provide answers to the comments you raised.\n\n## Q1: The novelty and contribution are limited. The main difference is in introducing the HSG.\n\nWe would like to clarify that our work introduces two substantial and original contributions that go beyond a simple architectural refinement.\n\n(1) **Identifying and addressing a core limitation of SignGraph**: SignGraph builds LSG and TSG modules to model the intra-frame and inter-frame cross-regions features. While cross-region interactions have been shown to improve CSLR performance, we find that cross-region interactions may also weaken the feature representation and location sensitivity of a single/local region.\n\n- (a) The distinctiveness of a local region’s representation is disrupted and weakened by features from other connected regions. \n\n- (b) The spatial structure of regions (i.e., the location of a region, such as the hand area, within a frame) is also weakened by the interactions with other connected regions. \n\nTo alleviate the above issue, we propose the Hierarchical Sign Graph module, which aggregates the same-region features from different-granularity feature maps of a frame. By highlighting the features in a single region, which is located at a specific position within a frame, HSG is possible to enhance the distinctiveness and location sensitivity of a local region itself, even after cross-region feature extraction. This is not merely an incremental change, but a theoretical and technical rethinking of how region interactions should be modeled in sign sequences.\n\n\n(2) **Identifying and addressing a core limitation of Gloss-Free SLT (GFSLT)**:  Additionally, our approach extends beyond the task scope of SignGraph. While SignGraph focuses solely on CSLR tasks, our MixSignGraph can address both CSLR and SLT tasks(including Sign2Text, Sign2Gloss2Text, and GFSLT).\n\nMore importantly, in SLT tasks, we find that the current **bottleneck in Gloss-Free SLT** lies in the **continuous feature distribution** of SL videos, which does not align with the **discrete feature distribution** required by translation models, where the inputs are typically discrete tokens (see Figure 4 and Section 3.1). In contrast, gloss-based SLT benefits from gloss annotations with CTC constraints to effectively discretize SL features. Building on this insight, we propose a TCTC pretraining method, which leverages pseudo gloss sequences and CTC to guide the discretization of SL video features.\n\n\n## Q2: TCTC may overfit to these datasets, the generalization ability may be reduced?\n\n- It is worth noting that text annotations are essential for training an SLT model, whether gloss-based or gloss-free, and regardless of whether the TCTC module is adopted. Our method does not introduce additional text annotations for training (i.e., it uses the same video-text training samples as other gloss-based and gloss-free SLT models). Instead, we focus on improving how text is used during the training stage by generating pseudo glosses to pretrain the model.\n\n- We also highlight that the principle behind TCTC (as detailed in Section 3.1) is that pseudo glosses, generated from the text, are used to discretize continuous SL features, thereby providing suitable input for translation models, whose inputs are typically discrete tokens (see Figure 4 and Section 3.1). Importantly, **TCTC is not designed to fit the text in the datasets**. Instead, it serves as a form of pseudo-supervision for the visual encoder, guiding  it to tokenize sign language videos effectively. The goal is not to directly fit the model to the training text, but to provide discrete and meaningful representations for SLT.\n\n- The comparison results of dev/test set on four datasets (Single domain: CSL-daily, Phoenix14T. Open domain: How2sign, Openasl) demonstrate our TCTC's generalization. TCTC maintains good generalization ability across different domains, further proving its effectiveness. \n\n ## Q3: Comparing to the original SignGraph method?\n\nIn fact, we have compared our method with SignGraph. Specifically, the SignGraph paper proposed two models: SignGraph and the enhanced version MultiSignGraph. In the main paper, we have presented comparisons with MultiSignGraph, which achieves better performance than SignGraph, as shown in Tables 3, 4, 5, 6, and 7. To provide a more comprehensive comparison, we will add the comparison with SignGraph in the final version.\n \n\n\n\n## Q4: Comparing with Swin transformer\n\nWe have already compared Swin Transformer (SwinT) with our method in Appendix: Section A.7 and Table 14, which is in page 17 of our paper. We also list results here for clarity:\n\n|Backbone|WER|Del/Ins|\n|-|-|-|\n|SwinT |45.4| 16.3/1.3\n|PyVig|35.4|12.1/1.3\n|SelfAttention|39.2|15.3/0.9 \n|ours|16.7|5.1/2.0\n\nAs shown in the table, the Swin Transformer and PyVig, which also utilize hierarchical structures, perform worse than our method. This highlights that MixSignGraph with HSG can effectively enhance local region representations following the extraction of cross-region features, addressing the limitations of SignGraph and further improving SLT performance. \n\nWe appreciate your questions and comments. Please let us know for any further questions."}, {"review_id": "XlNQzQwBXP", "reviewer": "Reviewer_yMHm", "summary": "The paper proposes MixSignGraph, a novel GCN-based framework for continuous sign language recognition (CSLR) and translation (SLT). The introduced novelty is mainly two-fold: i) MixSignGraph introduces a new Hierarchical Sign Graph (HSG) module that combines features at different levels of granularity, allowing the model to focus on both global and local shapes at the same time; and ii) it introduces Text-Based CTC Pre-training (TCTC) for gloss-free SLT tasks, where pseudo gloss sequences are built from translations by applying traditional NLP techniques like stemming and lemmatization.", "questions": "Please address the weaknesses I mentioned above, in particular those related with the scaling of the method to long sequences and the evaluation metrics. If these are addressed appropriately and a writing review is performed to clear the typos, I am willing to raise my score.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The HSG design is intuitive and effectively addresses a clear limitation of prior work.\n- The TCTC pre-training is an interesting practical workaround for gloss-free settings.\n- The experimental results consistently show improvements across CSLR and SLT tasks, with a performance competitive or superior to the state of the art.\n- The proposed approach is modality efficient, as it achieves high performance only from RGB data.\n\nWeaknesses:\n- Although the HSG design is intuitive, it lacks formal justification or theoretical analysis of why it improves representation.\n- The use of GCNs and multiple graph modules (LSG, TSG, HSG) might raise scalability issues on long video sequences. This issue is not analyzed nor is it mentioned in the limitations section.\n- Only lexical overlap metrics (ROUGE and BLEU) are used for evaluation. This is especially problematic in SLT, where there’s often no one-to-one correspondence between signs and words, and translations tend to be semantically but not lexically equivalent.\n- The writing has many typos (e.g. \"Limitations and *Borader* Impacts\"), which is not acceptable. These sometimes make the idea hard to follow. The authors should definitely fix these.", "was_revised": true, "final_justification": "Having read the rebuttal and the other reviews, I still lean towards acceptance. However, I wouldn't disagree if the decision went the other way, as this would enable you to make all the necessary changes to your manuscript and submit an improved version to another venue. Thus, I will keep my score (4).", "rebuttal": "We appreciate your insightful comments. Below, we provide individual responses to your questions.\n\n## Q1: Lack formal justification or theoretical analysis\n\nThank you for the comment. Here we try to provide a brief formal analysis for our HSG in MixSignGraph and discuss why HSG can boost discriminative local features.\n\n- For each frame $f_i\\in \\mathbb{R}^{[H\\times W \\times 3]}$ with height $H$ and width $W$, we get a number of patches (or nodes) $v _ i= \\\\{v _ {ij} \\\\} _ {j=1}^{N}$ with its feature $\\mu _ {ij}$.\n\n- Local Sign Graph (LSG) can be simplified as: $\\mu _ i =  \\mathcal{GCN}_L(\\mu _ i, e _ i^l) $ where $e _ i^l = \\{ e(v _ {ij}, v _ {ik}) \\mid j \\in [0, N), v _ {ik} \\in \\mathcal{N} _ {\\mathsf{K} _ {l}}(v _ {ij}) \\}$ and $\\mathcal{N} _ {\\mathsf{K} _ {l}}(v _ {ij}) =\\{v _ {ik} | v _ {ik} \\in  TopK (\\{\\mathcal{DIS}(v _ {ij}, v _ {ik})\\} _ {k=1}^{N}) \\}$. LSG extracts intra-frame cross-region features by aggregating neighbor $\\mathcal{N} _ {\\mathsf{K} _ {l}}(v _ {ij}) $ features within frames based on node similarity.\n\n- Temporal Sign Graph (TSG) can be simplified as: $\\\\{\\mu _ i\\\\}^{\\theta}_{i=1}= \\mathcal{GCN}_t\\left ( \\\\{\\mu _ i\\\\}^{\\theta} _ {i=1},  \\\\{e _ i^t\\\\}^{\\theta-1} _ {i=1} \\right) $ where $e^t _ {i}=\\\\{e(v _ {ij},v _ {(i+1)k}) |  \\mathcal{DIS}(v _ {ij}, v _ {(i+1)k} ) \\in TopK(M)\\\\}$, $M$ denotes similarity matrix. TSG extracts inter-frame cross-region features by aggregating neighbor features between frames based on node similarity. \n\n- Suppuse orginal node $\\mu_{ij}$ follows a Gaussian distribution:\n$\\mathcal{G}(\\chi _ {ij}^{(n)}, \\sigma _ {ij}^{(n)}), n \\in [1, N]$. Then $\\mu _ {ij}^{new} \\to \\mathcal{G}(\\bar{\\chi}, \\bar{\\sigma})$ , where $\\bar{\\chi}$, $\\bar{\\sigma}$ represents the weighted average of $\\mathcal{N}(\\chi _ {ij}^{(n)})$, $\\mathcal{N}(\\sigma _ {ij}^{(n)})$. $\\mathcal{N}$ denotes its neighbors. The difference between two neighbor nodes cloud be: $\\mathbb{E}[||\\mu _ {ij}^{new} -\\mu _ {ik}^{new} ||^2] \\to 0$\n\nAlthough, the LSG and TSG modules facilitate the extraction of cross-region features. However, the nodes within neighboring regions may become overly similar, especially when there are too many edges or the GCN layers are too deep. This leads to the over-smoothing phenomenon [1,2], which weakens the distinctiveness and location sensitivity of a local region of $v _ {i,j}$.\n\n\n\n- While HSG, in constract: $\\mu _ {i}^{h'}, \\mu _ {i}^{l'}= \\mathcal{GCN} _ H(\\{\\mu _ {i}^{h}, \\mu _ i^l\\}, e _ b^i)$ where $e _ i^b = \\\\{ e(v _ {ij}^h, v _ {ik}^l)|j \\in  [0,N^h), \nk = ( \\begin{matrix}\\left \\lfloor \\frac{\\left \\lfloor \\frac{j}{W^h} \\right \\rfloor}{s} \\right \\rfloor \\end{matrix}  \\times  W^l + \\left \\lfloor \\frac{j\\%W^h}{s} \\right \\rfloor )  \\\\}$, and $\\mu _ i^h$ refers to the feature map before cross-region extraction, which well preserves the integrity of local features and structure. \nHSG aggregates features in the same regions of $\\mu _ i^h$ and $\\mu _ i^l$. Therefore, the local information of nodes can be effectively enhanced, and then the over-smoothing phenomenon is alleviated.\nThat is to say, the hierarchical interaction between same-region features in different granularities ensures that both distinctiveness and location of local features are maintained, improving the model’s overall performance.\n\nWe will provide a more detailed analysis and add it in our paper.\n\n[1] Wu, Xinyi, et al. \"Demystifying oversmoothing in attention-based graph neural networks.\" Advances in Neural Information Processing Systems 36 (2023): 35084-35106.\n\n[2]Wu, Xinyi, et al. \"A Non-Asymptotic Analysis of Oversmoothing in Graph Neural Networks.\" The Eleventh International Conference on Learning Representations.\n\n\n## Q2: Scalability Issues with Graph Modules\n\nThank you for raising the concern about the potential scalability issues with LSG, TSG, HSG on long video sequences. We appreciate your insight, and we would like to clarify the following points:\n\n- LSG and HSG are built based on  single frames, which are not limited by the video length. These modules focus on aggregating features at the frame level; thus, they do not introduce scalability issues for long video sequences.\n\n- TSG is designed to connect **adjacent frames** in the sequence. While MixSignGraph currently builds the TSG on the entire video, this may pose challenges for modeling long videos. To address this, we have conducted an ablation study to explore the effect of connecting different numbers of frames in TSG (i.e., connecting every N frames temporally in TSG instead of collecting all frames).  The model's performance remains relatively stable regardless of how many frames are connected in TSG.\n\nTherefore, for long videos, instead of constructing a graph over the entire video, we can group the video into smaller temporal segments, and this grouping does not affect the model’s performance, confirming that scalability is not an issue for long video sequences.\n\n| Backbone | WER  | Del/Ins |\n|----------|------|---------|\n| Whole    | 16.73 | 5.1/2.0 |\n| N=10       | 16.72 | 5.0/1.5 |\n| N=50       | 16.75 | 5.3/2.2 |\n| N=100      | 16.72 | 5.2/1.3 |\n\nWe will add this analysis in the final version of our paper. \n\n\n\n ## Q3: Evaluation metrics\n\nThank you for the constructive comment regarding evaluation metrics. \n\n- **Adoption of ROUGE and BLEU**: We mainly followed previous SLT work and adopted the widely used metrics ROUGE and BLEU, to maintain a fair comparison with state-of-the-art models.\n\n- **Semantic Evaluation**: We acknowledge that the above metrics may not fully capture the semantic meaning of translations.  As you pointed out, ROUGE and BLEU are based on lexical overlap, which may fail to evaluate the **semantic quality** of translations. For example, given a **ground truth** (GT: \"I like dog.\") and two possible candidates (c1: \"I love dog.\" and c2: \"I hate dog.\"), Both predictions may have the same **ROUGE** and **BLEU** scores (Rouge(c1, GT) $\\approx $ 66.6), Rouge(c2, GT) $\\approx $ 66.6), even though \"I love dog\" is clearly more semantically accurate.\n\nTo address this issue, we follow the approach proposed in [3] and additionally adopt **BLEURT**, **CIDEr**, and **LLM** scores to better capture the semantic quality of our model’s outputs.\n\n- **BLEURT[1]**: A trained metric that uses a regression model, which is trained on human ratings data. It captures non-trivial **semantic similarities** between sentences.   \n\n- **CIDEr[2]**: A captioning metric that computes the **consensus** of the prediction compared to the ground truth by calculating the weighted cosine similarity of TF-IDF scores for various n-grams.\n- **LLM[3]**: An LLM-based evaluation metric proposed by [3]. We use the publicly available **GPT-4o-mini API** from OpenAI, prompting the model to generate a score from **0 to 5** for each translation pair to measure the matching degree between predicted sequence and ground-truth sequence, where 5 indicates the best match and 0 indicates  the worst. \n\nThe results for these additional evaluation metrics are shown below:\n\n| Dataset    | BLEURT $\\uparrow$ | CIDEr  $\\uparrow$ | LLM  $\\uparrow$ |\n|------------|--------|-------|------|\n| Phoenix14T(Ours) | 59.68  | 240.23| 3.1  |\n| CSL-daily(Ours)  | 57.47  | 233.23| 3.0  |\n|how2sign(Ours)|50.1 | 120.4| 1.9|\n|how2sign(Lost[3])| 45.3|  100.8 | 1.59|  \n\nWe will provide a more comprehensive assessment of our model's semantic performance. More detailed results will be included in the final version of the paper.\n\n[1] Sellam, Thibault, Dipanjan Das, and Ankur Parikh. \"BLEURT: Learning Robust Metrics for Text Generation.\" Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.\n\n[2] Vedantam, Ramakrishna, C. Lawrence Zitnick, and Devi Parikh. \"Cider: Consensus-based image description evaluation.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.\n\n[3] Jang, Youngjoon, et al. \"Lost in translation, found in context: Sign language translation with contextual cues.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n\n## Q4: Typos\n\nThank you for pointing this out!  We will thoroughly review the manuscript and correct all typos (e.g., \"Borader\" to \"Broader\"),  ensuring the ideas are conveyed more clearly and professionally in the final version. We appreciate your patience and feedback.\n\nWe appreciate your questions and comments. Please let us know for any further questions."}, {"review_id": "iXmruwuFfK", "reviewer": "Reviewer_zjcq", "summary": "This paper introduces MixSignGraph, which represents a sign sequence as mixed graphs of nodes. Compared to the previous work, termed as MultiSignGraph, it makes two key changes. The first is a Hierarchical Sign Graph module, and the second is Text-based CTC Pre-training strategy.", "questions": "See above", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros:\n1. The paper is written in a clear and concise manner, making it easy to follow the proposed methodology, experimental setup, and results. The logical organization enhances readability and ensures that the contributions are effectively communicated to the reader.\n2. The experimental evaluation demonstrates strong performance across multiple benchmarks, suggesting that the proposed method is effective and competitive with existing approaches. These results support the claim that the method offers practical benefits in real-world applications.\n3. The proposed approach introduces several novel components that distinguish it from previously published methods. These innovations contribute meaningfully to the field and offer new directions for future research in multi-modal or hierarchical learning frameworks.\n\nCons:\n1. While the paper employs different backbone architectures compared to baseline methods, it does not provide any discussion or comparison regarding computational costs—such as training time, inference speed, or model complexity. Including such an analysis would help readers better understand the trade-offs between performance gains and resource requirements.\n2. In Table 1, the meanings of the labels \"1\" and \"2\" are not explained. This lack of clarification makes it difficult for readers to interpret the experimental variations being compared. A brief explanation or footnote in the table would significantly improve its clarity and usefulness.\n\nMinor questions:\n1. It would be helpful to clarify how the first Hierarchical Sign Graph module behaves when the input feature vector $u_l$  is absent. Does the module skip processing at that level, or is there a default mechanism in place? Understanding this edge case would improve reproducibility and robustness assessment.\n2. In Table 2, the abbreviation \"TCP\" appears without prior definition. Please clarify what \"TCP\" stands for to avoid confusion and ensure accurate interpretation of the experimental settings.\n3. If certain datasets include gloss annotations, would the authors still employ the proposed Text-based CTC Pre-training strategy? It would be interesting to know whether the presence of such annotations influences the choice of pre-training method, and if so, how.", "was_revised": true, "final_justification": "Considering the rebuttal, i think this paper is novel with sufficient experiemental results. I will maintain my positive rating and lean towards accepting this paper.", "rebuttal": "We appreciate your insightful comments, which have helped strengthen our work. Below, we provide individual responses to your questions.\n\n## Q1: Computational costs\n\nWe actually did analyze the computational complexity of our proposed MixSigngraph in Appendix: Table 10 and Section A.5 (on page 15). Besides, we also provide training time and inference speed in the following table, where the metric was measured with a 3090 GPU on 280 frames. \n\nCompared with SIGN2GPT, TwoStream, our model achieves faster inference speed (translates about 2 SL video sequence per second), while achieving the promising performance on multiple SL tasks, as shown in the table below and Tabel 10 on page 15.\n\n\n|model|backbone|Inference Speed|Traning time|\n|-|-|-|-| \nSIGN2GPT| Dino-V2 ViT|1 seq/s|-\nTwoStream |Dual S3D|1.4 seq/s|-\nSignGraph |GCN|2.1 seq/s| 44 min/epoch\nMixSigngraph |GCN|2.0 seq/s| 45min/epoch\n\n## Q2: Unclear footnote  in Table 1,2\n\nThank you for pointing this out! We apologize for the unclear footnote in our Table. $LSG_1$, $TSG_1$ and $HSG_1$ means the first $LSG$, $TSG$, $HSG$ module, and $LSG_2$, $TSG_2$ and $HSG_2$ means the second $LSG$, $TSG$, $HSG$ module.  We will thoroughly review the manuscript and correct it, ensuring the ideas are conveyed more clearly and professionally in the final version. We appreciate your patience and feedback.\n\n\n## Q3: How does the first Hierarchical Sign Graph module behave when the input feature vector is absent?\n\nAs shown in Figure 2, Patchify Steam consists of convolution layers. Before downsampling to get $u_0^l$, we first generate  $u_0^h$, which is one of the initial input features $u_i^h$(i=0) of the first HSG.  As for $u_0^l$, after being processed by the LSG and TSG, it will be fed into the HSG as another inital input of first HSG.\nWe will add a more detailed description, ensuring the ideas are conveyed more clearly.\n\n## Q4: \"TCP\" appears without prior definition. \n\nWe apologize for the misspelling. 'TCP' should be written as the \"TCTC\" module. We will correct this error in the final version.\n\n\n## Q5: Use of Text-based CTC Pre-training with Datasets Containing Gloss Annotations\n\nWhen gloss annotations are available (e.g., dataset:CSL-daily and Phoenix14T), we evaluate our model on two different tasks: gloss-based SLT and gloss-free SLT.\n\n- **In the gloss-based SLT task**, TCTC is not employed. Instead, the ground truth (GT) gloss sequence is used to pretrain our backbone model.\n\n- **In the gloss-free SLT task**, gloss annotations are **not** used for training. Instead, **TCTC** is employed to generate pseudo glosses for pretraining the backbone model.\n\n\nIn Tables 5 and 6, we present both the gloss-based SLT performance (Sign2Text) and the gloss-free SLT performance (Gloss-free SLT) for CSL-daily and Phoenix14T. Generally, the gloss-based SLT models perform better than the gloss-free SLT models on both datasets. \n\n\nIn real-world scenarios, if gloss annotations are available, it is preferable to choose the ground truth gloss to pretrain the model in order to achieve better performance. We hope this clarifies how the presence of gloss annotations influences our choice of pre-training method. \n\nWe appreciate your questions and comments. Please let us know for any further questions."}]

Author rebuttal:

Meta-review: This paper proposes MixSignGraph, an approach for sign language recognition and translation. The key contribution is a hierarchical sign graph that integrates features at multiple levels of granularity, enabling the model to capture both local and global structures. Initially, the paper received one Accept, one Borderline Accept, and two Borderline Reject ratings. The reviewers acknowledged the presence of several novel components in the architecture that set the work apart from prior approaches. Some reviewers also praised the extensive evaluation, although others felt it was insufficient and suggested expanding the experiments to include additional baselines and metrics. In their rebuttal, the authors provided clarifications and additional results, which slightly shifted the picture to one Reject, one Accept, and two Borderline Accept ratings. While the reviews remain divided, the paper received sufficient support from the reviewers, and the AC concludes that there is no strong basis for rejection. The authors are encouraged to carefully address any remaining reviewer feedback when preparing the final version. In particular, they should expand and refine the quantitative evaluation, improve clarity around the metrics, and incorporate any additional promised results, analyses, or discussions from the rebuttal.
Decision: Accept (poster)