Title: MixSignGraph: A Sign Sequence is Worth Mixed Graphs of Nodes
Venue: NeurIPS 2025
Accepted: yes
Keywords: Sign Language Recognition and Translation

Graph convolutional network
Abstract: Recent advances in sign language research have benefited from CNN-based backbones, which are primarily transferred from traditional computer vision tasks (\eg object detection, image recognition). However, these CNN-based backbones usually excel at extracting features like contours and texture, but may struggle with capturing sign-related features. 
To capture such sign-related features, SignGraph model extracts the cross-region sign features by building the Local Sign Graph (LSG) module and the Temporal Sign Graph (TSG) module. However, we emphasize that although capturing cross-region dependencies can improve sign language performance, it may degrade the representation quality of local regions.
To mitigate this, we introduce MixSignGraph, which represents sign sequences as a group of mixed graphs for feature extraction. 
Specifically, besides the LSG module and TSG module that model the intra-frame and inter-frame cross-regions features, we design a simple yet effective Hierarchical Sign Graph (HSG) module, which enhances local region representations following the extraction of cross-region features, by aggregating the same-region features from different-granularity feature maps of a frame, \ie to boost discriminative local features.
In addition, to further improve the performance of gloss-free sign language task, we propose a simple yet counter-intuitive Text-based CTC Pre-training (TCTC) method, which generates pseudo gloss labels from text sequences for model pre-training. Extensive experiments conducted on the current five sign language datasets demonstrate that MixSignGraph surpasses the most current models on multiple sign language tasks across several datasets, without relying on any additional cues. Code and models are available at: \href{https://github.com/gswycf/SignLanguage}{\textcolor{blue}{https://github.com/gswycf/SignLanguage}}.

MixSignGraph: A Sign Sequence is Worth Mixed Graphs of Nodes

Abstract

Recent advances in sign language research have benefited from CNN-based backbones, which are primarily transferred from traditional computer vision tasks ( e.g. , object detection, image recognition). However, these CNN-based backbones usually excel at extracting features like contours and texture, but may struggle with capturing sign-related features. To capture such sign-related features, SignGraph model extracts the cross-region sign features by building the Local Sign Graph (LSG) module and the Temporal Sign Graph (TSG) module. However, we emphasize that although capturing cross-region dependencies can improve sign language performance, it may degrade the representation quality of local regions. To mitigate this, we introduce MixSignGraph, which represents sign sequences as a group of mixed graphs for feature extraction. Specifically, besides the LSG module and TSG module that model the intra-frame and inter-frame cross-regions features, we design a simple yet effective Hierarchical Sign Graph (HSG) module, which enhances local region representations following the extraction of cross-region features, by aggregating the same-region features from different-granularity feature maps of a frame, i.e. , to boost discriminative local features. In addition, to further improve the performance of gloss-free sign language task, we propose a simple yet counter-intuitive Text-based CTC Pre-training (TCTC) method, which generates pseudo gloss labels from text sequences for model pre-training. Extensive experiments conducted on the current five sign language datasets demonstrate that MixSignGraph surpasses the most current models on multiple sign language tasks across several datasets, without relying on any additional cues. Code and models are available at: https://github.com/gswycf/SignLanguage.

1 Introduction

Advancements in computer vision (CV) and natural language processing (NLP) technologies have significantly facilitated the development of sign language (SL) research, including Sign Language Recognition (SLR) and Sign Language Translation (SLT). The SLR task encompasses Isolated SLR (ISLR) (1) and Continuous SLR (CSLR) (2; 3; 4), aiming to recognize isolated or continuous signs as corresponding glosses or gloss sequences. While the SLT task focuses on translating continuous signs into spoken language (5). In the current SL framework, the standard approach first uses 2D/3D CNN-based backbones to extract visual features (6; 7; 8), then employs temporal modules (5; 9; 10) to capture dynamic changes in sign frames, and finally adopts a Connectionist Temporal Classification (CTC) decoder to obtain the gloss sequence or a translation model to generate the spoken sentence.

Unlike other CV tasks where contour and texture representations are crucial, SL tasks need to focus on both manual and non-manual features, particularly the collaboration of these cues in different

regions (11; 12), while the traditional CNNs may fail to capture the collaboration of signs across different regions, thereby limiting their ability to extract effective sign features. To enhance the CNN-based backbone's ability to extract sign-related features, existing work has focused on designing various CNN-based backbones by incorporating domain knowledge, such as skeleton information (9), depth images (13), or local areas (10). Additionally, some studies have adopted different training strategies e.g. , back translation (14), knowledge distillation (15) and contrastive learning (6). Besides, in SLT tasks, many current SOTA models also rely on gloss annotations to pre-train their backbones with CTC loss, aiming to improve the SLT performance. When gloss annotations are missing, i.e. , in Gloss-Free SLT (GFSLT) tasks, how to improve the SLT performance is still a challenging task.

To address the limitation of CNN-based backbones, SignGraph (4) transforms SL frames into graph nodes and proposes the Local Sign Graph (LSG) and Temporal Sign Graph (TSG) modules to model spatial and temporal cross-region dependencies. However, such strategies may inadvertently degrade fine-grained local features due to the emphasis on cross-region aggregation. Regarding improving the gloss-free SLT performance, the existing models have attempted to introduce attention mechanisms (16), contrastive language-image pre-training (17) or vector quantization (18).

Building upon prior work, we propose a simple yet effective sign graph neural network, MixSignGraph, for SL tasks (CSLR and SLT). In addition to extracting crossregion features, MixSignGraph introduces a Hierarchical Sign Graph (HSG) module to enhance local representations from feature maps of different granularities, mitigating the degradation of local features that may occur during cross-region modeling in SignGraph. The main intuitions are: Crossregion feature extraction may lead to the local representation of the current node being diluted by features from other regions, resulting in the degradation of local information. Moreover, the content within a single region can be represented at different levels of granularity through hierarchical feature maps generated by down-sampling. Therefore, we propose a Hierarchical Sign Graph (HSG) module, which connects corresponding regions across multi-granularity feature maps. This hierarchical design allows the model to reinforce local representations by

aggregating complementary information from different granularities, thereby mitigating the degradation caused by cross-region feature fusion. Besides, to further improve the performance of sign language tasks without gloss annotations, we propose a simple yet counter-intuitive Text-based CTC Pre-training (TCTC) method, which generates pseudo gloss labels from text labels for model pre-training. We make the following contributions:

To address the limitations of SignGraph (4), we propose a simple yet effective sign graph neural network, called MixSignGraph. In addition to extracting cross-region features, MixSignGraph introduces a Hierarchical Sign Graph (HSG) module, which enables bidirectional exchanges of feature maps with different granularities, to enhance local representations from hierarchical feature maps. The proposed HSG can mitigate the degradation of local features that may occur during cross-region modeling.

A simple, effective, yet counter-intuitive training method named Text-based CTC Pretraining (TCTC) is proposed for the GFSLT task, which generates pseudo gloss annotations based on text labels. The proposed TCTC effectively improves the performance of GFSLT and narrows the gap with gloss-based SLT models.

The extensive experiments on common SL tasks (including CSLR and SLT) over five public datasets demonstrate the superiority of the proposed MixSignGraph, which shows promising results and does not use any extra cues.

Local Graph

Building

Temporal Graph

Building

Hierarchical Graph

Building

Projection

Layer

Down

sampling

Patch Merging

Patchify Stem

N×

Global Feature Module

CTC

Translation

Model

Glosses

(CSLR)

Sentence

(SLT)

,

(

,

)

: Input (output) low, high resolution feature map of the

i

th round

Pretraining

Element sum operation

LSG

TSG

HSG

𝜇

!

"

𝜇

!

#

𝜇

$%&

#

𝜇

$%&

"

𝜇

$

#

𝜇

"

𝜇

#

𝜇

$%&

"

𝜇

$%&

#

𝜇

$%&

#

$

$

2 Preliminaries

For a SL sequence f = { f i } θ i =1 with θ frames, the target of CSLR task is to get a recognized gloss sequence g = { g i } ϑ i =1 with ϑ glosses, while the target of SLT task is to generate a spoken language sentence t = { t i } ς i =1 with ς words based on input f . SignGraph (4) introduces the following modules to model the SL sequence f :

Patchify Stem. To apply graph convolutional networks (GCNs) directly to SL frames instead of skeleton data, a patchify stem is employed to divide each RGB frame with height H , width W into N = HW/P 2 patches (or nodes) v i = { v ij } N j =1 , with the corresponding feature embedding µ ij ∈ R D . Here, v ij represents the j -th patch in the i -th frame f i , D denotes the feature dimension, and P is the patch size.

Local Sign Graph Learning. To capture correlations between different spatial regions, such as the face and hands, SignGraph (4) introduces a Local Sign Graph (LSG) module to extract intra-frame cross-region features. Specifically, it computes the feature distances between patches within each frame using a distance function DIS (19). Based on these distances, a K-nearest neighbor (KNN)based graph construction strategy is used to obtain local sign graph, and graph convolution is applied to integrate features across nodes, enabling effective modeling of intra-frame spatial dependencies.

Temporal Sign Graph Learning. To capture the movements of the body, hands, and face, SignGraph (4) introduces a Temporal Sign Graph (TSG) module that dynamically establishes connections between regions across consecutive frames, enabling the learning of inter-frame cross-region features. Similarly, it calculates feature distances between patches in adjacent frames and adopts a KNN-based graph construction strategy to effectively model cross-region features between frames.

Patch Merging. Considering that patches with a fixed window may not effectively capture sign features ( e.g. , hand regions may be split by different patches), SignGraph (4) adopts the patch merging module to downsample feature maps by a factor of 2, to generate larger-size patches.

3 Method

Overall Framework. As shown in Figure 2, when given the video frames, a patchify stem is adopted to convert each frame into a set of patches ( i.e. , nodes) v i = { v ij } N j =1 . Then, the three key modules are utilized to construct graphs and capture sign-related features as follows. First, by adopting the LSG and TSG modules, MixSignGraph can learn both the correlation of cross-region features within one frame and the interaction of cross-region features among adjacent frames. Second, by aggregating the same-region features from different feature maps of a frame, we build a hierarchical sign graph G h i , to bidirectionally exchange features and mine different-granularity one-region features. After that, we adopt a global feature module (2; 6) to learn global changes of whole frames. To tackle CSLR tasks, a classifier and a widely-used CTC loss (20) are adopted to predict the probability p ( g | f ) of the target gloss sequence. In SLT tasks, we adopt a translation model to convert feature sequences or gloss sequences into text sequences. To further improve GFSLT performances, we propose the Text-based CTC Pre-training (TCTC) mechanism, which generates pseudo gloss labels from text sequences for model pre-training.

Hierarchical Sign Graph Learning. Despite the benefits of capturing cross-region features for improving SL performance, it may degrade the representation quality of local regions. Besides, traditional CNNs can only downsample one region from coarse to fine representations, lacking the ability to facilitate interaction between different-granularity feature maps. To address this issue, we introduce the HSG module, which enhances local information from feature maps of different granularities after cross-region feature extraction. In this way, we can combine hierarchical features of the same region and make the

One

frame

Hierarchical graph of the same regions

across feature maps

model focus on both global shapes and detailed local features of a region at the same time. As shown in Figure 3, given two feature maps µ h i ∈ R [ H h × W h × C h ] in high resolution and µ l i ∈ R [ H l × W l × C l ] in low resolution for frame f i , we have N h = H h W h nodes v h i for µ h i and N l = H l W l nodes v l i for µ l i . Here, H h , W h , C h , H l , W l and C l are the height, width and the number of channels of µ h and µ l respectively, and H h H l = W h W l = s > 0 , where s is the downsampling factor. As shown in Figure 2 and 3, to construct the hierarchical sign graph, we first use a projection layer with weights Θ h 1 to map µ h i ∈ R [ H h × W h × C h ] to µ h ′ i = µ h i Θ h 1 , ensuring the same dimensionality as µ l i , where µ h ′ i ∈ R [ H h × W h × C l ] . Then, we add an undirected edge e ( v h ij , v l ik ) , if the node v h ij in µ h ′ i and the node v l ik in µ l i correspond to the one region.

In this way, we can get the edge set e b i between µ l i and µ h i for the frame f i , and obtain the hierarchical sign graph G H i = { ( v h ′ i , v l i ) , e b i } . After that, we apply a graph convolutional layer GCN H to enable bidirectional exchanges of the same regions across different-granularity feature maps to perform feature fusion.

Unlike LSG and TSG, which dynamically build edges based on the KNN algorithm to capture cross-region features, the HSG focuses on fusing features belonging to the same region at different granularities, aiming to enhance local features of one region.

Mixed Sign Graph Learning. Based on the previous work SignGraph, we insert the proposed HSG module after the LSG and TSG modules to enhance local information from feature maps of different granularities. For convenience, we represent the i th LSG , TSG and HSG module as LSG i , TSG i and HSG i , respectively.

3.1 Text-based CTC Pre-training

Gloss-based Tasks. For CSLR tasks, CTC loss with gloss annotations has become the de-facto loss function in current CSLR models. While in gloss-based SLT tasks, the current SOTA SLT models (22; 7) rely on pre-training, in which CTC loss with gloss sequences is used to optimize the backbone to extract effective video-level features, as described in Equation 3. After the pre-training, these SLT models further introduce a pre-trained translation model ( e.g. , mBART (23)) and fine-tune the translation model for SLT, as defined in Equation 4. Here, L CTC denotes CTC loss function, L CE denotes cross-entropy loss function, ˆ g and g respectively denote recognized gloss sequence and labeled gloss sequence, ˆ t and t respectively denote translated sentence and labeled sentence, Θ R and Θ T denote the parameters of recognition and translation models.

Gloss-free Tasks. The above pre-training process enables the backbone to learn segmentation and semantic information from gloss annotations, which are crucial for improving the SLT performance.

75

40

50

20

25

0

0

25

20

50

75

50

40

30

20

10

0

10

20

60

40

20

0

20

40

60

Input Features

Input Feature

100

60

40

10

300

400

50

20

200

0

20

0

0

100

0

100

40

60

80

.

.

10

-5

300

200

100

·

·

100

100

50

25

0

25

50

20

0

20

400

200

0

200

400

Gloss Embedding

Text Embedding

I3DFeature

Text Embeding

(a) Visualization of feature distribution of one test sample in PHOENIX14T dataset.

(b) Visualization of feature distribution of one test sample in the How2Sign dataset.

However, in the other SLT tasks, i.e. , gloss-free SLT tasks, there are no gloss annotations, since annotating glosses is a labor-intensive task.

Text-based CTC Pre-training (TCTC) : To improve the performance of these gloss-free SLT tasks, we propose a simple, effective, yet counter-intuitive training method called Text-based CTC Pretraining (TCTC), which generates pseudo gloss sequences from labeled sentences for pre-training. Specifically, in TCTC, we obtain the pseudo gloss sequence t ′ by simply pre-processing labeled sentence t , including removing punctuations, lemmatization and word-level tokenization. Then, we directly use the CTC loss with pseudo gloss sequence t ′ to optimize the backbone during initial pre-training, and then fine-tune the entire model for translation, as depicted in Equation 5, 6.

The principle of TCTC : Typically, CTC loss requires that the source sequence and the target sequence are aligned in the same order. Thus, CTC loss is often used in the CSLR task, where the sign video and gloss sequence have the same order. However, in SLT tasks, the alignment between the SL video and the text sequence is usually non-monotonic, due to the difference in grammatical rules between SL and spoken language. To tackle the challenge of GFSLT task, TCTC generates pseudo gloss sequences from labeled sentences and then adopts CTC loss for model pre-training. Although the generated pseudo gloss sequences from text/sentences may be different from the ground-truth gloss sequences, our TCTC is still crucial for improving SLT performance. The two possible reasons can be concluded as follows. First, given the discrete nature of languages, the inputs and outputs of translation models ( e.g. , mBART used in SLT) are in a categorical, discrete-valued space ( i.e. , typically as token embeddings (24)). Second, without CTC pre-training, the features of adjacent frames/clips (extracted by the backbone) tend to remain in a continuous-valued space, while CTC pre-training with gloss sequences guides the model to segment video features into discrete units. Therefore, by using CTC with pseudo gloss sequence t ′ to pre-train the backbone, TCTC provides a weak tokenization function, even if it may result in inaccurate segmentation.

To verify the effectiveness of TCTC, we visualize the feature distributions input to the translation model, i.e. , the features output from the CSLR model, while using or NOT using TCTC. As shown in the bottom left part of Figure 4, the embedding distributions of gloss sequence ( i.e. , red points) and words in spoken sentence ( i.e. , black points) are normally in a discrete-valued space, as described in the first reason of the previous paragraph. When moving to the top left part of Figure 4, the features extracted by NOT using TCTC are located at close positions ( e.g. , blue points on a continuous curve), while the features extracted by using TCTC scatter at different positions ( i.e. , green points in a discrete-valued space), as described in the second reason of the previous paragraph. It demonstrates

that introducing CTC loss with pseudo gloss annotations can guide the model to extract discrete and tokenized video features. From the perspective of feature distribution space, the pattern of feature distribution by using TCTC is similar to that of word embedding in spoken sentences, i.e. , in a discrete-valued space. It is worth mentioning that the TCTC mechanism not only makes sense for our proposed MixSignGraph, but also works for other backbones. To verify this phenomenon, we also visualize the feature distribution of the other pre-trained backbone I3D in right part of Figure 4, which is adopted for How2Sign and OpenASL datasets (see Section 4 for details). The above phenomenon proves that TCTC can provide a weak tokenization function, even if TCTC may bring inaccurate segmentation.

4 Experiments

Datasets. Our experiment evaluation is conducted on five publicly available SL datasets, including PHOENIX14 (25), PHOENIX14T (5), CSL-Daily (14), How2sign (26) and OpenASL (27). For more detailed descriptions of these datasets, please refer to Appendix A.2.

Architecture Setting. Our architecture is implemented using PyTorch 1.11. The setup includes the following components: (1) Patchify stem and Patch merging : We use a patchify stem and patch merging module in SignGraph to obtain patch (node) embeddings. (2) Distance function : In the baseline setting, we measure the distance between two nodes using the Euclidean distance. (3) Global feature module : This module comprises two 1D convolution blocks, a 2layer BiLSTM with a hidden size of 1024 for global feature modeling, and a fully connected layer for the final prediction. (4) Translation network : For a fair comparison, following the current SOTA SLT model (7), we adopt the pretrained mBART model provided by Huggingface as our translation network.

LSG

1

TSG

1

HSG

1

LSG

2

TSG

2

HSG

2

Dev

Test

WER Del/Ins WER Del/ins

%

%

%

%

%

%

22.3

8.4/2.5

22.2

8.1/2.7

!

19.2

5.6/2.2

21.0

4.8/2.3

!

19.6

5.6/2.1

21.5

5.1/2.5

!

19.9

6.7/2.1

21.6

6.2/3.1

!

19.3

5.3/2.1

20.8

5.8/2.0

!

19.5

6.6/1.7

21.2

5.4/2.4

!

19.4

6.2/2.9

21.5

5.9/3.6

!

!

18.7

5.1/2.3

20.6

5.2/1.7

!

!

18.6

4.3/1.8

20.2

5.5/1.7

!

!

17.8

6.1/2.3

19.6

5.3/2.9

!

!

!

17.1

5.3/2.0

19.7

5.0/3.0

!

!

!

17.4

5.7/2.1

19.6

5.1/2.9

!

!

!

!

!

!

16.7

4.9/2.1

19.0

4.9/2.9

Training Setting. For fair comparisons, we adopt the same data preprocessing and training settings as in previous work (2; 4). In regard to How2Sign and OpenASL datasets, we adopt the visual features generated by a pre-trained I3D model (8) . In addition, to ensure the model can be trained end-to-end on three 3090 GPUs with 24GB memory, the entire model is trained in half-precision.

Evaluation. We evaluate the performances of MixSignGraph on the following tasks: CSLR and SLT (including Sign2Gloss2Text, Sign2Text, Gloss-free Sign2Text). Please refer to Appendix A.3 for detailed descriptions of these tasks. To evaluate our model, we adopt the Word Error Rate (WER) metric for the CSLR task, and ROUGE-L F1 Score (28) and BLEU-1-4 (29) for SLT tasks.

4.1 Ablation Study

Following the previous work (5), we perform ablation studies on PHOENIX14T dataset to verify the effectiveness of the proposed MixSignGraph model.

Effects of Sign Graph Modules. As shown in Table 1, we conduct an ablation study to evaluate the individual and combined contributions of the LSG, TSG, and HSG modules. The baseline model without any graph module yields a WER of 22.3%/ 22.2% on the Dev/Test set. Furthermore, when HSG is combined with LSG and TSG, we observe consistent performance gains, which demonstrates that the proposed HSG module effectively mitigates the degradation of local features caused by cross-region feature extraction, and helps enhance the model's overall representation capability and CSLR performance.

Most importantly, the best performance is achieved when the HSG modules is inserted after LSG and TSG modules, resulting in the lowest WER of 16.7% (Dev) and 19.0% (Test), which demonstrates the importance of hierarchical local feature enhancement via HSG in complementing cross-region

spatial and temporal modeling. It confirms that HSG plays a critical role in reinforcing local features and integrating multi-granularity graph-based representations.

Effect of TCTC for Gloss-Free SLT. To demonstrate the effectiveness of our proposed TCTC mechanism, we compare a SLT model trained end-to-end ( i.e. , NOT using TCTC) and the same model pre-trained with TCTC, in terms of SLT performance. As shown in Table 2, our model pre-trained with TCTC significantly outperforms the one trained end-to-end, e.g. , our model improves gloss-free SLT performance by 23.14 ROUGE score and 19.8 BLEU4 score on PHOENIX14T dev set. In addition, on PHOENIX14T and CSL-Daily, the performance of our model pre-trained with TCTC is close to that of the gloss-based SLT model ( i.e. , pre-trained with gloss annotations), e.g. , 55.77 ROUGE and 54.54 ROUGE score. It indicates that the proposed TCTC mechanism can substantially bridge the performance gap between GFSLT and gloss-based SLT.

4.2 Qualitative Results

CSLR Qualitative Results. As shown in Table 3, we conduct qualitative analysis for MixSignGraph in the CSLR task, and show one sample from the test set of PHOENIX14, PHOENIX14T and

CSL-Daily, respectively. It can be found that MixSignGraph yields more accurate gloss predictions than MultiSignGraph, demonstrating that the newly-proposed MixSignGraph is more effective.

SLT Qualitative Results. As shown in Table 4, we present qualitative analysis of MixSignGraph in the SLT task, and show one sample from the test set of PHOENIX14T, CSL-Daily, How2Sign and OpenASL, respectively. It shows that the gloss-based Sign2Text model achieves the best performance while MixSignGraph pre-trained with TCTC also achieves satisfactory translation results.

4.3 Comparisons on CSLR Tasks

Evaluation on PHOENIX14T.

As shown in Table 5, we compare our model with existing models on CSLR performance, and we provide both the performances on the validation set and test set. Most of the current models adopt existing CNNbased backbones, and achieve good performance by injecting extra cues (11; 7; 30), adding extra constraints (6; 2; 15) or intro-

ducing attention mechanism (32). As for the GCN-based models, CoSign (35) mainly relies on pre-processed fine-grained skeleton data, and achieves 19.5%, 20.1% WER on dev, test set. While HST-GCN (34) adopts both CNN-based backbone and GCN-based backbone to extract RGB features and skeleton features respectively, and achieves 19.5%, 19.8% WER on dev, test set. It is worth mentioning that the SOTA model CrossL-Two (3) utilizes both RGB features and fine-grained skeleton features ( i.e. , keypoints in hands, body and face), and pre-trains its backbone on both PHOENIX14 and CSL-daily datasets, achieving 16.9%, 18.5% WER on dev, test set respectively. When moving to our MixSignGraph, it does not use any extra cues or pre-train on other SL datasets, but it still achieves comparable performance with CrossL-Two on test set. While on dev set, our model even outperforms CrossL-Two model by 0.2% WER.

Evaluation on PHOENIX14. We conduct a comparative analysis of our model and current CSLR models using the PHOENIX14 dataset. As illustrated in Table 5, our straightforward yet robust MixSignGraph model achieves the superior performance (16.5% WER) on the dev set and surpasses the majority of existing models, thus demonstrating the effectiveness of our MixSignGraph.

Evaluation on CSL-daily. We also show CSLR performance of our model on CSL-daily dataset. Table 6 shows that our model achieves 25.1%, 25.0% WER on the dev, test set, respectively. It indicates that our model can surpass the SOTA model by only using RGB modality.

4.4 Comparisons on SLT tasks

Evaluation on PHOENIX14T Dataset. As shown in Table 7, we compare our model with existing models using PHOENIX14T dataset on SLT tasks, which include Sign2Gloss2Text, Sign2Text, and Gloss-free Sign2Text (also known as Gloss-free SLT). Our MixSignGraph achieves comparable performance to the SOTA model (TwoStream) on both Sign2Gloss2Text and Sign2Text tasks. When moving to the gloss-free SLT task, our MixSignGraph pre-trained with TCTC mechanism significantly surpasses the current SOTA model, e.g. , 8.65 ROUGE score higher than GFSLT-VLP (16) on test set. Besides, MixSignGraph also narrows the performance gap between the gloss-free SLT model and the gloss-based SLT model, i.e. , 51.14 ROUGE score vs. 53.84 ROUGE score.

Evaluation on CSL-daily. As shown in Table 8, we also show the SLT performance of our MixSignGraph on CSL-daily dataset. It can be found that MixSignGraph outperforms the most of current

models on all the Sign2Gloss2Text, Sign2Text and gloss-free SLT tasks, especially on the gloss-free SLT task ( i.e. , 2.7 ROUGE score higher than the SOTA model).

Evaluation on How2Sign. How2Sign is a large American Sign Language dataset. Considering that the How2Sign dataset only provides sentence labels ( i.e. , NO gloss labels), we compare our proposed model with existing models in terms of gloss-free SLT performance. Note that we adopted the I3D features provided by previous work to accelerate training, which may compromise the SLT performance. As shown in Table 9, our MixSignGraph pre-trained with TCTC achieves a BLEU4 score of 11.28/10.41 on the dev/test set, outperforming the baseline model (which also uses I3D features) and thus establishing a new baseline for future work.

Evaluation on OpenASL. Similar to the How2Sign dataset, OpenASL is also a large dataset that contains only sentence labels and has a very rich vocabulary. We compare our proposed model with these baseline models, which adopt I3D as backbone for fair comparisons in terms of gloss-free SLT performance. As shown in Table 10, our model outperforms the baseline models and provides a new baseline for future work.

5 Conclusion

We propose MixSignGraph, a simple yet effective architecture that models sign sequences as graphs for feature extraction. To address local feature degradation caused by cross-region modeling in SignGraph, we design a Hierarchical Sign Graph (HSG) module that connects corresponding regions across multi-granularity features by bidirectional information exchange. We further introduce a Text-based CTC Pre-training (TCTC) strategy that generates pseudo-gloss sequences via simple NLP processing. Experiments on five benchmarks across CSLR and SLT tasks show that our model achieves strong performance without relying on any additional cues.

6 Limitations and Broader Impacts

Here, we list some potential ideas that can be further explored to improve the performance of SL tasks. Our TCTC mechanism for the gloss-free SLT task works well based on processed text, but it also encounters problems such as difficulty in CTC loss convergence, especially in cases with large vocabularies. Therefore, more effective methods to obtain pseudo labels for CTC pre-training are expected. In addition, we also highlight the potential negative social impacts. First, our method may experience unpredictable failures, so it should not be used in scenarios where such failures could have serious consequences. Second, our method is a data-driven approach and its performance may be influenced by biases in data, thus, caution is advised in the data collection process.

7 Acknowledgments

This work is supported in part by National Natural Science Foundation of China under Grant Nos. 62172208, 92467202, 62272216; Natural Science Foundation of Jiangsu Province (Key Program) under Grant No. BK20243040. This work is partially supported by Collaborative Innovation Center of Novel Software Technology and Industrialization. Training and testing were partially supported by Sulis Tier 2 HPC platform hosted by the Scientific Computing Research Technology Platform at the University of Warwick (EPSRC Grant EP/T022108/1 and the HPC Midlands+ consortium).

References

H. Hu, W. Zhou, and H. Li, 'Hand-model-aware sign language recognition,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 35, 2021, pp. 1558-1566.

Y. Min, A. Hao, X. Chai, and X. Chen, 'Visual alignment constraint for continuous sign language recognition,' in ICCV , 2021, pp. 11 542-11 551.

F. Wei and Y. Chen, 'Improving continuous sign language recognition with cross-lingual signs,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 23 612-23 621.

S. Gan, Y. Yin, Z. Jiang, H. Wen, L. Xie, and S. Lu, 'Signgraph: A sign sequence is worth graphs of nodes,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 13 470-13 479.

N. C. Camgoz, S. Hadfield, O. Koller, H. Ney, and R. Bowden, 'Neural sign language translation,' in CVPR , 2018, pp. 7784-7793.

S. Gan, Y. Yin, Z. Jiang, K. Xia, L. Xie, and S. Lu, 'Contrastive learning for sign language recognition and translation,' in Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23 , 2023, pp. 763-772.

Y. Chen, R. Zuo, F. Wei, Y. Wu, S. Liu, and B. Mak, 'Two-stream network for sign language recognition and translation,' Advances in Neural Information Processing Systems , vol. 35, pp. 17 043-17 056, 2022.

L. Tarrés, G. I. Gállego, A. Duarte, J. Torres, and X. Giró-i Nieto, 'Sign language translation from instructional videos,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 5624-5634.

S. Gan, Y. Yin, Z. Jiang, L. Xie, and S. Lu, 'Skeleton-aware neural sign language translation,' in MM , 2021, pp. 4353-4361.

H. Zhou, W. Zhou, Y. Zhou, and H. Li, 'Spatial-temporal multi-cue network for sign language recognition and translation,' TMM , 2021.

--, 'Spatial-temporal multi-cue network for continuous sign language recognition.' in AAAI , 2020.

S. Gan, Y. Yin, Z. Jiang, L. Xie, and S. Lu, 'Towards real-time sign language recognition and translation on edge devices,' in Proceedings of the 31st ACM International Conference on Multimedia , 2023, pp. 4502-4512.

S. Tang, D. Guo, R. Hong, and M. Wang, 'Graph-based multimodal sequential embedding for sign language translation,' TMM , 2021.

H. Zhou, W. Zhou, W. Qi, J. Pu, and H. Li, 'Improving sign language translation with monolingual data by sign back-translation,' in CVPR , 2021, pp. 1316-1325.

A. Hao, Y. Min, and X. Chen, 'Self-mutual distillation learning for continuous sign language recognition,' in ICCV , 2021, pp. 11 303-11 312.

A. Yin, T. Zhong, L. Tang, W. Jin, T. Jin, and Z. Zhao, 'Gloss attention for gloss-free sign language translation,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 2551-2562.

B. Zhou, Z. Chen, A. Clapés, J. Wan, Y. Liang, S. Escalera, Z. Lei, and D. Zhang, 'Gloss-free sign language translation: Improving from visual-language pretraining,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 20 871-20 881.

J. Gong, L. G. Foo, Y. He, H. Rahmani, and J. Liu, 'Llms are good sign language translators,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 18 36218 372.

K. Han, Y. Wang, J. Guo, Y. Tang, and E. Wu, 'Vision gnn: An image is worth graph of nodes,' Advances in Neural Information Processing Systems , vol. 35, pp. 8291-8303, 2022.

A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, 'Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,' in ICML , 2006, pp. 369-376.

L. Van der Maaten and G. Hinton, 'Visualizing data using t-sne.' Journal of machine learning research , vol. 9, no. 11, 2008.

Y. Chen, F. Wei, X. Sun, Z. Wu, and S. Lin, 'A simple multi-modality transfer learning baseline for sign language translation,' in CVPR , 2022, pp. 5120-5130.

Y. Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad, M. Lewis, and L. Zettlemoyer, 'Multilingual denoising pre-training for neural machine translation,' Transactions of the Association for Computational Linguistics , vol. 8, pp. 726-742, 2020.

T. Kudo and J. Richardson, 'SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,' in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , E. Blanco and W. Lu, Eds. Brussels, Belgium: Association for Computational Linguistics, Nov. 2018, pp. 66-71. [Online]. Available: https://aclanthology.org/D18-2012

O. Koller, J. Forster, and H. Ney, 'Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signers,' CVIU , vol. 141, pp. 108-125, Dec. 2015.

A. Duarte, S. Palaskar, L. Ventura, D. Ghadiyaram, K. DeHaan, F. Metze, J. Torres, and X. Giro-i Nieto, 'How2sign: a large-scale multimodal dataset for continuous american sign language,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 2735-2744.

B. Shi, D. Brentari, G. Shakhnarovich, and K. Livescu, 'Open-domain sign language translation learned from online video,' in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , 2022, pp. 6365-6379.

C.-Y. Lin, 'Rouge: A package for automatic evaluation of summaries,' in Text summarization branches out , 2004, pp. 74-81.

K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, 'Bleu: a method for automatic evaluation of machine translation,' in ACL , 2002, pp. 311-318.

R. Zuo and B. Mak, 'C2slr: Consistency-enhanced continuous sign language recognition,' in CVPR , 2022, pp. 5131-5140.

N. C. Camgoz, O. Koller, S. Hadfield, and R. Bowden, 'Sign language transformers: Joint end-to-end sign language recognition and translation,' in CVPR , 2020, pp. 10 023-10 033.

L. Hu, L. Gao, Z. Liu, and W. Feng, 'Continuous sign language recognition with correlation network,' in CVPR , 2023, pp. 2529-2539.

K. L. Cheng, Z. Yang, Q. Chen, and Y.-W. Tai, 'Fully convolutional networks for continuous sign language recognition,' in ECCV . Springer, 2020.

J. Kan, K. Hu, M. Hagenbuchner, A. C. Tsoi, M. Bennamoun, and Z. Wang, 'Sign language translation with hierarchical spatio-temporal graph neural network,' in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2022, pp. 3367-3376.

P. Jiao, Y. Min, Y. Li, X. Wang, L. Lei, and X. Chen, 'Cosign: Exploring co-occurrence signals in skeleton-based continuous sign language recognition,' in ICCV , 2023, pp. 20 676-20 686.

K. Yin and J. Read, 'Better sign language translation with STMC-transformer,' in COLING , Barcelona, Spain (Online), Dec. 2020, pp. 5975-5989. [Online]. Available: https://www.aclweb.org/anthology/2020. coling-main.525

D. Li, C. Xu, X. Yu, K. Zhang, B. Swift, H. Suominen, and H. Li, 'Tspnet: Hierarchical feature learning via temporal semantic pyramid for sign language translation,' in NIPS , vol. 33, 2020.

J. Zhao, W. Qi, W. Zhou, N. Duan, M. Zhou, and H. Li, 'Conditional sentence generation and cross-modal reranking for sign language translation,' IEEE Transactions on Multimedia , vol. 24, pp. 2662-2672, 2021.

R. C. Wong, N. C. Camgöz, and R. Bowden, 'Sign2gpt: leveraging large language models for gloss-free sign language translation,' in ICLR 2024: The Twelfth International Conference on Learning Representations , 2024.

J. Ye, X. Wang, W. Jiao, J. Liang, and H. Xiong, 'Improving gloss-free sign language translation by reducing representation density,' in NeurIPS , 2024.

D. Uthus, G. Tanzer, and M. Georg, 'Youtube-asl: A large-scale, open-domain american sign languageenglish parallel corpus,' Advances in Neural Information Processing Systems , vol. 36, pp. 29 029-29 047, 2023.

P. Rust, B. Shi, S. Wang, N. C. Camgoz, and J. Maillard, 'Towards privacy-aware sign language translation at scale,' in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Bangkok, Thailand: Association for Computational Linguistics, 2024, pp. 8624-8641. [Online]. Available: https://aclanthology.org/2024.acl-long.467

K. Lin, X. Wang, L. Zhu, K. Sun, B. Zhang, and Y. Yang, 'Gloss-free end-to-end sign language translation,' arXiv preprint arXiv:2305.12876 , 2023.

R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel, 'Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness,' in International Conference on Learning Representations , 2019.

N. C. Camgoz, O. Koller, S. Hadfield, and R. Bowden, 'Multi-channel transformers for multi-articulatory sign language translation,' in ECCV . Springer, 2020, pp. 301-319.

J. Pu, W. Zhou, H. Hu, and H. Li, 'Boosting continuous sign language recognition via cross modality augmentation,' in MM , 2020, pp. 1497-1505.

O. Koller, C. Camgoz, H. Ney, and R. Bowden, 'Weakly supervised learning with multi-stream cnn-lstmhmms to discover sequential parallelism in sign language videos,' TPAMI , 2019.

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, 'Language models are unsupervised multitask learners,' 2019.

T. N. Kipf and M. Welling, 'Semi-supervised classification with graph convolutional networks,' arXiv preprint arXiv:1609.02907 , 2016.

M. Parelli, K. Papadimitriou, G. Potamianos, G. Pavlakos, and P. Maragos, 'Spatio-temporal graph convolutional networks for continuous sign language recognition,' in ICASSP . IEEE, 2022.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al. , 'An image is worth 16x16 words: Transformers for image recognition at scale,' in International Conference on Learning Representations , 2020.

Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, 'Swin transformer: Hierarchical vision transformer using shifted windows,' in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 10 012-10 022.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, 'Attention is all you need,' in NIPS , 2017, pp. 5998-6008.

S. Brody, U. Alon, and E. Yahav, 'How attentive are graph attention networks?' arXiv preprint arXiv:2105.14491 , 2021.

W. Hamilton, Z. Ying, and J. Leskovec, 'Inductive representation learning on large graphs,' Advances in neural information processing systems , vol. 30, 2017.

Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon, 'Dynamic graph cnn for learning on point clouds,' ACM Transactions on Graphics (tog) , vol. 38, no. 5, pp. 1-12, 2019.

Y. Rong, W. Huang, T. Xu, and J. Huang, 'Dropedge: Towards deep graph convolutional networks on node classification,' in International Conference on Learning Representations , 2020. [Online]. Available: https://openreview.net/forum?id=Hkx1qkrKPr

T. Xiao, M. Singh, E. Mintun, T. Darrell, P. Dollár, and R. Girshick, 'Early convolutions help transformers see better,' Advances in neural information processing systems , vol. 34, pp. 30 392-30 400, 2021.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly summarize the main contributions of the paper, including the proposal of HSG and the TCTC training strategy. (see Sections 1, 3).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We include a dedicated 'Limitations and Borader Impacts' section in the main paper, where we discuss practical challenges, and the potential negative social impacts including high resource demands, data bias, and model reliability in real-world applications. (see Sections 6).

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our work focuses on empirical model design and evaluation for sign language translation. It does not include theoretical results that require formal assumptions or proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide comprehensive details of the model architecture, training settings , and evaluation protocol (Section 4). These include datasets, hyperparameters, and implementation details necessary to reproduce our results. We also plan to release the code and model checkpoints upon publication.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We plan to release the full codebase, model checkpoints, and detailed instructions for reproducing all experimental results, including data preprocessing, training, and evaluation scripts.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper provides comprehensive training and testing details, including data splits, model architecture, hyperparameters, learning rate, optimizer, and training epochs.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: While we report the main experimental results, we do not include error bars or statistical significance tests due to limited computational resources. In future work, we plan to provide more extensive statistical analysis over multiple random seeds or cross-validation splits.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper specifies the type of computing resources used, including GPU model (e.g., 3090), memory (e.g., 24GB) (See Section 4).

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conforms in every respect with the NeurIPS Code of Ethics. No part of our work involves deception, human subjects, sensitive data, or potentially harmful outcomes. We have ensured transparency, reproducibility, and fairness in our methodology and reporting.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our work discusses both potential positive and negative societal impacts.(See Section 6)

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The model weights used in this work are trained solely on publicly available datasets, and we do not release any new datasets or pretrained models that pose a high risk for misuse. Therefore, no additional safeguards are necessary.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We used publicly available datasets and pretrained model weights, all of which are properly cited in the paper. The licenses and terms of use for these assets were carefully reviewed and respected. We have also specified the sources, versions, and relevant URLs where appropriate.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release any new datasets, models, or code assets. All experiments are based on publicly available datasets and pretrained models, which are properly cited.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve any crowdsourcing or research with human subjects. All the datasets used in this paper are publicly available.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing or research with human subjects. All datasets used are publicly available and do not contain personally identifiable information.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [No]

Justification: Our core methodology does not involves Large Language Models (LLMs) as a central component.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Appendix

A.1 Related Work

Sign Language Tasks. The unique grammatical rules of sign language lead to some differences between the frameworks of CSLR and SLT tasks. Nevertheless, both frameworks contain a visual backbone to capture visual features from videos. However, it is a challenging task for the widely adopted CNN-based backbones to focus on sign-related features, since CNN backbones are initially designed to capture the texture and contours of objects (44). To guide CNN-based backbones to extract sign-related features, some current SL models introduce prior knowledge ( e.g. , local areas (45), skeletons (9; 30; 7) and depth images (13)) to manually inject expert knowledge into the backbone, making it focus on SL-related areas. Some other SL models try to add additional constraints ( e.g. , knowledge distillation (15; 2) and contrastive learning (6)) to optimize their backbones. Besides, there are also some SL models that turn their attention to obtaining more sign language samples by back translation (14), cross modality augmentation (46), pre-training with other sign language related datasets (22; 3) and adopting pre-trained visual models as their backbones (39). To capture effective SL-related features, SignGraph (4) represents frames as nodes and builds LSG and TSG modules to capture cross-region features both in one frame ( i.e. , the correlation of hands and face regions) and among adjacent frames ( e.g. , hand motions, facial expression changes).

In CSLR tasks, the CTC module (47) is adopted to align gloss sequences with SL videos. While SLT (5) models often adopt a pre-trained language model ( e.g. , mBART (23), GPT2 (48)) to get the translated sentence. In these SLT models, the traditional training paradigm usually involves two stages: first, pre-training their backbone using gloss sequences (equivalent to the CSLR task), then connecting a pre-trained translation model and fine-tuning the entire model. In the first stage, labeled gloss sequence is critical for improving SLT performances, as pre-training with gloss sequences can assist the model in learning segmentation and semantic information of SL. However, annotating gloss sequences for SL videos is labor-intensive and cumbersome. Therefore, gloss-free SLT tasks have gained attention (16; 17; 39; 40). The existing gloss-free SLT models usually adopt attention mechanisms (16) or contrastive language-image pre-training (17) to improve SLT performances, and have achieved promising results.

Graph Convolutional Network. Due to the graph structures, graph convolution layers (49) are more flexible for message passing and aggregation. Naturally, GCN has been effectively applied to skeleton-based data, where the connections between joints render the skeleton data inherently suitable for GCN. With the skeleton data, GCN has been applied to SL tasks, in which some models utilized GCNs to guide CNN-based backbones to extract skeleton-related features (9; 50; 34; 35). However, these approaches often rely heavily on structured inputs ( e.g. , skeleton). To reduce the dependence on structured data, the recent model VIT (51), which processes an image as patches with a transformer model, has replaced CNNs as a de facto architecture in many fields. After that, the VIG model (19) directly represents an image as a graph structure to capture irregular and complex objects, and achieves superior performance on image recognition tasks.

A.2 Datasets

Our experiment evaluation is conducted on five publicly available SL datasets, including three widely used datasets ( i.e. , PHOENIX, PHOENIX14T and CSL-Daily) and two new and large datasets ( i.e. , How2Sign and OpenASL), as described below.

PHOENIX14 (25): A widely used German SL dataset with 1295 glosses from 9 signers for CSLR. It includes 5672, 540 and 629 weather forecast samples for training, validation, and testing, respectively.

PHOENIX14T (5): A German SL dataset with both gloss annotations and translation annotations. It contains 7096, 519 and 642 samples from 9 signers for training, validation and testing, respectively. In regard to the two-stage annotations, the sign gloss annotations have a vocabulary of 1066 different signs for CSLR, while the German translation annotations have a vocabulary of 2877 different words for SLT.

CSL-Daily (14): A Chinese SL dataset with 18401, 1176, 1077 labeled videos from 10 signers for training, testing and validation. It includes 2000 gloss annotations for CSLR and 2343 Chinese words for SLT.

How2Sign (26): A multimodal and multiview continuous American Sign Language (ASL) dataset. It consists of a parallel corpus, which has more than 80 hours of sign language videos and a set of corresponding modalities ( i.e. , speech, English transcripts, and depth). In this dataset, there are 31128, 1741 and 2322 samples for training, validation and testing, respectively. In regard to the annotations, there are no gloss-level annotations, thus the dataset is used for SLT tasks. It is worth mentioning that our experiment only uses the RGB modality ( i.e. , NOT all modalities) with frontal view ( i.e. , NOT all views) to train and test our model.

OpenASL (27): A large ASL dataset with translation annotations for SLT. It includes 280 hours of ASL videos from more than 200 signers, and contains 96476, 997 and 999 samples for training, validation and testing, respectively.

We provide detailed information of the five datasets in Table 11. In addition, we also provide the token vocabulary obtained by the mBART tokenizer and the vocabulary of pseudo gloss labels in TCTC for reference. It is worth noting that the datasets we downloaded ( i.e. , shown in Table 11) may slightly differ from the official versions described in their own papers.

A.3 Evaluation Tasks.

We evaluate the performance of our model on the following tasks: : CSLR and SLT (including Sign2Gloss2Text, Sign2Text, Gloss-free Sign2Text. (1) CSLR: Recognizing a sign sequence as a corresponding gloss sequence, also known as Sign2Gloss. (2) SLT : Translating a sign sequence into a spoken sentence, and it can be classified into the following three categories: (2.1) Sign2Gloss2Text: First, the recognized gloss sequence is obtained based on the CSLR model. Then, the predicted glosses are translated to a spoken sentence by a translation model. (2.2) Sign2Text : Directly translating a SL video into a spoken sentence. Gloss annotations are required for pre-training a CSLR model, and visual features output by the CSLR model are sent to a translation model for generating the spoken sentence. (2.3) Gloss-free Sign2Text: Directly translating a SL video into a spoken sentence, while NOT using gloss annotations to pretrain a CSLR model. This task is also known as gloss-free SLT.

A.4 Visualization of MixSignGraph

To verify whether our model can effectively capture sign-related features, we select a sign video from the PHOENIX14T test set and visualize the constructed graph structure in both stages. In Figure 5, we show the graphs in the LSG , TSG and HSG modules. For clarity, only a subset of the edges is displayed. In the LSG module, our model links nodes with similar contents or related semantic representation ( e.g. , hand regions and face regions), to extract better intra-frame cross-region features. In the TSG module, our model builds edges among adjacent frames to track dynamic changes in gestures and facial expressions, to capture inter-frame cross-region features. In the HSG module, our model connects the same regions with different granularities ( i.e. , the corresponding regions in feature maps), to enhance one-region features. In addition, we also highlight some 'unimportant' edges between nodes in the background in red. As seen in Figure 5, the background nodes in the LSG module are naturally connected to their neighboring nodes, and there are still a few background nodes connected in the TSG module. Fortunately, background nodes do not 'disturb' nodes in sign-related regions, demonstrating the effectiveness of our model.

A.5 Model Comparison and Analysis

To provide a more comprehensive analysis, we compare our model with the existing models in the following aspects: tasks, architecture information, flops, number of parameters, and pre-trained datasets. As shown in Table 12, among the existing models, the TwoStream model adopts a dual S3D backbone and pre-train their backbone with Kinetics-400, which is an action recognition dataset. Based on the backbone in TwoStream, the CrossL-Two model further pre-trains their backbone with existing SL datasets. Sign2GPT adopts Dino-V2 ViT as the visual backbone, which is pre-trained on LVD-142M (a very large vision dataset), and adopts GPT2 as the translation model, which is also pre-trained on a very large text dataset WebText. In regard to MixSignGraph, it contains fewer parameters and FLOPs, and it is only pre-trained on small datasets ( i.e. , ImageNet-1K and CC25 datasets), while achieving the promising performance on multiple SL tasks.

A.6 Other Ablation Study

Effect of Hyperparameter K . In the MixSignGraph model, four hyperparameters: K 1 l and K 2 l in the LSG modules, K 1 t and K 2 t in the TSG modules need to be set to proper values. To determine the best values for these hyperparameters, we evaluate the CSLR performance by changing K l ( i.e. , both K 1 l and K 2 l ), K t ( i.e. , both K 1 t and K 2 t ) values from 2 to 9, 7 to 91, respectively. However, due to the large number of possible combinations of K 1 l , K 2 l , K 1 t and K 2 t , it is impractical to make an exhaustive search to find the globally optimal values of

20.0

20.0

19.5

19.5

19.0

19.0

WER

18.5

Dev

WER

18.5

Dev

18.0

Test

18.0

Test

17.5

17.5

17.0

17.0

16.5

2

4

以

6

7

8

9

16.5

7

21

35

K

49

63

77

91

20.0

20.5

19.5

20.0

19.0

19.5

WER

18.5

Dev

WER

19.0

Dev

18.0

Test

18.5

18.0

Test

17.5

17.5

17.0

16.5

2

3

4

5

6

7

8

9

16.5

7

21

35

49

63

77

16

K?

K2

these hyperparameters. Therefore, we fix three of the parameters at a time and adjust the fourth to find suitable values. Finally, based on the WER performance on the DEV set, we set K 1 l , K 2 l , K 1 t and K 2 t to 3, 4, 49 and 49, respectively, as shown in Figure 6.

Effect of Graph Type. In MixSignGraph, we build dynamic dense graphs ( i.e. , connecting the top K neighbors for each node) in the LSG module, dynamic sparse graphs ( i.e. , connecting only the top K

(a) Effect of graph types.

(b) Effect of order of proposed sign graph modules.

(a) Backbone . Comparison of different backbones.

(b) Distance function . Comparison of distance functions.

(c) Graph Convolution Effect of different GCN layers.

node pairs) in the TSG module, and fixed graphs ( i.e. , connecting the same regions between feature maps) in the HSG module. To verify the effectiveness of the above graph types, we evaluate the CSLR performance by changing the graph types in LSG , TSG and HSG modules. For simplicity, we only use one LSG module, one TSG module and one HSG module in the experiment. As shown in Table 13a, when using a spare graph in the LSG module, a dense graph in the TSG module, or a dynamic dense graph ( i.e. , building in the similar way with the LSG module) in the HSG module, there is a noticeable performance drop. That is to say, it is effective to adopt dense, sparse and fixed graphs in the LSG , TSG and HSG modules, and the combination of these mixed graphs can make our model achieve the best recognition performance.

Effect of Ordering of Sign Graph Modules. We also evaluate the CSLR performance by changing the ordering of three graph modules. As shown in Table 13b, different orderings of the graph modules lead to slight differences in the CSLR performance, i.e. , the ordering has a little effect on the overall effectiveness of feature extraction and integration. Nevertheless, the ordering TSG → LSG → HSG can help the model to achieve the best CSLR performance. Thus our model MixSignGraph adopts the ordering TSG → LSG → HSG by default.

Effects of Different Backbones. To verify the effectiveness of the MixSignGraph model, we replace our backbone with other patch-based networks ( e.g. , CvT (58), Swin Transformer (52)), or replace our graph modules with Self-Attention (SA) layers (53). As shown in Table 14a, simply adopting the patch-based SOTA backbones or substituting our graph modules with SA layers does not lead to appealing performances on the CSLR task. In contrast, our proposed MixSignGraph backbone achieves a satisfactory performance by effectively capturing both cross-region and one-region features.

Effect of Distance Function. In the MixSignGraph, we use the KNN algorithm to find the nearest neighbors for a node/patch based on the distance between two nodes. To select a suitable distance

function to measure the distance between nodes, we evaluate the CSLR performance with the following commonly used distance functions, i.e. , Cosine distance, Chebyshev distance and Euclidean distance. As shown in Table 14b, there are subtle performance differences, when using different distance functions. In MixSignGraph, we adopt Euclidean distance for better performance.

Effect of GCN Layer. To verify the effectiveness of the graph convolution layer, we test the CSLR performance by adopting the following representative variants of graph convolution, i.e. , GATv2Conv (54), SAGEConv (55), GCNConv (49), and EdgeConv (56). As shown in Table 14c, even using different GCN layers, our model still achieves good performance. It indicates that our MixSignGraph model has good flexibility in GCN layer selection. Nevertheless, among the above GCN layers, EdgeConv can help our model to achieve the best CSLR performance, i.e. , 16.7% WER. Therefore, we adopt the EdgeConv layer in MixSignGraph for better performance.

Effect of Patch Sizes. Patches with different sizes have different receptive fields, thus can capture different-range features and further affect the model performance. To observe the effect of patch sizes, we evaluate the CSLR performance by only adopting one LSG module, one TSG module and one HSG module, while changing the patch size. As shown in Table 14d, when using a patch size that is too small ( i.e. , 8 × 8) or too large ( i.e. , 32 × 32), the model MixSignGraph shows a worse performance. We found that a suitable patch size strikes the best balance, leading to good performance.

Effect of Multiscale Sign Graphs. To leverage the scale-invariant property of images (19), PyVIG adopts a pyramid architecture that gradually increases patch size from 4 to 32 by shrinking the spatial size of feature maps to enhance feature extraction capability. Similarly, to extract better sign features and achieve higher CSLR performance, we gradually add LSG , TSG and HSG modules after the early stage of the patchify stem. According to Table 14e, adding more LSG , TSG and HSG modules does not always bring performance gains. Therefore, MixSignGraph adopts two LSG , TSG and HSG modules, i.e. , increasing patch size from 16 to 32, aiming to achieve the best performance.

Effect of DropEdge. DropEdge (57) can alleviate the over-smoothing and overfitting problems in dense graphs by randomly removing a certain number of edges from the input graph at each training epoch. Considering that we build dense graphs for the LSG and HSG modules, we introduce DropEdge into these two modules and change the drop rates of DropEdge to evaluate the CSLR performance. As shown in Table 14f, adding DropEdge does not improve the performance of our MixSignGraph. It indicates that our model can achieve excellent performance without relying on external modules ( e.g. , DropEdge) and it can handle over-smoothing and overfitting problems well.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MixSignGraph, which represents a sign sequence as mixed graphs of nodes. Compared to the previous work, termed as MultiSignGraph, it makes two key changes. The first is a Hierarchical Sign Graph module, and the second is Text-based CTC Pre-training strategy.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n1. The paper is written in a clear and concise manner, making it easy to follow the proposed methodology, experimental setup, and results. The logical organization enhances readability and ensures that the contributions are effectively communicated to the reader.\n2. The experimental evaluation demonstrates strong performance across multiple benchmarks, suggesting that the proposed method is effective and competitive with existing approaches. These results support the claim that the method offers practical benefits in real-world applications.\n3. The proposed approach introduces several novel components that distinguish it from previously published methods. These innovations contribute meaningfully to the field and offer new directions for future research in multi-modal or hierarchical learning frameworks.\n\nCons:\n1. While the paper employs different backbone architectures compared to baseline methods, it does not provide any discussion or comparison regarding computational costs—such as training time, inference speed, or model complexity. Including such an analysis would help readers better understand the trade-offs between performance gains and resource requirements.\n2. In Table 1, the meanings of the labels \"1\" and \"2\" are not explained. This lack of clarification makes it difficult for readers to interpret the experimental variations being compared. A brief explanation or footnote in the table would significantly improve its clarity and usefulness.\n\nMinor questions:\n1. It would be helpful to clarify how the first Hierarchical Sign Graph module behaves when the input feature vector $u_l$  is absent. Does the module skip processing at that level, or is there a default mechanism in place? Understanding this edge case would improve reproducibility and robustness assessment.\n2. In Table 2, the abbreviation \"TCP\" appears without prior definition. Please clarify what \"TCP\" stands for to avoid confusion and ensure accurate interpretation of the experimental settings.\n3. If certain datasets include gloss annotations, would the authors still employ the proposed Text-based CTC Pre-training strategy? It would be interesting to know whether the presence of such annotations influences the choice of pre-training method, and if so, how.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes MixSignGraph, a novel GCN-based framework for continuous sign language recognition (CSLR) and translation (SLT). The introduced novelty is mainly two-fold: i) MixSignGraph introduces a new Hierarchical Sign Graph (HSG) module that combines features at different levels of granularity, allowing the model to focus on both global and local shapes at the same time; and ii) it introduces Text-Based CTC Pre-training (TCTC) for gloss-free SLT tasks, where pseudo gloss sequences are built from translations by applying traditional NLP techniques like stemming and lemmatization.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The HSG design is intuitive and effectively addresses a clear limitation of prior work.\n- The TCTC pre-training is an interesting practical workaround for gloss-free settings.\n- The experimental results consistently show improvements across CSLR and SLT tasks, with a performance competitive or superior to the state of the art.\n- The proposed approach is modality efficient, as it achieves high performance only from RGB data.\n\nWeaknesses:\n- Although the HSG design is intuitive, it lacks formal justification or theoretical analysis of why it improves representation.\n- The use of GCNs and multiple graph modules (LSG, TSG, HSG) might raise scalability issues on long video sequences. This issue is not analyzed nor is it mentioned in the limitations section.\n- Only lexical overlap metrics (ROUGE and BLEU) are used for evaluation. This is especially problematic in SLT, where there’s often no one-to-one correspondence between signs and words, and translations tend to be semantically but not lexically equivalent.\n- The writing has many typos (e.g. \"Limitations and *Borader* Impacts\"), which is not acceptable. These sometimes make the idea hard to follow. The authors should definitely fix these.\n\nQUESTIONS:\nPlease address the weaknesses I mentioned above, in particular those related with the scaling of the method to long sequences and the evaluation metrics. If these are addressed appropriately and a writing review is performed to clear the typos, I am willing to raise my score.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

comments: SUMMARY:
The paper introduces a new method named MixSignGraph for sign language recognition from videos. The method is an extension to SignGraph [4] that introduces local sign graph and temporal sign graph for sign language recognition. The proposed method adds an additional hierarchical sign graph to capture correlations between visual features. Furthermore, the paper introduces Text-based CTC Pre-training to further improve the performance.

STRENGTHS AND WEAKNESSES:
Positives:

-	The paper shows that by adding the hierarchical sign graph, the proposed method can improve the sign language recognition performance.

-	The paper has conducted extensive experiments on several datasets and alation studies to verify the proposed method.

Negatives:

-	Since the method is an extension of the SignGraph [4] method, the novelty and technical contribution of the method is limited. The main difference is by introducing the hierarchical sign graph into the method.

-	Although the Text-based CTC Pre-training shows improvements, however, the model may overfit to these datasets by using the texts in the datasets for pre-training. As a result, the generalization ability of the method may be reduced.

-	In tables 3, 4, 5, 6, 7, why not comparing to the original SignGraph [4] method?

-	Models such as the Swin transformer also utilize hierarchical representations. Will a Swin transformer perform as good as the hierarchical sign graph in the proposed method?

QUESTIONS:
Please see weaknesses above regarding questions.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes MixSignGraph, a novel graph neural network architecture for sign language tasks, including Continuous Sign Language Recognition (CSLR) and Sign Language Translation (SLT). Building upon SignGraph, the authors introduce a Hierarchical Sign Graph (HSG) module to enhance local representations by aggregating same-region features across multi-scale feature maps. This design mitigates the degradation of local information typically caused by cross-region feature fusion. Furthermore, the authors present a simple yet effective Text-based CTC Pre-training (TCTC) strategy for gloss-free SLT, which generates pseudo gloss labels from raw text to guide model pre-training. Extensive experiments on five public datasets demonstrate the model’s superiority over existing state-of-the-art (SOTA) approaches, without relying on any additional modalities or annotations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. Effective Enhancement of Local Representations via HSG:\nThe HSG module strengthens local feature quality by establishing hierarchical connections across multi-resolution feature maps, complementing spatial and temporal cross-region modeling.\n\n2. Innovative Gloss-Free Pre-training Strategy (TCTC):\nThe TCTC method enables pre-training using only textual labels, significantly improving gloss-free SLT performance and narrowing the gap with gloss-supervised models.\n\n\nWeaknesses\n\n1. Incremental Contribution over SignGraph.\nWhile the proposed method introduces performance gains by adding modules such as the Hierarchical Sign Graph (HSG) on top of SignGraph, the technical novelty appears limited. The framework largely builds upon prior work, and the architectural modifications, though effective, may not be conceptually groundbreaking.\n\n2. Insufficient Comparison with Recent State-of-the-Art.\nThe paper lacks comprehensive comparisons with several recent state-of-the-art models (e.g., [1, 2, 3]). Some of these models, including those that rely solely on pose or skeleton data, report superior or comparable results, yet are not adequately discussed or benchmarked against MixSignGraph.\n\n[1] Li, Zecheng, et al. \"Uni-Sign: Toward Unified Sign Language Understanding at Scale.\" arXiv preprint arXiv:2501.15187 (2025).\n\n[2] Jiao, Peiqi, Yuecong Min, and Xilin Chen. \"Visual Alignment Pre-training for Sign Language Translation.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[3] Chen, Zhigang, et al. \"C 2 RL: Content and Context Representation Learning for Gloss-free Sign Language Translation and Retrieval.\" IEEE Transactions on Circuits and Systems for Video Technology (2025).\n\n3. Semantic Inaccuracy in Pseudo Glosses.\nThe pseudo glosses used in TCTC are generated via basic linguistic preprocessing (e.g., lemmatization and tokenization), which may fail to capture the true semantics and structure of authentic gloss annotations. The paper should provide a more detailed analysis of how such semantic discrepancies affect performance, especially in gloss-free SLT tasks.\n\nQUESTIONS:\nPlz refer to the weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}